An underwater sonar image data enhancement method
By extracting target object regions and category labels through segmentation and classification models, and combining conditional generative adversarial networks and style coding modules, the problem of poor generalization performance in sonar image generation is solved. The generated sonar images have reasonable visual effects and conform to the sonar imaging rules, thus expanding the diversity and quality of training data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-01-08
- Publication Date
- 2026-06-02
Smart Images
Figure CN122135131A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data augmentation technology, and in particular to a method for augmenting underwater sonar image data. Background Technology
[0002] Underwater target detection, recognition, and segmentation have significant applications in marine resource exploration, underwater rescue, national defense, and marine engineering. Compared to optical imaging, sonar, with its advantages of not being limited by water transparency and having a long range, has become the primary means of long-range underwater imaging and detection. However, deep learning-based intelligent interpretation technology for sonar images heavily relies on large-scale, high-quality training samples. In practical applications, acquiring real underwater sonar images is costly and time-consuming, and samples for rare or sensitive targets are extremely scarce. Furthermore, pixel-level semantic annotation of sonar images is highly specialized and time-consuming, further limiting the scale of the dataset. Therefore, developing efficient data augmentation techniques capable of generating diverse, high-quality synthetic sonar images using limited samples has become a key requirement for promoting the practical application of this technology.
[0003] In one existing technique, a style transfer method based on generative adversarial networks (GANs) is used for sonar image sample synthesis. This type of method typically takes optical or simulated images as input and trains a GAN network to learn the imaging style of the sonar image, such as specific noise distribution, texture characteristics, and grayscale features, thereby converting the input image into a synthetic image with a sonar style. However, model training heavily relies on a large number of real sonar samples of specific target categories. When it is necessary to generate sonar images of target categories that have not appeared in the training data, the model cannot learn effective feature representations of such targets from the training data, making it difficult to generate visually plausible images that conform to the physical laws of sonar imaging.
[0004] Existing technologies suffer from poor generalization performance in simulated sonar image generation due to the absence of unknown categories in model training. Summary of the Invention
[0005] This invention provides a method for enhancing underwater sonar image data to address the problem of poor generalization performance in simulated sonar image generation.
[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides an underwater sonar image data enhancement method, comprising:
[0007] Get the source image containing the target object;
[0008] The source image is input into a pre-trained segmentation model to obtain the segmentation mask of the target object;
[0009] Based on the segmentation mask, the target object region is extracted from the source image;
[0010] The target object region is input into a pre-trained classification model to obtain the category label of the target object;
[0011] The target object region and the category label are input into the underwater sonar simulation image generation module to obtain a simulated sonar image of the target object;
[0012] Obtain the background database and the simulated sonar object database;
[0013] The background database consists of background sonar images without foreground targets acquired by sonar sensors in real aquatic environments.
[0014] The simulated sonar image, category label, and segmentation mask of the target object are stored in the simulated sonar object database;
[0015] A background sonar image is randomly selected from the background database. The simulated sonar image of the target object and a preset number of simulated sonar images taken from the simulated sonar object database are combined with the background sonar image to obtain a complete simulated sonar image.
[0016] In one optional implementation, the training process of the segmentation model includes:
[0017] The first step is to obtain the first segmentation training dataset and the second segmentation training dataset;
[0018] Wherein, the first segmentation training dataset refers to training images and segmentation masks in the optical domain, and the second segmentation training dataset refers to training images and segmentation masks in the remote sensing and simulator domains.
[0019] The second step is to train the first neural network model based on the first segmentation training dataset, using the first segmentation loss function as a constraint. The training is completed when the number of training iterations reaches the first maximum preset number of rounds, and the first segmentation model is obtained.
[0020] The third step is to train the first segmentation model based on the second segmentation training dataset, using the second segmentation loss function as a constraint. The training is completed when the number of training iterations reaches the second maximum preset number of rounds, and the second segmentation model is obtained.
[0021] The first neural network model is a U-Net model, and the first and second segmentation loss functions are composed of binary cross-entropy loss and Dice loss, as expressed in Equation 1:
[0022] (Equation 1)
[0023] Where T is the number of samples in the batch. It's a real label. It is the result of model prediction. yes and The number of intersections, i.e., the number of pixels with the same value. yes The number of elements in the middle. yes The number of elements in the middle.
[0024] In an optional implementation, before obtaining the first segmentation training dataset and the second segmentation training dataset, the method further includes:
[0025] The first step is to acquire the original datasets from the optical domain, the remote sensing domain, and the simulator domain.
[0026] The original dataset in the optical domain includes optical domain images and optical domain instance segmentation labels; the original dataset in the remote sensing domain includes remote sensing domain images and remote sensing domain bounding box labels; and the original dataset in the simulator domain includes simulator domain images and simulator domain segmentation masks.
[0027] The second step involves cropping each instance in the original optical domain dataset based on the optical domain instance segmentation label to obtain the optical domain processed image and the optical domain segmentation mask.
[0028] The third step involves extracting the target region from the original dataset in the remote sensing domain based on the target bounding box labels in the remote sensing domain, performing morphological processing on the target region, and obtaining the processed image and the segmentation mask in the remote sensing domain.
[0029] The optical domain processed image and the optical domain segmentation mask are used as the first segmentation training dataset, and the remote sensing domain processed image and the remote sensing domain segmentation mask are merged with the original simulator domain dataset to form the second segmentation training dataset.
[0030] In one optional implementation, the training process of the classification model includes:
[0031] The first step is to obtain the classification training dataset;
[0032] The classification training dataset includes optical images, remote sensing images, and sonar simulator images corresponding to all categories of the real sonar dataset;
[0033] The second step is to train the pre-trained classification model based on the classification training dataset, using a classification loss function as a constraint. The training is completed when the number of training iterations reaches the third maximum preset number of rounds, and the classification model is obtained.
[0034] The pre-trained classification model is a ResNet18 model trained on ImageNet, and the classification loss function is the cross-entropy loss function, expressed as Equation 2:
[0035] (Equation 2)
[0036] Where M is the number of categories, It is an indicator variable; it takes the value 1 when the true class of variable o is equal to c, and 0 otherwise. It is the predicted probability that sample o belongs to category c.
[0037] In one optional implementation, the step of inputting the target object region and the category label into the underwater sonar simulation image generation module to obtain a simulated sonar image of the target object includes:
[0038] Determine whether the category label is a target category that has already participated in training;
[0039] If the category label is a target category that has been trained, then the embedding vectors of the target object region and the category label are input into the conditional generative adversarial network to generate a simulated sonar image of the target object;
[0040] If the category label is a target category that has not participated in training, then the target object region is input into the style encoding module to obtain the style encoding vector, and the target object region and the style encoding vector are input into the conditional generative adversarial network to generate a simulated sonar image of the target object.
[0041] The conditional generative adversarial network is a CycleGAN-based model that includes a generator and a discriminator.
[0042] In one optional implementation, the training process of the conditional generative adversarial network includes:
[0043] The first step is to obtain a multi-source domain training dataset, a real sonar training dataset, and a class label set for objects of known categories;
[0044] The multi-source domain training dataset includes optical training images, remote sensing training images, and simulator training images of the known category objects, and the real sonar training dataset includes real sonar images and real sonar segmentation masks.
[0045] The second step is to extract the first target object region set from the real sonar image based on the real sonar segmentation mask.
[0046] The third step is to input the multi-source domain training dataset into the pre-trained segmentation model to obtain the segmentation mask of the multi-source domain training dataset.
[0047] The fourth step is to extract the second target object region set from the images in the multi-source domain training dataset based on the segmentation mask of the multi-source domain training dataset.
[0048] Fifth step: Based on the first target object region set, the second target object region set, and the category label set, the conditional generative adversarial network is trained. The generator part is constrained by the generation loss function, and the discriminator part is constrained by the discrimination loss function. The training is completed when the number of training times reaches the fourth maximum preset number of rounds, and the trained conditional generative adversarial network is obtained.
[0049] The generation loss function is expressed as Equation 3:
[0050] ; (Equation 3)
[0051] in, It is to combat losses. It is a cycle-consistent loss. It's a loss of identity. It is a loss due to physical constraints. , , These are weighting coefficients. The adversarial loss, cycle consistency loss, and identity loss are calculated according to the definition of the CycleGAN model.
[0052] In one optional implementation, the step of inputting the target object region into the style encoding module to obtain a style encoding vector includes:
[0053] The first step is to input the target object region into the style feature extractor to obtain the depth visual features of the target object region;
[0054] The style feature extractor is a pre-trained convolutional neural network with the last fully connected layer removed. The pre-trained convolutional neural network is a ResNet18 model trained on ImageNet.
[0055] The second step is to obtain the object visual feature library;
[0056] The third step is to calculate the similarity between the depth visual features of the target object region and the depth visual features of all objects in the object visual feature library, and obtain the similarity calculation result.
[0057] The similarity calculation uses a cosine distance metric, expressed as Equation 4:
[0058] (Equation 4)
[0059] in, It is the depth visual feature of the target object region. It is the depth visual feature of the i-th category object in the object visual feature library;
[0060] The fourth step is to obtain the depth visual features of a preset number of similar objects from the object visual feature library according to the similarity calculation results, in descending order of similarity, and perform a weighted average to obtain the style encoding vector of the target object.
[0061] The style encoding vector is represented by Equation 5:
[0062] (Equation 5)
[0063] in, It is the generated style encoding vector, and k is the preset similarity number. It is the depth visual feature of the j-th similar object. These are the corresponding weights, and =1.
[0064] In one optional implementation, the process of constructing the object visual feature library includes:
[0065] The first step is to obtain the material feature training dataset;
[0066] The second step is to input the material feature training dataset into the style feature extractor to obtain deep visual features;
[0067] The third step is to perform global average pooling on the depth visual features to obtain the style feature vector of the material feature training data.
[0068] The fourth step is to build a visual feature library for objects;
[0069] The fifth step is to store the style feature vector and category label of each training sample of the material feature training data into the object visual feature library.
[0070] In one optional implementation, the calculation process of the physical constraint loss includes:
[0071] The first step is to obtain a sample from the multi-source domain training dataset as the input sample;
[0072] The second step is to perform a geometric extraction operation on the segmentation mask of the input sample to obtain the geometric parameters of the input sample;
[0073] The geometric parameters include the equivalent area, perimeter, minimum bounding rectangle size, and major and minor axes of the minimum bounding ellipse of the input sample.
[0074] The third step is to approximate the input sample as a simple geometric shape or a combination of simple geometric shapes according to the geometric parameters and a preset rule, and calculate the theoretical sonar target intensity of the input sample based on the vertical incidence assumption.
[0075] The simple geometric shapes include spheres, cylinders, rectangular plates, circular plates, and ellipsoids;
[0076] The fourth step is to input the input sample into the generator of the conditional generative adversarial network to obtain the generated sample;
[0077] The physical constraint loss is expressed as Equation 6:
[0078] (Equation 6)
[0079] in, This represents the average pixel value of the generated samples. and These are the minimum and maximum values of the theoretical intensity range of the target object in the sonar image, respectively. It is the weighting coefficient of the physical constraints, with a value ranging from 0.1 to 0.5.
[0080] In one optional embodiment, the step of synthesizing the simulated sonar image of the target object and a preset number of simulated sonar images retrieved from the simulated sonar object database with the background sonar image to obtain a complete simulated sonar image includes:
[0081] The first step is to randomly insert the set of simulated sonar images into the background sonar image to obtain the first sonar composite image;
[0082] The simulated sonar image set includes simulated sonar images of the target object and a preset number of simulated sonar images retrieved from the simulated sonar object database;
[0083] The second step is to obtain the category label of each object in the simulated sonar image set and its position in the first sonar composite image, and obtain the second sonar composite image based on the preset overlap processing rules.
[0084] The third step is to calculate the brightness difference between each object in the simulated sonar image set and the background sonar image, and perform brightness adjustment processing according to the brightness adjustment ratio to obtain the third sonar composite image.
[0085] The calculation of the brightness adjustment ratio is expressed as Equation 7:
[0086] (Equation 7)
[0087] in, It is the brightness adjustment ratio of the i-th object in the simulated sonar image set. It is the average pixel value of the background sonar image. It is the average pixel value of the i-th object in the simulated sonar image set;
[0088] The fourth step involves applying Gaussian filtering to perform alpha fusion processing on each object in the third sonar composite image to obtain a complete sonar simulation image.
[0089] Compared with the prior art, the present invention has the following beneficial effects:
[0090] This invention provides a method for enhancing underwater sonar image data, comprising:
[0091] The process involves: acquiring a source image containing the target object; inputting the source image into a pre-trained segmentation model to obtain a segmentation mask for the target object; extracting the target object region from the source image based on the segmentation mask; inputting the target object region into a pre-trained classification model to obtain a category label for the target object; inputting the target object region and the category label into an underwater sonar simulation image generation module to obtain a simulated sonar image of the target object; acquiring a background database and a simulated sonar object database; wherein the background database consists of background sonar images without foreground targets collected by sonar sensors in a real aquatic environment; storing the simulated sonar image, category label, and segmentation mask of the target object into the simulated sonar object database; randomly selecting a background sonar image from the background database; and combining the simulated sonar image of the target object and a predetermined number of simulated sonar images retrieved from the simulated sonar object database with the background sonar image to obtain a complete simulated sonar image. Attached Figure Description
[0092] Figure 1 This is a schematic flowchart of an underwater sonar image data enhancement method provided in an embodiment of the present invention;
[0093] Figure 2 This is a schematic diagram of the sonar target approximation module structure provided in an embodiment of the present invention. Detailed Implementation
[0094] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0095] Reference Figure 1 This invention provides a method for enhancing underwater sonar image data, comprising the following steps:
[0096] S11, Obtain the source image containing the target object;
[0097] S12, input the source image into the pre-trained segmentation model to obtain the segmentation mask of the target object;
[0098] S13, Based on the segmentation mask, extract the target object region from the source image;
[0099] S14, Input the target object region into the pre-trained classification model to obtain the category label of the target object;
[0100] S15, input the target object region and the category label into the underwater sonar simulation image generation module to obtain the simulated sonar image of the target object;
[0101] S16, Obtain the background database and the simulated sonar object database;
[0102] S17, store the simulated sonar image, category label and segmentation mask of the target object into the simulated sonar object database;
[0103] S18, randomly select a background sonar image from the background database, and combine the simulated sonar image of the target object and a preset number of simulated sonar images taken from the simulated sonar object database with the background sonar image to obtain a complete sonar simulation image.
[0104] In step S11, a source image containing the target object is obtained.
[0105] The source image refers to the two-dimensional image containing the object to be processed, including but not limited to optical images, remote sensing images, and sonar images generated by sonar simulators. Optical images are obtained by collecting open-source datasets or by taking pictures with a camera. Remote sensing images are obtained by collecting open-source datasets. Sonar images generated by sonar simulators are obtained by sonar simulators of underwater simulators such as HoloOcean.
[0106] In step S12, the source image is input into a pre-trained segmentation model to obtain the segmentation mask of the target object.
[0107] Specifically, the training process of the segmentation model includes: acquiring a first segmentation training dataset and a second segmentation training dataset; wherein, the first segmentation training dataset refers to training images and segmentation masks in the optical domain, and the second segmentation training dataset refers to training images and segmentation masks in the remote sensing and simulator domains; training a first neural network model based on the first segmentation training dataset, constrained by a first segmentation loss function, and completing the training when the number of training iterations reaches a first maximum preset number of iterations, thereby obtaining a first segmentation model; training the first segmentation model based on the second segmentation training dataset, constrained by a second segmentation loss function, and completing the training when the number of training iterations reaches a second maximum preset number of iterations, thereby obtaining a second segmentation model;
[0108] The first and second maximum preset number of rounds are both set to 500 rounds. Both the first and second neural network models are U-Net models, meaning they are the same model. The U-Net model is first trained on the first segmentation training dataset to obtain a pre-trained model, and then further trained on the second segmentation training dataset. The first and second segmentation loss functions consist of binary cross-entropy loss and Dice loss, expressed as follows:
[0109] ;
[0110] Where T is the number of samples in the batch. It's a real label. It is the result of model prediction. yes and The number of intersections, i.e., the number of pixels with the same value. yes The number of elements in the middle. yes The number of elements in the text.
[0111] It should be noted that, before obtaining the first segmentation training dataset and the second segmentation training dataset, the method further includes: obtaining the original datasets in the optical domain, the original datasets in the remote sensing domain, and the original datasets in the simulator domain.
[0112] Specifically, the original dataset for the optical domain includes optical domain images and optical domain instance segmentation labels. Using the open-source COCO dataset, each instance is cropped based on its segmentation label to generate an image containing only a single object and its corresponding binary segmentation mask (0 for background, 1 for foreground). The original dataset for the remote sensing domain includes remote sensing domain images and remote sensing domain bounding box labels. Using the open-source DOTA remote sensing dataset, which provides bounding boxes, this dataset consists of RGB and grayscale images. The RGB images are from Google Earth and CycloMedia, and the grayscale images are from the panchromatic bands of GF-2 and JL-1 satellite images. Considering that the segmented objects will be converted into sonar images later, requiring objects that can reasonably appear in water, images of unreasonable objects such as tennis courts, basketball courts, and football fields from DOTA were removed, retaining images of types such as airplanes, ships, and vehicles. The target region is extracted based on the target bounding box labels in the remote sensing domain. Morphological processing is then performed on the target region, including OTSU automatic thresholding, opening operations to eliminate noise, closing operations to fill holes, and region growing, generating a pseudo-segmentation mask. This mask is then manually selected to ensure quality. The simulator-domain original dataset includes simulator-domain images and simulator-domain segmentation masks. Specifically, it uses the HoloOcean sonar simulator, an open-source underwater simulator, to directly generate simulated sonar images containing the target object and pixel-level segmentation masks. All processed images and masks are uniformly scaled or cropped to 256*256 pixels. The optical domain processed images and their segmentation masks are used as the first segmentation training dataset, and the remote sensing domain processed images and their segmentation masks are merged with the simulator-domain original dataset to form the second segmentation training dataset.
[0113] In step S13, the target object region is extracted from the source image based on the segmentation mask.
[0114] Specifically, the segmentation mask is multiplied pixel by pixel with the source image, that is, the background pixels are set to 0, to obtain a foreground image containing only the target pixels.
[0115] In step S14, the target object region is input into a pre-trained classification model to obtain the category label of the target object.
[0116] Specifically, the training process of the classification model includes: acquiring a classification training dataset, which includes optical images, remote sensing images, and sonar simulator images corresponding to all categories of the real sonar dataset. The real sonar dataset comes from open-source forward-looking sonar image datasets, including the UATD dataset, FLS_Detetction_Dataset, and marine debris dataset. The categories include aircraft, ships, human bodies, cubes, spheres, tires, circular cages, square cages, metal barrels, underwater vehicles, beverage bottles, cans, chains, hooks, propellers, shampoo bottles, wires, and valves.
[0117] Furthermore, based on the aforementioned classification training dataset, the pre-trained classification model is trained, constrained by a classification loss function. Training is completed when the number of training iterations reaches the third maximum preset number of iterations, resulting in the classification model. The third maximum preset number of iterations is set to 500. The pre-trained classification model is a ResNet18 model trained on ImageNet, and the classification loss function is the cross-entropy loss function, expressed as:
[0118] ;
[0119] Where M is the number of categories, It is an indicator variable; it takes the value 1 when the true class of variable o is equal to c, and 0 otherwise. It is the predicted probability that sample o belongs to category c.
[0120] It should be noted that the target object region is input into the classification model, and a confidence threshold is set. When the maximum class probability output by the classification model is lower than the confidence threshold When this happens, the target object is classified as an unknown category.
[0121] In step S15, the target object region and the category label are input into the underwater sonar simulation image generation module to obtain a simulated sonar image of the target object.
[0122] Specifically, the process involves determining whether the category label is a target category that has already been trained. If the category label is a target category that has already been trained, the embedding vectors of the target object region and the category label are input into the Conditional Generative Adversarial Network (CGN) to generate a simulated sonar image of the target object. If the category label is a target category that has not been trained, the target object region is input into the style encoding module to obtain a style encoding vector. The target object region and the style encoding vector are then input into the CGN to generate a simulated sonar image of the target object. The CGN is a CycleGAN-based model that includes a generator and a discriminator. In CycleGAN, the generator receives two inputs: a segmented object image and a conditional vector. The conditional vector can be the embedding of the category label (for known categories) or a style encoding vector (for unknown categories). The conditional vector is injected into each layer of the generator through conditional normalization instances. For unknown categories, the style encoding is obtained by weighted averaging of known object features, and the generated image will have a mixture of these styles.
[0123] It should be noted that the training process of the conditional generative adversarial network includes: acquiring a multi-source domain training dataset of known object categories, a real sonar training dataset, and category labels. The multi-source domain training dataset includes optical training images, remote sensing training images, and simulator training images of the known object categories. The real sonar training dataset includes real sonar images and real sonar segmentation masks. The known object categories represent a subset of target object categories, while the remaining target object categories are used as target categories not involved in training to verify the model's generalization ability.
[0124] Further, based on the real sonar segmentation mask, a first target object region set is extracted from the real sonar image; the multi-source domain training dataset is input into the pre-trained segmentation model to obtain a segmentation mask for the multi-source domain training dataset; based on the segmentation mask for the multi-source domain training dataset, a second target object region set is extracted from the multi-source domain training dataset image. Based on the first target object region set, the second target object region set, and the category label, the conditional generative adversarial network is trained. The generator part is constrained by a generation loss function, and the discriminator part is constrained by a discriminative loss function. Training is completed when the number of training iterations reaches the fourth maximum preset number of rounds, resulting in a trained conditional generative adversarial network. The fourth maximum preset number of rounds is set to 500 rounds.
[0125] The generation loss function is expressed as:
[0126] ;
[0127] in, It is to combat losses. It is a cycle-consistent loss. It's a loss of identity. It is a loss due to physical constraints. , , It's a weighting coefficient, set , , To avoid the physical constraint loss interfering with model convergence in the early stages of training, a linear warm-up strategy is used for adjustment. , is represented as:
[0128] ;
[0129] in, This represents the current training iteration number. This represents the total number of iterations in the warm-up phase. The base value is 1.0. back, Keep .
[0130] Specifically, the adversarial loss, cycle consistency loss, and identity loss are calculated according to the definition of the CycleGAN model. The adversarial loss is expressed as:
[0131] ;
[0132] The discriminant loss function is expressed as follows:
[0133] ;
[0134] Where x is the real sonar image, z is the input segmentation mask and conditional vector, G is the generator, and D is the discriminator;
[0135] The cycle-consistent loss is expressed as:
[0136] ;
[0137] The loss of identity is represented as:
[0138] ;
[0139] Where x is the distribution of the actual sonar image. The samples are c, where c is the condition vector. It is another conditional vector.
[0140] It is worth noting that the aforementioned physical constraint loss The calculation process includes: acquiring a sample from the multi-source domain training dataset as an input sample; and obtaining the theoretical sonar target intensity of the input sample through a sonar target approximation module. Specifically, refer to... Figure 2 The geometric parameters of the input sample are obtained by performing a geometric extraction operation on the segmentation mask of the input sample. The geometric parameters include the equivalent area, perimeter, minimum bounding rectangle size, and major and minor axes of the minimum bounding ellipse of the input sample. Each geometric parameter is obtained as follows (using the OpenCV library in Python): Equivalent area A: The number of foreground pixels is calculated using the cv2.countNonZero() function; Perimeter P: The contour is extracted using the cv2.findContours() function, and the perimeter of the contour is calculated using the cv2.arcLength() function; Minimum bounding rectangle: The rotated rectangle is obtained using the cv2.minAreaRect() function, and its length L and width W (taking the long and short sides) are extracted; Minimum bounding ellipse: The ellipse is fitted using the cv2.fitEllipse() function, and the returned ellipse contains the lengths of the major and minor axes, which are divided by 2 to obtain the semi-axis lengths a and b.
[0141] Furthermore, based on the geometric parameters, the input sample is approximated as a simple geometric shape or a combination of simple geometric shapes according to a preset rule. Based on the perpendicular incidence assumption, the theoretical sonar target intensity of the input sample is calculated. The simple geometric shapes include spheres, cylinders, rectangular plates, circular plates, and ellipsoids. The preset rule is: ① If the equivalent circularity of the input sample is... If the aspect ratio of the input sample is 1, then it is approximated as a sphere; ② If the aspect ratio of the input sample is 1, then it is approximated as a sphere. And the equivalent circularity of the input sample If the rectangularity of the input sample is..., then it is approximated as a cylinder; ③ If the rectangularity of the input sample... If the ellipticity of the input sample is such that it is approximated as a flat plate; ④ If the ellipticity of the input sample is such that... Between 1.2 and 3, the object is approximated as an ellipsoid; ⑤ If the input sample does not conform to any of the above rules, the target object is decomposed into multiple sub-regions that conform to the above rules, each sub-region approximating a simple geometric shape. If any simple geometric shape rule is met, the corresponding simple geometric shape type and its parameters are returned. Otherwise, recursive decomposition is performed: the convex hull of the contour is calculated using the cv2.convexHull() function, which is the smallest convex polygon containing all points of the contour. Convex defects are calculated using cv2.convexityDefects(), which is the concave region between the contour and the convex hull. The convex part is obtained by the convex hull index, which is the continuous point sequence on the convex hull. The deepest point in each defect is extracted from the convex defects, and these points are the candidate concave points. From all concave points, the Euclidean distance between each pair of concave points is calculated (excluding invalid point pairs: ① the distance between the two points is less than 10% of the contour perimeter, ② the line connecting the two points intersects the contour more than twice), and the pair with the largest distance is selected from the valid point pairs as the split point. The line connecting the two concave points is used as a dividing line to divide the input sample into two sub-regions using a segmentation mask. For each sub-region, the same decomposition process is recursively called until all sub-regions conform to simple geometric shape rules (the geometric parameters of all sub-regions are repeatedly calculated and compared with preset rules). Finally, the input sample is represented as a combination of these simple geometric shapes.
[0142] Furthermore, the input sample is input into the generator of the conditional generative adversarial network to obtain generated samples, and the physical constraint loss is expressed as:
[0143] ;
[0144] in, This represents the average pixel value of the generated samples. and These are the minimum and maximum values of the theoretical intensity range of the target object in the sonar image, respectively. It is the weighting coefficient of the physical constraints, with a value ranging from 0.1 to 0.5.
[0145] It should be noted that, based on the geometric approximation results and under the assumption of perpendicular incidence, the target intensity of each simple geometric shape is calculated as follows: assuming the incident sound intensity is 1 (because target intensity is a relative value, it is usually referenced to the incident sound intensity at 1 meter), the theoretical target intensity calculation formula for the approximate spherical geometry is:
[0146] ;
[0147] Where r is the radius of the sphere, r is determined by... Calculated;
[0148] The theoretical formula for calculating the target strength of an approximate cylindrical geometry (axial incidence) is as follows:
[0149] ;
[0150] Where r is the radius of the cylinder, which can be obtained through... The calculations show that L is the length of the cylinder, and λ is the wavelength of the sound wave, which is taken as 0.015m (assuming a frequency of 100kHz and a sound speed of 1500m / s). The wavelength λ of the sound wave is determined by the sound speed c and the frequency f. .
[0151] The theoretical formula for calculating the target strength of a plate with approximate geometry (normal incidence) is as follows:
[0152] ;
[0153] Where A is the area of the flat plate and R is the reflection coefficient, which is taken as the empirical value of the reflection coefficient of metal, i.e., 0.9.
[0154] The theoretical target strength calculation formula for the approximate geometry of an ellipsoid is as follows:
[0155] ;
[0156] Where a is half of the major axis and b is half of the minor axis.
[0157] For combined geometries, calculate the total target strength using the following steps: For each simple geometric part, calculate its target strength value. Convert each target intensity value into a linear scattered acoustic intensity value. The total scattered sound intensity value is obtained by summing all the linear scattered sound intensity values. ; Calculate the overall target intensity .
[0158] It is worth noting that inputting the target object region into the style encoding module to obtain the style encoding vector includes: inputting the target object region into the style feature extractor to obtain the depth visual features of the target object region; the style feature extractor is a pre-trained convolutional neural network with the last fully connected layer removed, and the pre-trained convolutional neural network is a ResNet18 model trained on ImageNet.
[0159] Furthermore, an object visual feature library is acquired. The construction process of the object visual feature library includes: acquiring a material feature training dataset; the material feature training dataset is obtained by sonar placing standardized geometric shape samples of different base materials in a controlled underwater environment; the controlled underwater environment refers to a large anechoic pool to eliminate complex multipath reflections and environmental interference, recording and fixing all imaging parameters such as water temperature, salinity, sonar frequency, transmission power, distance, and incident angle for each experiment; the standardized geometric shape samples include spheres, cubes, cylinders, and plates; the base materials include steel, aluminum, copper, wood, plastic, concrete, and granite. The material feature training dataset is input into the style feature extractor to obtain depth visual features; the depth visual features encode the comprehensive visual attributes of the target, such as shape, texture, color, and structure; the depth visual features are subjected to global average pooling to obtain the style feature vector of the material feature training data; the style feature vector and category label of each training sample of the material feature training data are stored in the object visual feature library. The similarity between the depth visual features of the target object region and the depth visual features of all objects in the object visual feature library is calculated to obtain the similarity calculation result.
[0160] The similarity calculation uses a cosine distance metric, expressed as:
[0161] ;
[0162] in, It is the depth visual feature of the target object region. The depth visual features are those of the i-th category object in the object visual feature library. Based on the similarity calculation results, depth visual features of a preset number of similar objects are obtained from the object visual feature library in descending order of similarity. The preset number k is set to 5 and dynamically adjusted according to the uniqueness of the target object's features. If the similarity between the target object and the obtained k similar objects is very high (>0.8), the value of k is decreased; conversely, if the similarity is generally low, the value of k is increased. A weighted average is performed on the depth visual features of the retrieved k known objects to obtain the style encoding vector of the target object.
[0163] The style encoding vector is represented as follows:
[0164] ;
[0165] in, It is the generated style encoding vector, and k is the preset similarity number. It is the depth visual feature of the j-th similar object. These are the corresponding weights, and =1.
[0166] S16, construct the background database and the simulated sonar object database.
[0167] Specifically, the background database is collected by sonar sensors in real aquatic environments, requiring that the environment does not contain any target objects and only contains pure underwater background information (sonar images with foregrounds are manually selected). The sonar equipment parameters used are: a multibeam forward-looking sonar, acquiring data at a frequency of 3.0 MHz, with a field of view of 30°. The sonar has a 15° elevation angle and consists of 128 beams. The elevation angle is set between 15° and 30°, and the data acquisition is performed by a remotely operated underwater vehicle (ROV) carrying the sonar equipment. Data acquisition is conducted in various typical hydrological and geological environments, including sandy, muddy, rocky, and vegetated substrates. Data is repeatedly acquired at different times and under different hydrological conditions in the same area to obtain the diversity of background texture and noise caused by natural variations. The processed, clean background sonar images are stored in the background database. The simulated sonar object database adopts a scalable storage structure, supporting rapid retrieval based on object attributes, geometric features, and visual features. In step S15, each time a target object region is input into the trained conditional generative adversarial network, the generated simulated sonar image is automatically captured and saved, along with its category label and segmentation mask.
[0168] S18, randomly select a background sonar image from the background database, and combine the simulated sonar image of the target object and a preset number of simulated sonar images taken from the simulated sonar object database with the background sonar image to obtain a complete sonar simulation image.
[0169] Specifically, a set of simulated sonar images is randomly inserted into the background sonar image to obtain a first synthesized sonar image. The set of simulated sonar images includes simulated sonar images of the target object and a preset number of simulated sonar images retrieved from the simulated sonar object database; the preset number is generally no more than 10, but can be 0, without strict limitation. The category label and position of each object in the simulated sonar image set in the first synthesized sonar image are obtained, and it is checked whether there is pixel-level overlap between any two objects. If overlap exists, a preset overlap processing rule is applied according to the category relationship of the objects. This process is an iterative optimization process. The preset overlap processing rule is as follows: ① For overlap processing between objects of different categories: When overlapping objects belong to different categories (all objects marked as "unknown" are treated as different categories), one of the objects is randomly selected as the "retention benchmark", and the pixel areas overlapping with the benchmark object in the other object images are cleared. The proportion of the cleared area to the total area of the object is calculated. Based on this ratio, different strategies are adopted: If the ratio is less than the preset overlap threshold of 20%, only the overlapping pixels of the object on the sonar image are cleared, and its corresponding segmentation mask remains unchanged; if the ratio is greater than the preset overlap threshold, the object is randomly inserted into another position with a preset probability (p, set to 0.7), and the object is deleted with a probability of 1-p. ② For the overlapping processing of multiple objects of the same category: only the sonar pixel values of the objects placed later in the overlapping area are cleared, and the pixel values of the objects placed first are retained. This operation does not trigger re-insertion or deletion, and the segmentation mask is not changed, thus forming a natural composite target region on the image. Whenever an object insertion operation occurs in the above process, an overlap check step is required to judge and process the overlap of the new layout again until the overlap between all objects is compliant. Finally, a sonar composite image with a reasonable layout of all objects is output, which is called the second sonar composite image.
[0170] Furthermore, the brightness difference between each object in the simulated sonar image set and the background sonar image is calculated, and brightness adjustment processing is performed according to the brightness adjustment ratio. The brightness value of the object in the simulated sonar image set is multiplied by the corresponding brightness adjustment ratio to obtain the third sonar composite image.
[0171] The calculation of the brightness adjustment ratio is expressed as follows:
[0172] ;
[0173] in, It is the brightness adjustment ratio of the i-th object in the simulated sonar image set. It is the average pixel value of the background sonar image. This is the average pixel value of the i-th object in the simulated sonar image set. The segmentation mask for each object in the simulated sonar image set is obtained, Gaussian blurring is performed, and an alpha fusion mask for each object is generated. The alpha fusion mask is then used to perform a weighted fusion of the object image and the background image.
[0174] ;
[0175] in, Let be the average pixel value of the i-th object in the simulated sonar image set after Gaussian blurring. It is the average pixel value of the i-th object in the simulated sonar image set. It is the average pixel value of the background sonar image. This is the alpha value after Gaussian blurring, ranging from 0 to 1. The fused image is output as the final complete sonar simulation image. The complete sonar simulation image contains pixel-level annotation information.
[0176] In summary, this invention provides an underwater sonar image data augmentation method, comprising: acquiring a source image containing a target object; inputting the source image into a pre-trained segmentation model to obtain a segmentation mask for the target object; extracting a target object region from the source image based on the segmentation mask; inputting the target object region into a pre-trained classification model to obtain a category label for the target object; inputting the target object region and the category label into an underwater sonar simulation image generation module to obtain a simulated sonar image of the target object; acquiring a background database and a simulated sonar object database; wherein the background database consists of background sonar images without foreground targets collected by sonar sensors in a real aquatic environment; storing the simulated sonar image, category label, and segmentation mask of the target object into the simulated sonar object database; randomly selecting a background sonar image from the background database; and performing a composite processing of the simulated sonar image of the target object and a predetermined number of simulated sonar images retrieved from the simulated sonar object database with the background sonar image to obtain a complete simulated sonar image.
[0177] This invention provides a method for enhancing underwater sonar image data to address the problem of poor generalization performance in simulated sonar image generation.
[0178] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A method for enhancing underwater sonar image data, characterized in that, include: Get the source image containing the target object; The source image is input into a pre-trained segmentation model to obtain the segmentation mask of the target object; Based on the segmentation mask, the target object region is extracted from the source image; The target object region is input into a pre-trained classification model to obtain the category label of the target object; The target object region and the category label are input into the underwater sonar simulation image generation module to obtain a simulated sonar image of the target object; A background database and a simulated sonar object database are acquired, wherein the background database consists of background sonar images without foreground targets acquired by sonar sensors in a real aquatic environment; The simulated sonar image, category label, and segmentation mask of the target object are stored in the simulated sonar object database; A background sonar image is randomly selected from the background database. The simulated sonar image of the target object and a preset number of simulated sonar images taken from the simulated sonar object database are combined with the background sonar image to obtain a complete simulated sonar image.
2. The underwater sonar image data enhancement method according to claim 1, characterized in that, The training process of the segmentation model includes: Obtain a first segmentation training dataset and a second segmentation training dataset, wherein the first segmentation training dataset refers to training images and segmentation masks in the optical domain, and the second segmentation training dataset refers to training images and segmentation masks in the remote sensing domain and the simulator domain; Based on the first segmentation training dataset, the first neural network model is trained, and the first segmentation loss function is used for constraint. The training is completed when the number of training times reaches the first maximum preset number of rounds, and the first segmentation model is obtained. Based on the second segmentation training dataset, the first segmentation model is trained, and the second segmentation loss function is used for constraint. The training is completed when the number of training times reaches the second maximum preset number of rounds, and the second segmentation model is obtained. The first neural network model is a U-Net model, and the first and second segmentation loss functions are composed of binary cross-entropy loss and Dice loss, as expressed in Equation 1: ; (Equation 1) Where T is the number of samples in the batch. It's a real label. It is the result of model prediction. yes and The number of intersections, i.e., the number of pixels with the same value. yes The number of elements in the middle. yes The number of elements in the text.
3. The underwater sonar image data enhancement method according to claim 2, characterized in that, Before obtaining the first segmentation training dataset and the second segmentation training dataset, the method further includes: Obtain the original datasets for the optical domain, the remote sensing domain, and the simulator domain. The original dataset for the optical domain includes optical domain images and optical domain instance segmentation labels. The original dataset for the remote sensing domain includes remote sensing domain images and remote sensing domain bounding box labels. The original dataset for the simulator domain includes simulator domain images and simulator domain segmentation masks. The original dataset in the optical domain is cropped for each instance based on the instance segmentation label in the optical domain to obtain the processed image in the optical domain and the segmentation mask in the optical domain. The original dataset in the remote sensing domain is used to extract target regions based on the target bounding box labels in the remote sensing domain. Morphological processing is then performed on the target regions to obtain the processed image and the segmentation mask in the remote sensing domain. The optical domain processed image and the optical domain segmentation mask are used as the first segmentation training dataset, and the remote sensing domain processed image and the remote sensing domain segmentation mask are merged with the original simulator domain dataset to form the second segmentation training dataset.
4. The underwater sonar image data enhancement method according to claim 1, characterized in that, The training process of the classification model includes: Obtain a classification training dataset, wherein the classification training dataset includes optical images, remote sensing images, and sonar simulator images corresponding to all categories of the real sonar dataset; Based on the classification training dataset, the pre-trained classification model is trained, and a classification loss function is used for constraint. The training is completed when the number of training times reaches the third maximum preset number of rounds, and the classification model is obtained. The pre-trained classification model is a ResNet18 model trained on ImageNet, and the classification loss function is the cross-entropy loss function, expressed as Equation 2: (Equation 2) Where M is the number of categories, It is an indicator variable; it takes the value 1 when the true class of variable o is equal to c, and 0 otherwise. It is the predicted probability that sample o belongs to category c.
5. The underwater sonar image data enhancement method according to claim 1, characterized in that, The step of inputting the target object region and the category label into the underwater sonar simulation image generation module to obtain a simulated sonar image of the target object includes: Determine whether the category label is a target category that has already participated in training; If the category label is a target category that has been trained, then the embedding vectors of the target object region and the category label are input into the conditional generative adversarial network to generate a simulated sonar image of the target object; If the category label is a target category that has not participated in training, then the target object region is input into the style encoding module to obtain the style encoding vector, and the target object region and the style encoding vector are input into the conditional generative adversarial network to generate a simulated sonar image of the target object. The conditional generative adversarial network is a CycleGAN-based model that includes a generator and a discriminator.
6. The underwater sonar image data enhancement method according to claim 5, characterized in that, The training process of the conditional generative adversarial network includes: Obtain a multi-source domain training dataset, a real sonar training dataset, and a category label set for known category objects. The multi-source domain training dataset includes optical training images, remote sensing training images, and simulator training images of the known category objects. The real sonar training dataset includes real sonar images and real sonar segmentation masks. Based on the real sonar segmentation mask, a first target object region set is extracted from the real sonar image; Input the multi-source domain training dataset into the pre-trained segmentation model to obtain the segmentation mask of the multi-source domain training dataset; Based on the segmentation mask of the multi-source domain training dataset, extract the second target object region set from the image of the multi-source domain training dataset; Based on the first target object region set, the second target object region set, and the category label set, the conditional generative adversarial network is trained. The generator part is constrained by the generation loss function, and the discriminator part is constrained by the discrimination loss function. The training is completed when the number of training times reaches the fourth maximum preset number of rounds, and the trained conditional generative adversarial network is obtained. The generation loss function is expressed as Equation 3: ; (Equation 3) in, It is to combat losses. It is a cycle-consistent loss. It's a loss of identity. It is a loss due to physical constraints. , , These are weighting coefficients. The adversarial loss, cycle consistency loss, and identity loss are calculated according to the definition of the CycleGAN model.
7. The underwater sonar image data enhancement method according to claim 5, characterized in that, The target object region is input into the style encoding module to obtain a style encoding vector, including: The target object region is input into a style feature extractor to obtain the depth visual features of the target object region. The style feature extractor is a pre-trained convolutional neural network with the last fully connected layer removed. The pre-trained convolutional neural network is a ResNet18 model trained on ImageNet. Obtain the object visual feature library; The similarity between the depth visual features of the target object region and the depth visual features of all objects in the object visual feature library is calculated to obtain the similarity calculation result. The similarity calculation uses a cosine distance metric, expressed as Equation 4: (Equation 4) in, It is the depth visual feature of the target object region. It is the depth visual feature of the i-th category object in the object visual feature library; Based on the similarity calculation results, the depth visual features of a preset number of similar objects are obtained from the object visual feature library in descending order of similarity, and a weighted average is performed to obtain the style encoding vector of the target object. The style encoding vector is represented by Equation 5: (Equation 5) in, It is the generated style encoding vector, and k is the preset similarity number. It is the depth visual feature of the j-th similar object. These are the corresponding weights, and =1.
8. The underwater sonar image data enhancement method according to claim 7, characterized in that, The construction process of the object visual feature library includes: Obtain the material feature training dataset; The material feature training dataset is input into the style feature extractor to obtain deep visual features; The depth visual features are subjected to global average pooling to obtain the style feature vector of the material feature training data; Build an object visual feature library; The style feature vector and category label of each training sample of the material feature training data are stored in the object visual feature library.
9. The underwater sonar image data enhancement method according to claim 6, characterized in that, The calculation process for the physical constraint loss includes: Obtain a sample from the multi-source domain training dataset as the input sample; A geometric extraction operation is performed on the segmentation mask of the input sample to obtain the geometric parameters of the input sample, wherein the geometric parameters include the equivalent area, perimeter, minimum bounding rectangle size, and major and minor axes of the minimum bounding ellipse of the input sample; Based on the geometric parameters, the input sample is approximated as a simple geometric shape or a combination of simple geometric shapes according to a preset rule. Based on the vertical incidence assumption, the theoretical sonar target intensity of the input sample is calculated. The simple geometric shapes include spheres, cylinders, rectangular plates, circular plates, and ellipsoids. The input sample is fed into the generator of the conditional generative adversarial network to obtain the generated sample; The physical constraint loss is expressed as Equation 6: (Equation 6) in, This represents the average pixel value of the generated samples. and These are the minimum and maximum values of the theoretical intensity range of the target object in the sonar image, respectively. It is the weighting coefficient of the physical constraints, with a value ranging from 0.1 to 0.
5.
10. The underwater sonar image data enhancement method according to claim 1, characterized in that, The simulated sonar image of the target object and a predetermined number of simulated sonar images retrieved from the simulated sonar object database are combined with the background sonar image to obtain a complete simulated sonar image, including: The simulated sonar image set is randomly inserted into the background sonar image to obtain the first sonar composite image. The simulated sonar image set includes the simulated sonar image of the target object and a preset number of simulated sonar images taken from the simulated sonar object database. Obtain the category label and position of each object in the simulated sonar image set in the first sonar composite image, and obtain the second sonar composite image based on a preset overlap processing rule; The brightness difference between each object in the simulated sonar image set and the background sonar image is calculated, and brightness adjustment processing is performed according to the brightness adjustment ratio to obtain the third sonar composite image; The calculation of the brightness adjustment ratio is expressed as Equation 7: (Equation 7) in, It is the brightness adjustment ratio of the i-th object in the simulated sonar image set. It is the average pixel value of the background sonar image. It is the average pixel value of the i-th object in the simulated sonar image set; Gaussian filtering is used to perform alpha fusion processing on each object in the third sonar composite image to obtain a complete sonar simulation image.