Underwater hull fouling marking optimization method based on multi-modal large model guidance

Abstract

The invention relates to an underwater ship fouling marking optimization method based on multi-modal large model guidance, which comprises the following steps of: obtaining an existing underwater fouling data set, and performing channel conversion on a marking file in the underwater fouling data set to train a segmentation model; capturing an output probability score corresponding to the test image at a solution wharf of the segmentation model by using the weight file obtained by training to obtain a fouling and background probability matrix; generating a suggested area and a suggested point set, carrying out manual point selection, and judging an ignored area; manually selected points serve as prompt points for interactive guide updating of the large model, a new segmentation result is obtained through the large model and serves as a new annotation file, the new annotation file and the original annotation file are compared to judge whether iteration is ended or not, and if iteration is not ended, the segmentation model is retrained based on the new annotation file. Compared with the prior art, the method can effectively improve the marking precision of the underwater stained image, is suitable for the underwater image segmentation task with a complex scene, reduces the manual workload, and improves the efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an underwater hull fouling annotation optimization method guided by a multimodal large model. Background Art

[0002] Underwater hull fouling detection is a critical component of ship operations, maintenance, and environmental protection. Hull fouling not only directly leads to a significant decrease in fuel efficiency but can also mask structural issues such as metal fatigue or corrosion, posing significant safety risks. Furthermore, biofouling in the fouled area can have long-term adverse effects on marine ecosystems. Therefore, efficient and accurate fouling detection is of strategic importance for reducing operating costs, improving navigation safety, and protecting the marine environment.

[0003] The labeling of underwater data faces significant challenges due to small-scale datasets, domain gaps, and initial rough labeling. First, the complexity and variability of the underwater environment makes it difficult for manual labeling to achieve fine-grained accuracy, resulting in initial labeling results often having problems with blurred boundaries and error accumulation. Second, manual labeling not only requires a lot of professional time and energy, but also makes it difficult to achieve an ideal level of labeling quality and consistency. These problems are further exacerbated when the dataset is small, resulting in poor performance of the model in cross-scenario applications with domain gaps, and the reliance on rough labeling limits its applicability in fine-grained segmentation tasks.

[0004] In recent years, segmentation models based on the Transformer architecture have become an important technological breakthrough in the field of computer vision due to their excellent global feature modeling capabilities and cross-scenario application performance. Their performance faces many limitations in small-scale datasets, domain differences, and rough annotation scenarios. The Transformer model relies on large-scale, high-quality annotated datasets to ensure that the self-attention mechanism can fully capture the global relationship between features. However, the initial rough annotations are ambiguous in expressing boundary information, making it difficult for the model to effectively distinguish fine-grained features of adjacent areas in complex underwater scenes. In addition, the model's tendency to over-focus on large-scale features when trained on small-scale datasets further weakens its ability to segment details of contaminated areas, making it difficult to cope with complex task requirements in specific fields.

[0005] In recent years, multimodal large-scale SAM models have demonstrated remarkable potential in general image segmentation tasks due to their outstanding zero-shot segmentation capabilities and interactive prompting advantages. They are particularly widely applicable in diverse scenarios such as underwater and open environments. However, the general capabilities of SAM models are significantly limited in small-scale datasets, with significant domain differences and coarse annotation. Therefore, the prior art proposes combining the Transformer model with multimodal large-scale SAM models to improve image segmentation performance. For example, CN118298169A discloses an image segmentation method based on large-scale guidance, which includes: constructing a large-scale guided segmentation network model; wherein the large-scale guided segmentation network model includes a large-scale optimization module and a multi-task image processing module; the large-scale optimization module is used to segment and filter the input image to obtain a segmentation mask that is close to the real scene label as guidance information; the multi-task image processing module is used to extract feature maps of the input image and integrate the guidance information to achieve image segmentation; the constructed large-scale guided segmentation network model is trained using a set of images of a preset type; and the trained large-scale guided segmentation network model is used to obtain the image segmentation result of the image to be segmented.

[0006] However, this method is not suitable for underwater image processing. First, the lack of domain knowledge in underwater scenes, such as lighting complexity and the diversity of fouling morphology, makes it difficult for the model to effectively capture fine-grained features. Furthermore, although its zero-shot segmentation capability is highly dependent on the quality of the hints, the segmentation results generated by the model are often limited to a coarse-grained level due to the blurred boundaries and complex target areas caused by the initial rough annotation, making it difficult to meet the high-precision requirements for boundary details and target microstructure in underwater hull fouling detection. Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide an underwater hull fouling annotation optimization method based on multimodal large model guidance. It combines the proprietary segmentation model Segformer with the large model SAM2, captures the domain characteristics of the underwater scene through Segformer, and combines the candidate area generation capability of SAM2, effectively utilizes active learning technology to iteratively optimize the annotation results, significantly improves the segmentation accuracy and annotation consistency, and while reducing the annotation cost, enhances the model's feature extraction and detail depiction capabilities in complex environments, providing a new technical path and engineering practice value for underwater data optimization and hull fouling detection.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to a first aspect of the present invention, a method for optimizing underwater hull fouling annotation based on multimodal large model guidance is provided, the method comprising the following steps:

[0010] Step 1) obtaining an existing underwater contamination dataset, wherein the underwater contamination dataset includes original underwater images and original annotation files;

[0011] Step 2) performing channel conversion on the original annotation file by mapping, converting the three-channel annotation file into a single-channel one;

[0012] Step 3) using the channel-converted annotated file and the original underwater image to train the segmentation model;

[0013] Step 4) Using the weight file trained in step 3), the original underwater image is used as a test image, and the output probability score corresponding to each test image is captured in the decoding head of the segmentation model, and converted into a contamination probability matrix and a background probability matrix outputted by the test image;

[0014] Step 5) generating a suggested region and a suggested point set based on the contamination probability matrix and the background probability matrix;

[0015] Step 6) Manually select points based on the proposed area and the proposed point set and identify the ignored area;

[0016] Step 7) The points selected in step 6) are used as prompt points for interactive guidance update of the large model, and are input into the large model together with the corresponding original underwater image to obtain a new segmentation result;

[0017] Step 8) Use the new segmentation result as a new annotation file and compare the new annotation file with the original annotation file. If the comparison result meets the preset conditions, the iteration ends and the new annotation file is output as the final lossless annotation of the underwater hull. Otherwise, return to step 3) and retrain the segmentation model based on the new annotation file and the original underwater image for the next iteration.

[0018] As a preferred technical solution, in step 2), the white area in the annotation file is mapped to label 1, indicating contamination, and the black area is mapped to 0, indicating background.

[0019] As a preferred technical solution, the step 4) includes the following steps:

[0020] Step 4-1) The decoding head of the segmentation model captures the predicted score matrix for each test image, and inputs the predicted score matrix into the Softmax function to convert it into the output probability score corresponding to each pixel, thereby obtaining the total probability matrix of the defacement and background.

[0021] Step 4-2) extracts the contamination probability matrix and the background probability matrix from the total probability matrix through a tensor split statement.

[0022] As a preferred technical solution, the step 5) includes the following steps:

[0023] Step 5-1) Extract pixels with a probability greater than a first threshold from the background probability matrix. The set of pixels constitutes the background proposal region. Cluster the pixels in the background proposal region based on their coordinate positions to obtain multiple clusters. The center of each cluster is used as the pixel representing the background, set to label 0, and used as a negative sample. The point set consisting of the negative samples is the background proposal point set.

[0024] Step 5-2) Extract the pixel points with a probability greater than the second threshold in the contamination probability matrix, and the set of pixel points is the contamination suggestion area; cluster the pixel points in the contamination suggestion area based on the coordinate position to obtain multiple clusters, and take the center of each cluster as the pixel point representing the background, set it as label 1, and serve as a positive sample. The point set consisting of the positive samples is the contamination suggestion point set.

[0025] As a preferred technical solution, the first threshold and the second threshold are determined based on an adaptive threshold method, and the steps are as follows:

[0026] Convert the background probability matrix and the contamination probability matrix into probability distribution histograms respectively, and calculate the gradient and difference of the histograms, where the gradient is the speed of change of the density value corresponding to the probability value with respect to the probability value, and the difference is the change of the density value between adjacent probability values, that is, the absolute difference between each pair of adjacent probability values;

[0027] According to the changes in gradient and difference, the middle range of probability distribution is screened;

[0028] The right endpoint of the middle range obtained based on the background probability matrix is ​​used as the first threshold, and the left endpoint of the middle range obtained based on the stain probability matrix is ​​used as the second threshold.

[0029] As a preferred technical solution, the step 6) includes the following steps:

[0030] Step 6-1) Visualize the proposed point set on the original underwater image, and discard the label points in the proposed point set that do not match the actual label based on the visualization result to obtain a new proposed point set;

[0031] Step 6-2) Based on the new set of suggested points, manually select a first preset number of points in the suggested area and set corresponding labels as a first prompt point set;

[0032] Step 6-3) manually screening the ignored areas that were not classified as suggested areas in step 5) but actually belong to the suggested areas. If ignored areas exist, a second preset number of points are selected from the ignored areas and corresponding labels are set as the second prompt point set. Otherwise, a third preset number of points are selected from the suggested areas, the selected points being different from the points in the first prompt point set, and corresponding labels are set as the second prompt point set.

[0033] Step 6-4) summarizes the first cue point set and the second cue point set as cue points for interactively guiding the update of the large model in step 7).

[0034] As a preferred technical solution, in step 8), the judgment condition for whether the iteration is completed is:

[0035] Compare the new segmentation results with the original annotation file and calculate the segmentation evaluation indicators, which include the average intersection-over-union ratio, the average Dice coefficient and the average pixel accuracy of each category;

[0036] The incremental change rate of the segmentation evaluation index of the current iteration relative to the previous iteration is calculated. If the incremental change rate of each segmentation evaluation index is lower than the third threshold for a preset number of consecutive times, the preset condition is met and the iteration is terminated. Otherwise, the next iteration is performed.

[0037] As a preferred technical solution, the third threshold is set using the percentile method and is set to the 90th percentile of the incremental change rate of the segmentation evaluation indicator in the historical iteration.

[0038] As an optimal technical solution, the segmentation model adopts the Segformer segmentation model.

[0039] As a preferred technical solution, the large model adopts the SAM2 model.

[0040] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0041] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention uses the segmentation model and the large model to work together, introduces active learning technology, and achieves significant optimization of the annotation process. Unlike the traditional method of relying on manual direct annotation of the stained area, the present invention transfers the core link of manual participation to the prompt point selection stage, avoiding the huge workload of manual fine annotation of the entire image from beginning to end, and with the help of the generated suggested areas, significantly reduces the human annotation error and significantly improves the annotation accuracy. Afterwards, the large model automatically completes the segmentation calculation of the complex area, replacing the tedious manual pixel-level description. By reducing the workload of manual intervention, it effectively improves the efficiency and reliability of the data set annotation, and provides higher-quality annotation data for subsequent model training.

[0044] (2) In the process of generating prompt information, the present invention fully integrates domain-specific knowledge and original data. First, through the preliminary prediction of the contamination and background suggestion areas by the segmentation model, the contamination probability matrix and background probability matrix predicted by the model are used to assist in manually identifying the areas where the model is uncertain (fuzzy boundaries, complex target areas) and potential error areas (ignored areas), providing a scientific basis for the selection of prompt points. Secondly, the manual combination of domain understanding and common sense judgment of the ignored areas actively selects supplementary prompt points, avoiding blind labeling on the entire image, and accurately focusing limited manpower on the details that are most difficult for the model to grasp, thereby improving the labeling accuracy and efficiency. This two-layer verification mechanism also significantly improves the accuracy of prompt point selection, and enables the large model to accurately understand the boundary information and microstructure features indicated by these prompt points, generating new segmentation results that are more refined and more consistent with the real structure. Through this optimization process, the labeling accuracy has been further improved, laying a solid foundation for the training of high-quality segmentation models.

[0045] (3) Each iteration of the present invention retrains the segmentation model based on the updated and more accurate annotation file. As the annotation quality continues to improve, the performance of the trained segmentation model is gradually enhanced. The improved model can generate a more accurate and focused uncertainty map, thereby guiding the next round of manual and large model optimization at a more subtle level. This iterative mechanism allows the model to gradually learn high-quality annotation knowledge generated by the collaboration of large models and humans during the training process, which is more in line with the complex lighting and morphological diversity of underwater pollution. Even if the initial annotation is rough or the domain knowledge is not embedded enough, through multiple rounds of optimization, it is possible to eventually generate annotation data that meets the high-precision requirements, indirectly making up for the lack of domain knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the method flow of the present invention;

[0047] Figure 2 Flowchart of data partitioning and preprocessing in the present invention;

[0048] Figure 3 This is a flowchart of the initial training of the Segformer segmentation model in the present invention;

[0049] Figure 4 A flowchart of model prediction and probability matrix generation in the present invention;

[0050] Figure 5 Flowchart of manual point selection and ignored area selection in the present invention;

[0051] Figure 6 Flowchart for generating new segmentation results for the SAM2 large model in the present invention;

[0052] Figure 7 Schematic diagram of an original annotation file and its corresponding original underwater image in an embodiment of the present invention, wherein (7a) is the original annotation file and (7b) is the original underwater image;

[0053] Figure 8 This is a schematic diagram of the initial prediction results generated by the Segformer model in an embodiment of the present invention;

[0054] Figure 9 In the embodiment of the present invention, Figure 8 The probability distribution histogram generated based on the prediction results, where (9a) is the probability distribution histogram corresponding to the background probability matrix, and (9b) is the probability distribution histogram corresponding to the contamination probability matrix;

[0055] Figure 10 Schematic diagram of the recommended point set and recommended area generated in an embodiment of the present invention;

[0056] Figure 11 A visualization diagram of prompt points in an embodiment of the present invention;

[0057] Figure 12 This is a schematic diagram of a new segmentation result generated in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0059] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0060] In response to the challenges of small dataset size, domain gaps, and rough initial annotation, this embodiment provides an underwater hull fouling annotation optimization method guided by a multimodal large model. By adopting a progressive annotation file iterative optimization strategy from coarse to fine, the fine-grainedness and consistency of the dataset annotation are gradually improved. This method makes full use of the scene-related visual clues provided by the proprietary segmentation model Segformer, deeply mines domain features, and guides the common sense capabilities of the general large model SAM2 to achieve efficient collaboration between the two. Combined with the Prompt mechanism, dynamic annotation updates are generated through manual point selection and high-confidence prediction areas of the model to accurately capture the fouling and background areas. Ultimately, this method significantly enhances the annotation accuracy and improves the generalization ability of the joint model in complex underwater environments, providing efficient and reliable technical support for underwater fouling detection and segmentation.

[0061] When implementing the method of the present invention, the existing dataset is first divided into a training set and a test set. The Segformer model is trained using the original underwater images in the dataset (referred to as the original images) and their corresponding annotation files, GT (Ground Truth), to construct a preliminary segmentation model. Next, the entire dataset is input for prediction, and the Segformer decoding head outputs a predicted score value for each pixel in each image. After softmax processing, the predicted score value is converted into a stain or background probability matrix corresponding to each pixel. Based on this probability matrix, the coordinates of the pixels in the stain probability matrix that are greater than a set threshold are first clustered to form multiple clusters. The center points of the clusters are considered to be the stain suggestion point set and are labeled as 1. Simultaneously, the same clustering is performed on the pixels in the background probability matrix that are greater than the threshold, resulting in multiple clusters. The center points of each cluster are labeled as 0, forming the background suggestion point set. All pixels in the stain probability matrix that are greater than the set threshold constitute the stain suggestion area, and all pixels in the background probability matrix that are greater than the set threshold constitute the background suggestion area. Defects are defined as positive samples, and backgrounds as negative samples. After obtaining preliminary suggested regions for the defacement and background, a manual point selection mechanism is introduced. These selected points, along with the original image, are fed into the SAM2 model for segmentation, generating new defacement and background segmentation results. This results in an updated ground truth (GT). The updated GT is then fed back into the original image and the same steps are repeated to obtain a new segmentation result. This process continues for multiple iterations until the evaluation metrics of the newly generated GT no longer differ significantly from those of the original GT.

[0062] This interactive active learning method, guided by a large model, combines two high-performance segmentation models and significantly improves the accuracy of labeled files in the original dataset by introducing a manual point selection mechanism. Experts select points within the high-confidence suggested areas generated by Segformer, effectively narrowing the selection range and significantly improving selection efficiency. Furthermore, the inclusion of manual point selection allows experts to combine original image information with specialized domain knowledge to precisely focus on subtle boundary and texture features within the image, thus preventing critical details from being overlooked during automated labeling.

[0063] Specifically, such as Figure 1 As shown, the method includes the following steps:

[0064] Step 1) Obtain an existing underwater contamination dataset, which includes original underwater images and original annotation files.

[0065] Step 2) The original annotation file is converted into a single-channel file by mapping the original annotation file into a single-channel file.

[0066] In this step, the OpenCV library is used to read the existing annotation file. The white area in the annotation file, that is, the pixel points with the three-channel RGB value of (255,255,255) are mapped to label 1, indicating contamination; the black area, that is, the pixel points with the three-channel RGB value of (0,0,0) are mapped to 0, indicating background.

[0067] For each pixel in the image, the RGB value Pixel(x,y)=(R,G,B):

[0068] Defaced area (set as label 1): L(x,y) = 1, if R = 255, G = 255, B = 255;

[0069] Background area (set to label 0): L(x,y)=1, if R=0, G=0, B=0;

[0070] Among them, L(x,y) represents the label value of the position (x,y).

[0071] The set pixel labels are converted into single-channel images, and the generated single-channel images are saved using the cv2.imwrite statement of OpenCV.

[0072] The overall process of step 2) is as follows:

[0073] start

[0074] Read RGB input image as rgb img ;

[0075] Create a blank single-channel image label img , the same size as the input image;

[0076] For each pixel in the input image:

[0077] Get the RGB value of the pixel;

[0078] If R=255 and G=255 and B=255:

[0079] Label img Set to 1;

[0080] Otherwise if R=0 and G=0 and B=0:

[0081] Label img Set to 0;

[0082] Use OpenCV to save single-channel label image label img ;

[0083] Finish

[0084] Figure 7A schematic diagram of a labeled file and an original underwater image in an embodiment is shown.

[0085] In one embodiment, before proceeding to subsequent steps, the existing underwater fouling dataset is partitioned and preprocessed into a training set and a test set to ensure that the ratio of the number of training sets to the test sets is within a reasonable range. The dataset is also structured into a Cityscapes format to ensure that the dataset has a unified organizational structure and labeling standards, which facilitates the training and evaluation of the subsequent segmentation model.

[0086] like Figure 2 As shown, the data set is divided and preprocessed, and the total number of data sets is N+M=N all . Divide the training set on the existing underwater pollution dataset and validation set where X i represents the input image, Y i Represents a single-channel annotation file. The ratio of training set to validation set is controlled within the specified range, generally α=3, β=5, to ensure and The data set structure is converted to Cityscapes format.

[0087] Step 3) Use the channel-converted annotation files and the original underwater images to train the segmentation model.

[0088] In this example, Segformer is used as the image segmentation model. Segformer has excellent generalization capabilities, achieving efficient performance across different resolutions and scenarios, and is suitable for a variety of practical applications. This model can effectively handle complex backgrounds and detailed areas, excels in segmentation accuracy at class boundaries, and can output relatively accurate prediction scores, thereby improving the accuracy of annotation results.

[0089] In one embodiment, an AdamW optimizer is selected, wherein the initial learning rate is set to 0.00006, and the weight decay is set to 0.005, which can be expressed in terms of parameters: α = 0.00006, λ = 0.005.

[0090] The AdamW implementation process is as follows, where t represents the number of iterations:

[0091]

[0092] g t =g t-1 +λθ t-1 ,

[0093] m t =β1m t-1+(1-β1)g t ,

[0094]

[0095] Among them, g t is the current gradient, θ t-1 is the model weight in the previous gradient, λ is the regularization coefficient, and t represents the t-th iteration. Denotes the loss function f t About the weight θ at the last iteration t-1 The gradient of m t and v t are the first-order moment and the second-order moment respectively. The obtained m t and v t Requires bias correction:

[0096]

[0097] After the deviation is corrected, the gradient value of the next iteration can be calculated as follows:

[0098]

[0099] where θ t Indicates the value of the parameter at the tth iteration. m t and v t are the first-order moment estimate and the second-order moment estimate in the AdamW optimizer, respectively. ε is a small constant to prevent division by zero errors. α is the learning rate, and λ is the weight decay parameter. The loss function uses cross entropy, which is defined as follows:

[0100]

[0101] Where N represents the number of samples, y i Indicates the expected output, x i represents the actual input, f(x i ) represents the actual output. After 160K iterations, the loss function reaches convergence and the training ends.

[0102] like Figure 3 As shown, the above steps obtained and Input into Segformer model M seg , whose output is the predicted probability distribution Where θ is the model parameter. Minimize the cross entropy loss function during training. And when the loss function satisfies L ce (t)-L ceWhen (t-1)<ε, training is stopped and the weight file is obtained. t is the number of iterations, and the maximum number of iterations is 160K.

[0103] Step 4) Using the weight file trained in step 3), the original underwater image is used as the test image. The output probability score corresponding to each test image is captured in the decoding head of the segmentation model and converted into the contamination probability matrix and background probability matrix output by the test image.

[0104] When obtaining prediction scores in the Segformer decoding header, the post-processed model output is prioritized for score acquisition to ensure that the predicted scores are consistent with the output size and closer to the final output. The Softmax function is then used to convert the predicted scores into a probability distribution matrix between 0 and 1. This ensures that predictions for all categories are normalized to a valid probability distribution, providing a more scientific basis for subsequent probability analysis and threshold screening. Furthermore, this method intuitively reflects the model's prediction confidence for each category, thereby improving the reliability and accuracy of the results.

[0105] Specifically, such as Figure 4 As shown, step 4) includes the following steps:

[0106] Step 4-0) Define the total data set as where X K is the input image, Y K is the corresponding label, and P is the total number of training sets plus test sets. Scaled to H×W=1024×1024, as X input .X input As input, it goes into the Segformer model.

[0107] Step 4-1) The decoder head of the Segformer segmentation model captures the predicted score matrix for each test image, and inputs the predicted score matrix into the Softmax function to convert it into the output probability score corresponding to each pixel point, obtaining the total probability matrix of the defacement and background.

[0108] Specifically, for the Segformer model, the encoder is assumed to contain l stages, and each stage extracts features through a multi-head attention mechanism Where Q = F l W Q , K=F l W k , V=F l W v , and W Q , W k , W vis the weight parameter. The input of the first stage of the encoder is F l-1 , the output of the first stage is F l , F l = TransformerEncoder(F l-1 ),l=1,2,....,L. Embedding of the initial input image patch That is, through the convolution operation and Downsampling, the input image is divided into multiple patches, the size of each patch is as input to the first stage.

[0109] The decoder uses MLP lightweight decoding to decode the output F of different stages of the encoder. l The features are aggregated and the final segmentation result is generated. The output F of each stage of the encoder l Will be projected to have a uniform number of channels C out :

[0110]

[0111] Then, the features of different scales are upsampled to the original image resolution through bilinear interpolation: F up,l =Upsample(F proj,l ,size(W',H')).

[0112] Finally, the upsampled features of all stages are concatenated and fused:

[0113]

[0114] The final prediction score matrix Y∈R is obtained through convolution operation H’×W’×C :

[0115] Y = Conv 1×1 (F fusion ),Y∈R H’×W’×C ,

[0116] In the prediction score matrix, the score of each pixel needs to be converted into a probability value between 0 and 1. The Softmax function is used for conversion:

[0117] Y pre =Softmax(Conv 1×1 (F fusion )),Y pre ∈R H’×W’×C ,

[0118] The Softmax function is often used for normalization operations. The details are as follows:

[0119]

[0120] where h(x,y i ) represents the original output of the model (the probability of belonging to the stain and background, ∈ (0.1)), that is, it represents the unnormalized probability score, n represents the number of categories, x i represents the input feature, y i Indicates the expected output.

[0121] The total probability matrix Y of the stain and background is obtained through the Softmax function pre , whose dimensions are 2×512×512.

[0122] Step 4-2) Split the statement by tensor and From the total probability matrix Y pre Extract the contamination probability matrix P df and the background probability matrix P bg .

[0123] In this embodiment, the labels of the stain and background are set to 1 and 0 respectively. pre , from which the background probability matrix P is taken bg and the contamination probability matrix P df .

[0124] Figure 8 Shows the corresponding Figure 7 (7b) The segmentation result of the original underwater image after being processed by the Segformer model.

[0125] Step 5) Generate a proposed region and a proposed point set based on the contamination probability matrix and the background probability matrix.

[0126] Specifically, such as Figure 5 As shown, step 5) includes the following steps:

[0127] Step 5-1) Extract the background probability matrix P bg Pixels with a probability greater than the first threshold α:

[0128] S bg ={(x,y)|P bg (x,y)>α},

[0129] The pixel S bg The set formed is the background suggestion area X bg , X bg ={x1,x2,....,x h}.

[0130] Suggest region X for background bgThe pixels in the image are clustered using the K-Means algorithm based on the coordinate position (x, y) to obtain multiple clusters. The center of each cluster is used as the pixel representing the background, set to label 0, and used as a negative sample. The point set composed of negative samples is the background suggestion point set. The details are as follows:

[0131] From X bg Select k points as the initial cluster centers:

[0132] C bg ={C1,c2,.........,c l},c l ∈X bg ,l=1,2,......,k,

[0133] Calculate the Euclidean distance of each data point to all cluster centers and assign each data point to the cluster S corresponding to the cluster center with the shortest Euclidean distance l :

[0134]

[0135] After assigning each data point to the corresponding cluster, update the cluster center of each cluster:

[0136]

[0137] where |S l | represents cluster S l The number of data points in .

[0138] Then repeat the above operation of updating the cluster center of each cluster to minimize the following objective function:

[0139]

[0140] That is, minimize the sum of squared errors from all data points to the center of the cluster to which they belong.

[0141] After minimizing the objective function, we get the center coordinates of multiple clusters and set labels for the cluster centers. The background cluster center points are defined as negative sample points, and the labels are set to 0:

[0142] L bg ={(c l ,0)|l=1,2,...,k},

[0143] Among them, L bg Set as background suggestion point set.

[0144] Step 5-2) Extract the contamination probability matrix P df Pixels with a probability greater than the second threshold β:

[0145] S df ={(x,y)|P df (x,y)>β},

[0146] The pixel S df The set formed is the defacement suggestion area X df , X df ={x1,x2,....,x n}.

[0147] Suggested area X for defacement df The pixels in the image are clustered using the K-Means algorithm based on the coordinate position (x, y) to obtain multiple clusters. The center of each cluster is used as the pixel representing the background, set as label 1, and used as the positive sample. The point set consisting of the positive samples is the defacement suggestion point set. The details are as follows:

[0148] from df Select k points as the initial cluster centers:

[0149] C df ={c1,c2,.........,c j},c j ∈X df ,j=1,2,......,k,

[0150] Calculate the Euclidean distance of each data point to all cluster centers and assign each data point to the cluster S corresponding to the cluster center with the shortest Euclidean distance j :

[0151]

[0152] After assigning each data point to the corresponding cluster, update the cluster center of each cluster:

[0153]

[0154] where |S j | represents cluster S j The number of data points in .

[0155] Then repeat the above operation of updating the cluster center of each cluster to minimize the following objective function:

[0156]

[0157] That is, minimize the sum of squared errors from all data points to the center of the cluster to which they belong.

[0158] After minimizing the objective function, we get the center coordinates of multiple clusters and set labels for the cluster centers. The center points of the contaminated clusters are defined as positive sample points, and their labels are set to 1:

[0159] L df ={(c j ,1)|j=1,2,...,k},

[0160] Among them L df Set as the defacement suggestion point set.

[0161] In a preferred embodiment, the first threshold α and the second threshold β are determined based on an adaptive threshold method, such as Figure 5 As shown, the steps are as follows:

[0162] S1, the background probability matrix P bg and the contamination probability matrix P df Convert them into probability distribution histograms respectively, and calculate the gradient and difference of the histograms. The gradient is the speed at which the density value corresponding to the probability value changes with respect to the probability value, which represents the rate of change of the probability distribution. The difference is the change in the density value between adjacent probability values, that is, the absolute difference between each pair of adjacent probability values, which reflects the amplitude of the probability change.

[0163] The probability distribution histogram is expressed as:

[0164]

[0165] Among them, N bg , N df They represent the number of pixels in the background and contamination probability matrices respectively, I(x) is the indicator function, H(p) represents the histogram value corresponding to the probability p, represents the density of the number of pixels corresponding to the probability p, and Λ represents the original image.

[0166] The gradient is calculated based on the probability histogram, which represents the rate of change of the probability distribution grad:

[0167]

[0168] The difference is calculated based on the probability histogram, which reflects the change amplitude value diff between adjacent probability values:

[0169] diff(H(p))=|H(p)-H(p-1)|

[0170] Among them, H(p+1) and H(p-1) are the histogram values ​​of the previous and next probability intervals corresponding to probability p, respectively.

[0171] S2, according to the changes in gradient and difference, filter the middle range of probability distribution and get the middle range as [mid start,mid end ].

[0172] Calculate the absolute change in gradient and difference ΔG(x,y) and ΔD(x,y):

[0173] ΔG(x,y)=|G(x,y)-G mean ∣,

[0174] ΔD(x,y)=|D(x,y)-D mean ∣,

[0175] Among them, G mean , D mean are the global averages of gradient and difference, respectively.

[0176] Arrange ΔG(x,y) and ΔD(x,y) from low to high according to the value, and calculate the 90th percentile value P of the gradient and difference change amplitude. 90 (G) and P 90 (D):

[0177] P 90 (G)=Percentile(ΔG,90), P 90 (D)=Percentile(ΔD,90),

[0178] Filter the area that meets the conditions: {(x,y)|ΔG(x,y) <P 90 (G)and ΔD(x,y) <P 90 (D)}.

[0179] The above calculations are done by calculating the gradient and difference of the histogram, and the intervals where the gradient and difference change below the 90th percentile are selected as the middle range. Based on the changes in gradient and difference, the middle range is selected as follows:

[0180] mid start ≤p≤mid end ,

[0181] Among them, mid start and mid end Represent the starting and ending values ​​of the selected intermediate probability interval. This range is the relatively smooth and low-variance part of the probability distribution.

[0182] S3, the right endpoint mid of the middle range obtained based on the background probability matrix end As the first threshold α, the left endpoint mid of the middle range obtained based on the contamination probability matrix is ​​set start As the second threshold β.

[0183] In this embodiment, corresponding to Figure 8 The probability distribution histogram of Figure 9 shown.

[0184] Through visual operation, X df 、X bg It is displayed in the form of a heat map corresponding to the original image, such as Figure 10 As shown in the right figure, the red area represents the defacement suggestion area, the blue area represents the background suggestion area. The yellow area represents the uncertain area, which requires expert intervention to judge. The uncertain pixel x w The following conditions are met:

[0185]

[0186] The proposed point set L df With L bg Visualization on the original image, such as Figure 10 As shown in the left figure, red points represent positive samples and blue points represent negative samples.

[0187] In this step, all cluster centers obtained through the clustering operation constitute the proposed point set. By reducing the number of pixels in the proposed point set, this method not only effectively improves the representativeness of the proposed point set, but also significantly enhances the efficiency and accuracy of subsequent processing. Furthermore, the adaptive thresholding method is used for threshold selection, which dynamically adjusts the threshold based on the characteristics of different images, avoiding the problem of fixed thresholds being insufficiently applicable to different samples. This enhances the robustness of this method in complex scenes and the accuracy of the segmentation results.

[0188] Step 6) Manually select points based on the proposed area and the proposed point set and identify the ignored area.

[0189] Specifically, step 6) includes the following steps:

[0190] Step 6-1) Visualize the proposed point set on the original underwater image, and discard the label points in the proposed point set that do not match the actual ones according to the visualization results to obtain a new proposed point set R new ;

[0191] Step 6-2) Based on the new proposed point set R new , manually select 8 points in the suggested area and set corresponding labels as the first prompt point set prompt1:

[0192] prompt1={p i ∣p i ∈R sug ,1≤i≤n1},n1=8,

[0193] R sug =S df ∪S bg,

[0194] Among them, n1 represents the number of selected points prompt1, R sug Indicates a suggested area.

[0195] According to p i The suggested area to which it belongs, set its label l i :If p i Belong to the defacement suggestion area, l i =1; if p i Belongs to the background area, l i =0.

[0196] Step 6-3) Manually filter the ignored regions R that are not classified as suggested regions in step 5 but actually belong to the suggested regions ig (i.e., uncertainty area):

[0197]

[0198] Among them, Ψ is the set of the actual contaminated area and background area in the original image, R sug is the proposed region obtained in step 5).

[0199] If there is an ignored area, then in the ignored area r i ∈R ig Select 5 points as the second prompt point set prompt2:

[0200] prompt2={p k ∣p k ∈R ig ,1≤k≤n2},n2=5,

[0201] Otherwise, in the proposed region R sug Three additional points are selected as the second prompt point set prompt2. The selected points are different from the points in the first prompt point set:

[0202] prompt2={p k ∣p k ∈R sug ,1≤k≤n2},n2=3,

[0203] And according to p k In the actual situation of the area in the original image, set its label l k :If p k Belong to the contaminated area, l k =1; if p k belongs to the background area, l k =0.

[0204] Step 6-4) summarizes the first prompt point set prompt1 and the second prompt point set prompt2 as the prompt point prompt for interactive guidance update of the large model in step 7): prompt = prompt1∪prompt2, where,

[0205] prompt1={p i ∣p i ∈R sug ,1≤i≤n1},n1=5;

[0206] prompt2={p k ∣p k ∈R ig ,1≤k≤n2},n2={3,5}.

[0207] In one embodiment, prompt visualization is performed on the original image such as Figure 11 As shown, red represents positive sample points and blue represents negative sample points.

[0208] In this step, the proposed point set is visualized and overlaid on the original image, allowing experts to select points within the visualized proposed areas and further identify ignored areas. By visualizing the proposed areas and the proposed point set, experts can intuitively understand the segmentation results generated by the Segformer segmentation model, quickly identify key areas, and avoid omissions. In the process of selecting points and identifying ignored areas, experts combine original image information and domain knowledge to effectively address the model's shortcomings in special scenarios or boundary processing, thereby improving the comprehensiveness and accuracy of the annotated data and providing more reliable, high-quality input data for subsequent model optimization.

[0209] Step 7) The points selected in step 6) are used as prompt points for interactive guidance update of the large model, and are input into the large model SAM2 together with the corresponding original underwater image to obtain a new segmentation result.

[0210] Input prompt and original image into SAM2 model to get new segmentation result. Let original image be I, and take prompt point set prompt and I as input, and input them into SAM2 model M SAM2 :

[0211] S new =M SAM2 (I, prompt),

[0212] Among them, S new is the new segmentation result, which is a matrix with the same size as the input image I. The pixel value s(x,y) represents the segmentation label of the coordinate (x,y):

[0213] s(x,y)=1, if (x,y) belongs to the contaminated area;

[0214] s(x,y)=0, if (x,y) belongs to the background area.

[0215] In one embodiment, the new segmentation result is as follows: Figure 12 shown.

[0216] This step leverages the powerful segmentation capabilities of the SAM2 model to automatically generate high-quality segmentation results, significantly reducing the workload of manual annotation. Furthermore, an interactive guidance mechanism enables effective supervision and adjustment of the model's output, particularly in complex or fuzzy areas, incorporating expert knowledge to further improve the accuracy and consistency of annotations. This guided update process dynamically optimizes dataset annotations, improving overall dataset quality and providing more reliable input data for subsequent model training, significantly enhancing the performance of segmentation models in practical applications.

[0217] Step 8) Figure 6 As shown in FIG, the new segmentation result is used as the new annotation file GT′, and the new annotation file GT′ is compared with the original annotation file GT. If the comparison result meets the preset conditions, the iteration ends and the new annotation file GT′ is used as the final non-destructive annotation output of the underwater hull. Otherwise, the process returns to step 3), and a new training dataset is constructed based on the new annotation file GT′ and the original underwater image I. The segmentation model is retrained and the next iteration is performed.

[0218] The new training data set D′ is expressed as:

[0219]

[0220] Define the number of iterations as k, the annotation file of the kth iteration is GT', prompt (k) Denotes the Prompt obtained after the kth iteration, and D′ is repeated through the above process. The recursive expression is:

[0221] GT (k+1) =M SAM2 (I, prompt (k) ),

[0222] The original manually annotated ground truth is constant and provides a stable reference as a baseline. The SAM2 output changes dynamically, and by comparing it with the constant ground truth, we can reflect the improvement trend of the model during the optimization process.

[0223] Specifically, such as Figure 6 As shown, the judgment condition for whether the iteration ends is:

[0224] Compare the new segmentation results with the original annotation files and calculate the segmentation evaluation indicators, including the mean intersection over union (mIoU), the mean Dice coefficient (mDice), and the average pixel accuracy (mPA).

[0225] Save the mIoU, mDice, and mPA obtained after each iteration, and calculate the incremental change rate of the segmentation evaluation index of the current iteration relative to the previous iteration:

[0226] ΔmIoU=|mIoU n -mIoU n-1 ∣,

[0227] ΔmDice=|mDice n -mDice n-1 ∣,

[0228] ΔmPA=|mPA n -mPA n-1 ∣,

[0229] Where n represents the number of iterations.

[0230] If, after a certain iteration, the rate of change Δm of all indicators is lower than the set threshold σ, it is recorded as "meeting the convergence condition". If the condition is met b times in a row, the iteration is terminated. The set threshold σ is set using the percentile method, and the 90th percentile of Δm in the historical iteration is calculated as the threshold:

[0231] σ n =percentile(Δm,90),

[0232] This can filter out large abnormal changes and only focus on the normal fluctuation range. In this embodiment, b is selected as 3.

[0233] In one embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0234] Many components in an electronic device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0235] The computing unit can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as image processing methods and / or model training methods. For example, in some embodiments, the image processing methods and / or model training methods can be implemented as computer software programs that are tangibly contained in machine-readable media, such as storage units. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of the image processing methods and / or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to perform image processing methods and / or model training methods in any other appropriate manner (e.g., by means of firmware).

[0236] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0238] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0239] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for optimizing underwater hull fouling annotation based on multimodal large model guidance, characterized in that: The method comprises the following steps: Step 1) obtaining an existing underwater contamination dataset, wherein the underwater contamination dataset includes original underwater images and original annotation files; Step 2) performing channel conversion on the original annotation file by mapping, converting the three-channel annotation file into a single-channel one; Step 3) using the channel-converted annotated file and the original underwater image to train the segmentation model; Step 4) Using the weight file trained in step 3), the original underwater image is used as a test image, and the output probability score corresponding to each test image is captured in the decoding head of the segmentation model, and converted into a contamination probability matrix and a background probability matrix outputted by the test image; Step 5) generating a suggested region and a suggested point set based on the contamination probability matrix and the background probability matrix; Step 6) Manually select points based on the proposed area and the proposed point set and identify the ignored area; Step 7) The points selected in step 6) are used as prompt points for interactive guidance update of the large model, and are input into the large model together with the corresponding original underwater image to obtain a new segmentation result; Step 8) Use the new segmentation result as a new annotation file and compare the new annotation file with the original annotation file. If the comparison result meets the preset conditions, the iteration ends and the new annotation file is output as the final lossless annotation of the underwater hull. Otherwise, return to step 3) and retrain the segmentation model based on the new annotation file and the original underwater image for the next iteration.

2. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: In the step 2), the white area in the annotation file is mapped to label 1, indicating contamination, and the black area is mapped to label 0, indicating background.

3. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: The step 4) comprises the following steps: Step 4-1) The decoding head of the segmentation model captures the predicted score matrix for each test image, and inputs the predicted score matrix into the Softmax function to convert it into the output probability score corresponding to each pixel, thereby obtaining the total probability matrix of the defacement and background. Step 4-2) extracts the contamination probability matrix and the background probability matrix from the total probability matrix through a tensor split statement.

4. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: The step 5) comprises the following steps: Step 5-1) Extract pixels with a probability greater than a first threshold from the background probability matrix. The set of pixels constitutes the background proposal region. Cluster the pixels in the background proposal region based on their coordinate positions to obtain multiple clusters. The center of each cluster is used as the pixel representing the background, set to label 0, and used as a negative sample. The point set consisting of the negative samples is the background proposal point set. Step 5-2) Extract the pixel points with a probability greater than the second threshold in the contamination probability matrix, and the set of pixel points is the contamination suggestion area; cluster the pixel points in the contamination suggestion area based on the coordinate position to obtain multiple clusters, and take the center of each cluster as the pixel point representing the background, set it as label 1, and serve as a positive sample. The point set consisting of the positive samples is the contamination suggestion point set.

5. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 4 is characterized in that: The first threshold and the second threshold are determined based on an adaptive threshold method, and the steps are as follows: Convert the background probability matrix and the contamination probability matrix into probability distribution histograms respectively, and calculate the gradient and difference of the histograms, where the gradient is the speed of change of the density value corresponding to the probability value with respect to the probability value, and the difference is the change of the density value between adjacent probability values, that is, the absolute difference between each pair of adjacent probability values; According to the changes in gradient and difference, the middle range of probability distribution is screened; The right endpoint of the middle range obtained based on the background probability matrix is ​​used as the first threshold, and the left endpoint of the middle range obtained based on the stain probability matrix is ​​used as the second threshold.

6. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: Described step 6) comprises the following steps: Step 6-1) Visualize the proposed point set on the original underwater image, and discard the label points in the proposed point set that do not match the actual label based on the visualization result to obtain a new proposed point set; Step 6-2) Based on the new set of suggested points, manually select a first preset number of points in the suggested area and set corresponding labels as a first prompt point set; Step 6-3) manually screening the ignored areas that were not classified as suggested areas in step 5) but actually belong to the suggested areas. If ignored areas exist, a second preset number of points are selected from the ignored areas and corresponding labels are set as the second prompt point set. Otherwise, a third preset number of points are selected from the suggested areas, the selected points being different from the points in the first prompt point set, and corresponding labels are set as the second prompt point set. Step 6-4) summarizes the first cue point set and the second cue point set as cue points for interactively guiding the update of the large model in step 7).

7. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: In step 8), the condition for determining whether the iteration is finished is: Compare the new segmentation results with the original annotation file and calculate the segmentation evaluation indicators, which include the average intersection-over-union ratio, the average Dice coefficient and the average pixel accuracy of each category; The incremental change rate of the segmentation evaluation index of the current iteration relative to the previous iteration is calculated. If the incremental change rate of each segmentation evaluation index is lower than the third threshold for a preset number of consecutive times, the preset condition is met and the iteration is terminated. Otherwise, the next iteration is performed.

8. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 7 is characterized in that: The third threshold is set using the percentile method and is set to the 90th percentile of the incremental change rate of the segmentation evaluation indicator in the historical iteration.

9. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: The segmentation model adopts the Segformer segmentation model.

10. The underwater hull fouling annotation optimization method based on multimodal large model guidance according to claim 1 is characterized in that: The large model adopts the SAM2 model.

Images