An underwater hull fouling labeling optimization method based on multi-modal large model guidance

Abstract

The application relates to a kind of underwater hull fouling annotation optimization methods based on multi-modal large model guidance, comprising the following steps: obtaining existing underwater fouling dataset, and the annotation file in it is converted channel training segmentation model;Using the weight file obtained by training, the output probability score corresponding to the test image is captured in the decoding head of the segmentation model, and the fouling and background probability matrix is obtained;Generate suggestion area and suggestion point set, carry out artificial point selection and judge to ignore area;The point selected artificially is used as the prompt point of the interactive guidance update of large model, and the new segmentation result is obtained by using the large model, and is used as new annotation file, compare new annotation file and original annotation file to judge whether iteration is ended, if not, retrain the segmentation model based on new annotation file.Compared with the prior art, the application can effectively improve the annotation accuracy of underwater fouling image, be suitable for underwater image segmentation task with complex scene, reduce artificial workload and improve efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and computer vision, and particularly relates to a method for underwater ship hull fouling label optimization based on multi-modal large model guidance. BACKGROUND

[0002] Underwater ship hull fouling detection is a key link in the fields of ship operation, maintenance and environmental protection. Hull fouling not only directly leads to a significant decrease in fuel efficiency, but also may cover up structural problems such as metal fatigue or corrosion, thereby burying major safety hazards. In addition, biological attachment in the fouling area may have long-term adverse effects on the marine ecosystem. As can be seen, the efficient and accurate detection of fouling has important strategic significance for reducing operating costs, improving sailing safety and protecting the marine environment.

[0003] The labeling work of underwater data faces significant challenges under the conditions of small-scale data sets, domain differences and initial rough labeling. First, the complex and changeable underwater environment makes it difficult for manual labeling to achieve fine-grained accuracy, resulting in problems such as boundary ambiguity and error accumulation in the initial labeling results. Second, manual labeling not only requires a lot of time and effort of professional personnel, but also is difficult to achieve an ideal level in terms of labeling quality and consistency. In the case of small-scale data sets, these problems are further exacerbated, making the model perform poorly in cross-scene applications with domain differences, and limiting its applicability in fine-grained segmentation tasks due to its dependence on rough labeling.

[0004] In recent years, segmentation models based on the Transformer architecture have become an important technical breakthrough in the field of computer vision due to their excellent global feature modeling capability and cross-scene application performance. However, their performance is limited in small-scale data sets, domain differences and rough labeling scenarios. Transformer models rely on large-scale, high-quality labeled data sets to ensure that the self-attention mechanism can fully capture the global relationships between features. However, the initial rough labeling has ambiguity in the expression of boundary information, making it difficult for the model to effectively distinguish the fine-grained features of adjacent regions in complex underwater scenes. In addition, the model's tendency to focus too much on large-scale features when trained on small-scale data sets further weakens its ability to segment the details of the fouling area, making it difficult to meet the complex task requirements in specific domains.

[0005] Multi-modal large model SAM has shown extraordinary potential in general image segmentation tasks in recent years due to its excellent zero-shot segmentation capability and interactive prompt advantage, especially in diverse scenarios such as underwater and open environments. However, the general capability of the SAM model is significantly limited under the conditions of small-scale data sets, significant domain differences, and rough annotation. Therefore, the prior art proposes to combine the Transformer model and the multi-modal large model SAM to improve the performance of image segmentation. For example, CN118298169A discloses an image segmentation method based on large model guidance, which comprises the following steps: constructing a large model guided segmentation network model; wherein the large model guided segmentation network model comprises a large model optimization module and a multi-task image processing module; the large model optimization module is used to segment and filter the input image to obtain a segmentation mask close to the real scene label as guidance information; the multi-task image processing module is used to extract the feature map of the input image and fuse the guidance information to realize image segmentation; a preset type of image set is used to train the constructed large model guided segmentation network model; and the trained large model guided segmentation network model is used to obtain the image segmentation result of the image to be segmented.

[0006] However, this method is not suitable for underwater image processing. First of all, the lack of domain knowledge in underwater scenes, such as the complexity of lighting and the diversity of fouling morphology, makes it difficult for the model to effectively capture fine-grained features. In addition, although its zero-shot segmentation capability is highly dependent on the quality of the prompt, in the face of the boundary ambiguity and complex target area brought by the initial rough annotation, the segmentation result generated by the model is often limited to the coarse-grained level, making it difficult to meet the high-precision requirements for boundary details and target microstructure in underwater ship hull fouling detection. SUMMARY

[0007] The purpose of the present application is to overcome the defects of the prior art and provide a multi-modal large model guided underwater ship hull fouling annotation optimization method. The method combines a specialized segmentation model Segformer with a large model SAM2, captures the domain features of the underwater scene through Segformer, and combines the candidate region generation capability of SAM2 to effectively utilize active learning technology to iteratively optimize the annotation results, significantly improving the segmentation accuracy and annotation consistency, while reducing the annotation cost, and strengthening the model's feature extraction and detail description capability in complex environments, providing a new technical path and engineering practical value for underwater data optimization and ship hull fouling detection.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] According to a first aspect of the present application, a multi-modal large model guided underwater ship hull fouling annotation optimization method is provided, which comprises the following steps:

[0010] Step 1) Obtain an existing underwater fouling data set, which includes original underwater images and an original annotation file;

[0011] Step 2) Convert the original annotation file to a single channel by mapping the three-channel annotation file.

[0012] Step 3) Train a segmentation model using the channel-converted annotation file and the original underwater image.

[0013] Step 4) Use the weight file obtained by training in step 3) to take the original underwater image as a test image, capture the output probability score corresponding to each test image at the decoding head of the segmentation model, and convert it to a fouling probability matrix and a background probability matrix output by the test image.

[0014] Step 5) Generate a set of suggestion regions and suggestion points based on the fouling probability matrix and the background probability matrix.

[0015] Step 6) Manually select points and determine ignored regions based on the set of suggestion regions and suggestion points.

[0016] Step 7) Take the points selected in step 6) as prompt points for large model interactive guided update, and input them together with the corresponding original underwater image into the large model to obtain a new segmentation result.

[0017] Step 8) Take the new segmentation result as a new annotation file, compare the new annotation file with the original annotation file, if the comparison result meets the preset condition, end the iteration, take the new annotation file as the final underwater hull non-destructive annotation output, otherwise, return to step 3) and retrain the segmentation model based on the new annotation file and the original underwater image for the next iteration.

[0018] As a preferred technical solution, in step 2), the white area in the annotation file is mapped to label 1, indicating fouling, and the black area is mapped to 0, indicating background.

[0019] As a preferred technical solution, step 4) includes the following steps:

[0020] Step 4-1) Capture the prediction score matrix of each test image at the decoding head of the segmentation model, input the prediction score matrix into the Softmax function to convert it into the output probability score corresponding to each pixel point, and obtain the total probability matrix of fouling and background.

[0021] Step 4-2) Extract the fouling probability matrix and the background probability matrix from the total probability matrix through the tensor segmentation statement.

[0022] As a preferred technical solution, step 5) includes the following steps:

[0023] Step 5-1) extracting the pixel points with probabilities greater than the first threshold value in the background probability matrix, and the set of the pixel points constitutes a background suggestion region; clustering the pixel points in the background suggestion region based on the coordinate positions to obtain multiple clusters, taking the center of each cluster as a pixel point representing the background, setting as a label 0, and as a negative sample, and the point set constituted by the negative sample is a background suggestion point set;

[0024] Step 5-2) extracting the pixel points with probabilities greater than the second threshold value in the stain probability matrix, and the set of the pixel points constitutes a stain suggestion region; clustering the pixel points in the stain suggestion region based on the coordinate positions to obtain multiple clusters, taking the center of each cluster as a pixel point representing the background, setting as a label 1, and as a positive sample, and the point set constituted by the positive sample is a stain suggestion point set.

[0025] As a preferred technical solution, the first threshold value and the second threshold value are determined based on an adaptive threshold method, and the steps are as follows:

[0026] Converting the background probability matrix and the stain probability matrix into probability distribution histograms respectively, calculating the gradient and the difference value of the histogram, wherein the gradient is the change speed of the density value corresponding to the probability value with respect to the probability value, and the difference value is the change of the density value between adjacent probability values, that is, the absolute difference between each pair of adjacent probability values;

[0027] According to the change of the gradient and the difference value, the middle range of the probability distribution is screened;

[0028] Taking the right end point of the middle range obtained based on the background probability matrix as the first threshold value, and taking the left end point of the middle range obtained based on the stain probability matrix as the second threshold value.

[0029] As a preferred technical solution, the step 6) includes the following steps:

[0030] Step 6-1) visualizing the suggestion point set on the original underwater image, and removing the label points in the suggestion point set which do not conform to the actual situation according to the visualization result to obtain a new suggestion point set;

[0031] Step 6-2) based on the new suggestion point set, manually selecting a first preset number of points in the suggestion region and setting the corresponding labels as a first hint point set;

[0032] Step 6-3) manually screening an ignored region which is not classified as a suggestion region in step 5) but actually belongs to the suggestion region, if the ignored region exists, selecting a second preset number of points in the ignored region and setting the corresponding labels as a second hint point set, otherwise, selecting a third preset number of points in the suggestion region, the selected points being different from the points in the first hint point set, and setting the corresponding labels as the second hint point set;

[0033] Step 6-4) The first set of hint points and the second set of hint points are combined as hint points for the large model interactive guidance update in step 7).

[0034] As a preferred technical solution, in step 8), the judgment condition for whether the iteration is ended is:

[0035] The new segmentation result is compared with the original annotation file to calculate a segmentation evaluation index, and the segmentation evaluation index includes an average intersection over union, an average Dice coefficient, and a class average pixel accuracy.

[0036] The incremental change rate of the segmentation evaluation index of the current iteration relative to the last iteration is calculated, if the incremental change rate of each segmentation evaluation index is continuously lower than a third threshold value for a preset number of times, the preset condition is met, the iteration is terminated, otherwise, the next iteration is performed.

[0037] As a preferred technical solution, the third threshold value is set by using a percentile method, and is set as the 90th percentile of the incremental change rate of the segmentation evaluation index in the history iteration.

[0038] As a preferred technical solution, the segmentation model uses a Segformer segmentation model.

[0039] As a preferred technical solution, the large model uses a SAM2 model.

[0040] According to the second aspect of the present application, an electronic device is provided, comprising a memory and a processor, the memory has a computer program stored thereon, and the processor implements the method when executing the program.

[0041] According to the third aspect of the present application, a computer readable storage medium is provided, which has a computer program stored thereon, and the program is executed by a processor to implement the method.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] (1) The present application cooperates the segmentation model with the large model, introduces the active learning technology, and realizes the significant optimization of the annotation process. Unlike the traditional way of relying on manual direct annotation of the dirty area, the present application transfers the core link of manual participation to the hint point selection stage, avoids the huge workload of manual fine annotation of the whole image from beginning to end, and significantly reduces the human annotation error by means of the generated suggestion area, and significantly improves the annotation accuracy. Thereafter, the large model automatically completes the segmentation calculation of the complex area, replaces the manual tedious pixel-level description, reduces the workload of manual intervention, effectively improves the efficiency and reliability of the data set annotation, and provides higher quality annotation data for subsequent model training.

[0044] (2) In the prompt information generation process, the application fully integrates the field-specific knowledge and the original data. First, through the preliminary prediction of the suggestion area of the pollution and the background by the segmentation model, the pollution probability matrix and the background probability matrix predicted by the model are used to assist the manual identification of the uncertain area (fuzzy boundary, complex target area) and the potential error area (ignored area) of the model, providing a scientific basis for the selection of the prompt point. Second, the manual selection of the supplementary prompt point is combined with the field understanding and common sense judgment of the ignored area, which avoids blind target annotation on the whole image, focuses the limited manpower on the details that the model is most difficult to grasp, and improves the annotation accuracy and efficiency. The double verification mechanism significantly improves the accuracy of the prompt point selection, and enables the large model to accurately understand the boundary information and microstructure characteristics indicated by the prompt points, generating a new segmentation result that is more fine and more consistent with the real structure. Through this optimization process, the annotation accuracy is further improved, laying a solid foundation for the training of high-quality segmentation models.

[0045] (3) Each iteration of the application re-trains the segmentation model based on the more accurate annotation file after updating, and as the annotation quality improves, the performance of the trained segmentation model also gradually increases. The improved model can generate more accurate and focused uncertainty maps, guiding the next round of human and large model optimization at a more subtle level. This iterative mechanism allows the model to gradually learn the high-quality annotation knowledge generated by the collaboration of the large model and the human, which is more consistent with the complex lighting and morphological diversity of underwater pollution. Even if the initial annotation is rough or the field knowledge embedding is insufficient, through multiple rounds of optimization, the final annotation data that meets the high-precision requirements can be generated, indirectly compensating for the lack of field knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The method flowchart of the application is shown in the figure;

[0047] Figure 2 The flowchart of data division and preprocessing in the application is shown in the figure;

[0048] Figure 3 The flowchart of the initial training of the Segformer segmentation model in the application is shown in the figure;

[0049] Figure 4 The flowchart of model prediction and probability matrix generation in the application is shown in the figure;

[0050] Figure 5 The flowchart of manual point selection and ignored area selection in the application is shown in the figure;

[0051] Figure 6 The flowchart of the SAM2 large model generating a new segmentation result in the application is shown in the figure;

[0052] Figure 7 Fig. 7 is a schematic diagram of an original annotation file and a corresponding original underwater image in an embodiment of the present application, wherein (7a) is the original annotation file, and (7b) is the original underwater image;

[0053] Figure 8 Fig. 8 is a schematic diagram of an initial prediction result generated by a Segformer model in an embodiment of the present application;

[0054] Figure 9 Fig. 9 is a schematic diagram of a probability distribution histogram generated on the prediction result in an embodiment of the present application; Figure 8 Fig. 9 is a schematic diagram of a probability distribution histogram generated on the prediction result in an embodiment of the present application;

[0055] Figure 10 Fig. 10 is a schematic diagram of a generated suggestion point set and suggestion region in an embodiment of the present application;

[0056] Figure 11 Fig. 11 is a visualized diagram of a hint point in an embodiment of the present application;

[0057] Figure 12 Fig. 12 is a schematic diagram of a generated new segmentation result in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0059] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive, not exclusive. The terms "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "has", "having", "have" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of steps or units (elements) not only comprises those steps or units but can also comprise other steps or units not expressly listed or inherent to such process, method, system, product or apparatus. The terms "connected", "coupled", and similar referents in the context of this application are to be construed as not necessarily being limited to a direct or physical connection or coupling, but can also include an electrical connection or coupling, whether direct or indirect. The term "multiple" refers to two or more. The term "and / or" describes the association relationship of associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like in the present application are only to distinguish similar objects, and do not represent a specific order of the objects.

[0060] In view of the challenges of small data set size, existence of Domain Gap and rough initial annotation, the embodiment provides a method for underwater hull fouling annotation optimization based on multi-modal large model guidance. By adopting the gradual annotation file iterative optimization strategy from coarse to fine, the fine granularity and consistency of the data set annotation are gradually improved. This method makes full use of the scene-related visual clues provided by the special segmentation model Segformer, deeply excavates the field features, and guides the common sense ability of the general large model SAM2, realizing the efficient cooperation of the two. Combined with the Prompt mechanism, dynamic annotation updates are generated by artificial point selection and high confidence prediction area of the model, accurately capturing the fouling and background area. Finally, this method significantly enhances the annotation accuracy and improves the generalization ability of the joint model in complex underwater environments, providing efficient and reliable technical support for underwater fouling detection and segmentation.

[0061] In the implementation of the method of the present application, first, the existing data set is divided into a training set and a test set, and the original underwater image (referred to as the original image) in the data set and its corresponding annotation file GT (Ground Truth) are used to train the Segformer model to build a preliminary segmentation model. Then, the entire data set is input for prediction, and the decoding head of the Segformer will output the predicted score value of each pixel point in each image. After Softmax processing, the predicted score value will be converted into a pollution or background probability matrix corresponding to each pixel point. Based on this probability matrix, first, the pixel points in the pollution probability matrix that are greater than a set threshold are clustered to form multiple clusters, and the center points of the clusters are regarded as a pollution suggestion point set and marked as label 1. At the same time, the pixel points in the background probability matrix that are greater than the threshold are clustered in the same way to obtain multiple clusters, and the center points of each cluster are marked as label 0 to form a background suggestion point set. All pixel points in the pollution probability matrix that are greater than the set threshold form a pollution suggestion region, and all pixel points in the background probability matrix that are greater than the set threshold form a background suggestion region. Define pollution as positive samples and background as negative samples. After obtaining the preliminary suggestion regions of pollution and background, the artificial point selection mechanism is introduced. The selected points are input into the SAM2 model together with the original image for segmentation to generate new pollution and background segmentation results, and the GT is updated accordingly. The updated GT and the original image are used as input, and the same steps are taken to obtain new segmentation results. This process is iterated several times until the evaluation indicators of the newly generated GT and the original GT no longer change significantly, and the iteration process is terminated.

[0062] The interactive active learning method based on large model guidance combines two high-performance segmentation models and significantly improves the accuracy of the label file in the original data set by introducing an artificial point selection mechanism. Experts select points in the high-confidence suggestion region generated by Segformer, effectively reducing the point selection range and significantly improving the point selection efficiency. At the same time, by introducing the artificial point selection link, experts can combine the original image information and professional knowledge to accurately focus on the subtle boundaries and texture features in the image, thereby avoiding missing key details due to automated processing of the annotation file.

[0063] Specifically, as shown in Figure 1 the method includes the following steps:

[0064] Step 1) Obtain an existing underwater pollution data set, which includes original underwater images and original annotation files.

[0065] Step 2) Perform channel conversion on the original annotation file through mapping to convert the three-channel annotation file into a single-channel.

[0066] In this step, the existing annotation file is read using the OpenCV library. The white area in the annotation file, i.e., the pixel point with three-channel rgb value (255, 255, 255), is mapped to label 1, representing contamination; the black area, i.e., the pixel point with three-channel rgb value (0, 0, 0), is mapped to 0, representing background.

[0067] For the RGB value of each pixel in the image Pixel(x, y) = (R, G, B):

[0068] Contamination area (set to label 1): L(x, y) = 1 if R = 255, G = 255, B = 255;

[0069] Background area (set to label 0): L(x, y) = 1 if R = 0, G = 0, B = 0;

[0070] Where L(x, y) represents the label value at position (x, y).

[0071] The set pixel label is converted to a single-channel image, and the generated single-channel image is saved using the cv2.imwrite statement of OpenCV.

[0072] The overall process of step 2) is as follows:

[0073] Start

[0074] Read the RGB input image as rgb img ;

[0075] Create a blank single-channel image label img with the same size as the input image;

[0076] For each pixel point in the input image:

[0077] Get the RGB value of the pixel;

[0078] If R = 255 and G = 255 and B = 255:

[0079] Set label img to 1;

[0080] Otherwise if R = 0 and G = 0 and B = 0:

[0081] Set label img to 0;

[0082] Save the single-channel label image label img using OpenCV;

[0083] End

[0084] Figure 7A schematic diagram of the annotation file and the original underwater image in an embodiment is shown.

[0085] In an embodiment, the existing underwater fouling dataset is divided and preprocessed before entering the subsequent steps, divided into a training set and a test set, the number ratio of the training set and the test set is ensured to be within a reasonable range, and the dataset structure is processed into Cityscapes format to ensure that the dataset has a unified organizational structure and label standard, facilitating the training and evaluation of the subsequent segmentation model.

[0086] As shown in Figure 2 , the dataset is divided and preprocessed, and the total number of the dataset is N+M=N all . The training set and the validation set are divided on the existing underwater fouling dataset. i , wherein X i represents the input image, and Y t represents the single-channel annotation file. The ratio of the training set and the validation set is controlled within a specified range by using , generally α=3 and β=5, to ensure that the number ratio of the training set and the validation set is within a reasonable range. The dataset structure is processed into Cityscapes format.

[0087] Step 3) The segmentation model is trained using the channel-converted annotation file and the original underwater image.

[0088] In this embodiment, Segformer is used as the image segmentation model. Segformer has excellent generalization ability and can achieve efficient performance in different resolutions and scenes, and is suitable for various practical application environments. This model can effectively process complex backgrounds and detail areas, and performs excellently in the segmentation accuracy of class boundaries, and can output relatively accurate prediction scores, thereby improving the accuracy of the annotation results.

[0089] In an embodiment, the AdamW optimizer is selected, wherein the initial learning rate is set to 0.00006 and the weight decay is set to 0.005, which can be expressed by parameters as follows: α=0.00006, λ=0.005.

[0090] The implementation process of AdamW is as follows, wherein t represents the number of iterations:

[0091]

[0092] g t-1 = g t-1 + λθ t ,

[0093] m t-1 ​​+(1-β1)g t ,

[0094]

[0095] Among them, g t It is the current gradient, θ t-1 λ is the model weight in the previous gradient, λ is the regularization coefficient, and t represents the t-th iteration. The loss function f represents t Regarding the weight θ from the previous iteration t-1 The gradient of m. t and v t These are the first and second moments, respectively. The resulting m... t and v t Deviation correction is required:

[0096]

[0097] After bias correction, the gradient value for the next iteration can be calculated, as shown below:

[0098]

[0099] Where θ t This represents the value of the parameter in the t-th iteration. t and v t These are the first-order and second-order moment estimates in the AdamW optimizer, respectively. ε is a small constant to prevent division by zero errors. α is the learning rate, and λ is the weight decay parameter. The loss function utilizes cross-entropy loss, defined as follows:

[0100]

[0101] Where N represents the number of samples, y i Indicates the expected output, x i f(x) represents the actual input. i The output is represented by ). After 160K iterations, the loss function converges, and training ends.

[0102] like Figure 3 As shown, the results obtained from the aforementioned steps and Input into Segformer model M seg Its output is the predicted probability distribution. Where θ represents the model parameters. During training, the cross-entropy loss function is minimized, and the loss function satisfies L... ce (t)-L ceWhen (t-1) < ε, stop training to obtain a weight file, t is the number of iterations, and the maximum number of iterations is 160K.

[0103] Step 4) Using the weight file obtained by training in step 3), the original underwater image is taken as a test image, the output probability score corresponding to each test image is captured at the decoding head of the segmentation model, and it is converted into a pollution probability matrix and a background probability matrix output by the test image.

[0104] When obtaining the prediction score in the Segformer decoding head part, the post-processed model output result is preferentially selected for score acquisition to ensure that the prediction score is consistent with the output result size and the prediction score is closer to the final output result. Subsequently, the prediction score is converted into a probability distribution matrix between 0 and 1 by using the Softmax function. By using the Softmax function, it is ensured that the prediction results of all categories are normalized into an effective probability distribution, thereby providing a more scientific basis for subsequent probability analysis and threshold screening. In addition, this method can intuitively reflect the prediction reliability of the model for each category, thereby improving the reliability and accuracy of the results.

[0105] Specifically, as shown in FIG. 4, step 4) includes the following steps: Figure 4

[0106] Step 4-0) defines the total data set as where X K is the input image, Y K is the corresponding label, and P is the total number of the training set plus the test set. The total data set is scaled to HxW = 1024x1024 as X input . X input is input into the Segformer model.

[0107] Step 4-1) captures the prediction score matrix of each test image at the decoding head of the Segformer segmentation model, inputs the prediction score matrix into the Softmax function to convert it into the output probability score corresponding to each pixel point, and obtains the total probability matrix of pollution and background.

[0108] Specifically, for the Segformer model, let the encoder contain l stages, and each stage extracts features through a multi-head attention mechanism where Q = F l W Q , K = F l W k , V = F l W v , and W Q , W k , W v ​Let F be the weight parameter. The input in the l-th stage of the encoder is F. l-1 The output of stage l is F l F l =TransformerEncoder(F l-1 Embedding of initial input image patches, l = 1, 2, ..., L. That is, to use convolution operations to... and Downsampling divides the input image into multiple patches, each patch having a size of [size missing]. As input for the first stage.

[0109] The decoder section uses lightweight MLP decoding, which converts the output F from different stages of the encoder into a single decoder. l Feature aggregation is performed to generate the final segmentation result. The encoder output F at each stage... l It will be projected onto a screen with a uniform number of channels C. out :

[0110]

[0111] Then, features at different scales are upsampled to the original image resolution using bilinear interpolation: F up,l =Upsample(F proj,l ,size(W',H')).

[0112] Finally, the upsampled features from all stages are concatenated and fused:

[0113]

[0114] The final prediction score matrix Y∈R is obtained through convolution operations. H’×W’×C :

[0115] Y = Conv 1×1 (F fusion ),Y∈R H’×W’×C ,

[0116] In the predicted score matrix, the score of each pixel needs to be converted into a probability value between 0 and 1. This transformation is performed using the Softmax function.

[0117] Y pre =Softmax(Conv 1×1 (F fusion )),Y pre ∈R H’×W’×C ,

[0118] The Softmax function is commonly used for normalization operations; details are as follows:

[0119]

[0120] Where h(x,y) i ) represents the model's raw output (probabilities belonging to dirt and background, respectively, ∈ (0.1)), which represents the unnormalized probability score, where n represents the number of categories, x i Represents the input features, y i This indicates the expected output.

[0121] The total probability matrix Y of dirt and background is obtained by using the Softmax function. pre Its dimensions are 2×512×512.

[0122] Step 4-2) Partitioning statements using tensors and From the total probability matrix Y pre Extract the contamination probability matrix P from df and background probability matrix P bg .

[0123] In this embodiment, the labels for dirt and background are set to 1 and 0, respectively. For the total probability matrix Y... pre Extract the background probability matrix P from it. bg and the probability matrix of contamination P df .

[0124] Figure 8 Showing the corresponding Figure 7 The original underwater image of (7b) is segmented by the Segformer model.

[0125] Step 5) Generate a set of suggested regions and suggested points based on the contamination probability matrix and the background probability matrix.

[0126] Specifically, such as Figure 5 As shown, step 5) includes the following steps:

[0127] Step 5-1) Extract the background probability matrix P bg Pixels with a probability greater than the first threshold α:

[0128] S bg ={(x,y)|P bg (x,y)>α},

[0129] The pixel S bg The set constitutes the background suggestion region X bg X bg ={x1,x2,...,x h}

[0130] Background suggestion area X bgThe pixels in the dataset are clustered using the K-Means algorithm based on their coordinates (x, y) to obtain multiple clusters. The center of each cluster is used as the background pixel, labeled 0, and serves as a negative sample. The set of negative samples constitutes the background suggestion point set. Details are as follows:

[0131] From X bg k points are selected as initial cluster centers:

[0132] C bg ={C1,c2,.........,c l},c l ∈X bg l = 1, 2, ..., k

[0133] Calculate the Euclidean distance from each data point to all cluster centers, and assign each data point to the cluster S corresponding to the cluster center with the shortest Euclidean distance. l :

[0134]

[0135] After assigning each data point to the corresponding cluster, update the cluster center of each cluster:

[0136]

[0137] Among them |S l | represents cluster S l Number of data points in the middle.

[0138] Then repeat the above operation of updating the cluster centers of each cluster to minimize the following objective function:

[0139]

[0140] That is, to minimize the sum of squared errors of all data points to the center of their respective clusters.

[0141] After minimizing the objective function, a set of center coordinates for multiple clusters is obtained. Labels are then assigned to these cluster centers. The set of background cluster center points is defined as negative sample points, with a label set to 0.

[0142] L bg ={(c l ,0)|l=1,2,...,k},

[0143] Among them, L bg Set as background suggestion point set.

[0144] Step 5-2) Extract the contamination probability matrix P df Pixels with a probability greater than the second threshold β:

[0145] S df ={(x,y)|P df (x,y)>β},

[0146] The pixel S df The resulting set is the suggested soiling area X. df X df ={x1,x2,...,x n}

[0147] Recommended area X for soiling df The pixels in the dataset are clustered using the K-Means algorithm based on their coordinates (x, y) to obtain multiple clusters. The center of each cluster is used as the pixel representing the background, labeled as label 1, and used as a positive sample. The set of positive samples constitutes the dirty suggestion point set. Details are as follows:

[0148] from df Select k points as initial cluster centers:

[0149] C df ={c1,c2,.........,c j},c j ∈X df j = 1, 2, ..., k

[0150] Calculate the Euclidean distance from each data point to all cluster centers, and assign each data point to the cluster S corresponding to the cluster center with the shortest Euclidean distance. j :

[0151]

[0152] After assigning each data point to the corresponding cluster, update the cluster center of each cluster:

[0153]

[0154] Among them |S j | represents cluster S j Number of data points in the middle.

[0155] Then repeat the above operation of updating the cluster centers of each cluster to minimize the following objective function:

[0156]

[0157] That is, to minimize the sum of squared errors of all data points to the center of their respective clusters.

[0158] After the minimization of the objective function is completed, a plurality of cluster center coordinate sets are obtained, and labels are set for the cluster centers. A set of cluster center points of the stain is defined as positive sample points, and the label is set to 1:

[0159] L df = {(c j ,1) | j = 1, 2,..., k},

[0160] wherein L df is set as the stain suggestion point set.

[0161] In a preferred embodiment, the first threshold value a and the second threshold value β are determined based on an adaptive threshold method, as shown in the following steps: Figure 5

[0162] S1, the background probability matrix P bg and the stain probability matrix P df are respectively converted into probability distribution histograms, and the gradient and the difference value of the histogram are calculated, wherein the gradient is the change speed of the density value corresponding to the probability value with respect to the probability value, indicating the change rate of the probability distribution, and the difference value is the change of the density value between adjacent probability values, i.e. the absolute difference between each pair of adjacent probability values, reflecting the amplitude of the probability change.

[0163] The probability distribution histogram is represented as:

[0164]

[0165] wherein N bg and N df respectively represent the number of pixel points in the background and the stain probability matrix, I(x) is an indicator function, H(p) represents the histogram value corresponding to the probability p, represents the density of the number of pixel points corresponding to the probability p, and Λ represents the original image.

[0166] The gradient is calculated based on the probability histogram, representing the change rate of the probability distribution grad:

[0167]

[0168] The difference value is calculated based on the probability histogram, and the difference value reflects the change amplitude value diff between adjacent probability values:

[0169] diff(H(p)) = |H(p) - H(p-1)|

[0170] wherein H(p+1) and H(p-1) are respectively the histogram values of the previous and next probability intervals corresponding to the probability p.

[0171] S2, according to the change of the gradient and the difference value, the middle range of the probability distribution is screened, and the middle range is obtained as [mid start ​mid end ]。

[0172] The absolute change amplitudes ΔG(x,y) and ΔD(x,y) of the gradient and the difference value are calculated:

[0173] ΔG(x,y) = |G(x,y) - G mean |,

[0174] ΔD(x,y) = |D(x,y) - D mean |,

[0175] where G mean and D mean are the global average values of the gradient and the difference value, respectively.

[0176] According to the size of the value, ΔG(x,y) and ΔD(x,y) are arranged from low to high, and the 90th percentile values P 90 (G) and P 90 (D) of the gradient and the difference value change amplitudes are calculated:

[0177] P 90 (G) = Percentile(ΔG, 90), P 90 (D) = Percentile(ΔD, 90),

[0178] The region that meets the condition is screened out: {(x,y) | ΔG(x,y) < P 90 (G) and ΔD(x,y) < P 90 (D)}.

[0179] The above-mentioned interval where the gradient and the difference value change amplitudes are below the 90th percentile is set as the middle range by calculating the gradient and the difference value of the histogram. The middle range is screened out according to the gradient and the difference value change as follows:

[0180] mid start ≤ p ≤ mid end ,

[0181] where mid start and mid end represent the start and end values of the screened middle probability interval, respectively. This range is the relatively smooth and low change amplitude part in the probability distribution.

[0182] S3, the right end point mid end of the middle range obtained based on the background probability matrix is taken as the first threshold α, and the left end point mid start of the middle range obtained based on the contamination probability matrix is taken as the second threshold β.

[0183] In this embodiment, corresponding toFigure 8 The probability distribution histogram is as follows Figure 9 As shown.

[0184] Through visual operations, X df X bg Displayed as a heatmap corresponding to the original image, such as Figure 10 As shown in the right image, the red area represents the suggested smudged area, the blue area represents the suggested background area, and the yellow area represents the uncertain area, requiring expert intervention for judgment. The uncertain pixel x... w The following conditions must be met:

[0185]

[0186] Recommendation point set L df With L bg Visualization on the original image, such as Figure 10 As shown in the left figure, red dots represent positive samples and blue dots represent negative samples.

[0187] In this step, all cluster centers obtained through clustering operations constitute the proposal point set. This method effectively improves the representativeness of the proposal point set and significantly enhances the efficiency and accuracy of subsequent processing by reducing the number of pixels within the set. Simultaneously, the use of an adaptive thresholding method for threshold selection dynamically adjusts the threshold based on the characteristics of different images, avoiding the problem of insufficient applicability of fixed thresholds to different samples. This enhances the robustness of the method in complex scenes and the accuracy of the segmentation results.

[0188] Step 6) Manually select points based on the suggested region and suggested point set, and identify areas to be ignored.

[0189] Specifically, step 6) includes the following steps:

[0190] Step 6-1) Visualize the suggested point set on the original underwater image. Based on the visualization results, discard the labeled points in the suggested point set that do not match the actual data, and obtain a new suggested point set R. new ;

[0191] Step 6-2) Based on the new suggestion point set R new Eight points are manually selected within the suggested area and labeled accordingly, forming the first set of prompt points (prompt1).

[0192] prompt1 = {p i |p i ∈R sug ,1≤i≤n1},n1=8,

[0193] R sug =S df ∪S bg,

[0194] Where n1 represents the number of selected points prompt1, R sug Indicates the suggested area.

[0195] According to p i Set the label for the relevant suggestion area. i If p i This area is considered a soiling suggestion area. i =1; if p i Belongs to the background area, l i =0.

[0196] Step 6-3) Manually filter the areas R in Step 5) that were not classified as suggested areas but actually belong to suggested areas. ig (i.e., the uncertain region):

[0197]

[0198] Where Ψ represents the set of the actual soiled area and the background area in the original image, and R... sug This is the suggested area obtained in step 5).

[0199] If an ignored region exists, then in the ignored region r i ∈R ig Five points are selected as the second set of prompt points (prompt2):

[0200] prompt2={p k |p k ∈R ig ,1≤k≤n2},n2=5,

[0201] Otherwise, in the suggested region R sug Three additional points are selected as the second set of prompt points (prompt2), and these selected points are different from those in the first set of prompt points.

[0202] prompt2={p k |p k ∈R sug ,1≤k≤n2},n2=3,

[0203] And according to p k Based on the actual situation of the area in the original image, set its label l. k If p k This area is contaminated. k =1; if p k Belongs to the background area, l k =0.

[0204] Step 6-4) Combine the first prompt point set prompt1 and the second prompt point set prompt2 as the prompt point prompt for the large model interactive guided update in step 7): prompt = prompt1 U prompt2, where,

[0205] prompt1 = {p i ∣p i ∈ R sug , 1≤i≤n1}, n1 = 5;

[0206] prompt2 = {p k ∣p k ∈ R ig , 1≤k≤n2}, n2 = {3, 5}.

[0207] In an embodiment, the prompt visualization is shown in the original image as Figure 11 , where red represents positive sample points and blue represents negative sample points.

[0208] In this step, the proposed point set is visualized and superimposed on the original image, so that the expert can perform point selection operations in the visualized proposed area and further identify the ignored area. By visualizing the proposed area and the proposed point set, the expert can intuitively understand the segmentation result generated by the Segformer segmentation model, quickly identify the key area and avoid missing. In the process of selecting points and identifying ignored areas, the expert combines the original image information and domain knowledge to effectively make up for the shortcomings of the model in special scenarios or boundary processing, thereby improving the comprehensiveness and accuracy of the labeled data, and providing more reliable high-quality input data for subsequent model optimization.

[0209] Step 7) Input the points selected in step 6) as the prompt point for the large model interactive guided update, together with the corresponding original underwater image, into the large model SAM2 to obtain a new segmentation result.

[0210] Input the prompt and the original image into the SAM2 model to obtain a new segmentation result. Let the original image be I, input the prompt point set prompt and I into the SAM2 model M SAM2 :

[0211] S new = M SAM2 (I, prompt),

[0212] where S new is the new segmentation result, which is a matrix consistent with the size of the input image I, and the pixel value s(x, y) represents the segmentation label of the coordinate (x, y):

[0213] s(x, y) = 1 if (x, y) belongs to the fouling area;

[0214] s(x, y) = 0, if (x, y) belongs to the background region.

[0215] In one embodiment, the new segmentation result obtained is as shown in Figure 12 .

[0216] This step takes advantage of the powerful segmentation capability of the SAM2 model to automatically generate high-quality segmentation results, significantly reducing the workload of manual annotation. At the same time, the interactive guidance mechanism enables effective supervision and adjustment of the segmentation results output by the model, especially in complex or boundary ambiguous regions, combining expert knowledge to further improve the accuracy and consistency of annotation. Through this guided update process, the dataset annotation can be dynamically optimized, improving the overall quality of the dataset and providing more reliable input data for subsequent model training, thereby significantly enhancing the performance of the segmentation model in practical applications.

[0217] Step 8) as shown in Figure 6 , the new segmentation result is taken as a new annotation file GT', and the new annotation file GT' and the original annotation file GT are compared. If the comparison result meets the preset condition, the iteration is ended, and the new annotation file GT' is taken as the final output of the underwater ship hull non-destructive annotation. Otherwise, return to step 3), based on the new annotation file GT' and the original underwater image I, a new training dataset is formed, and the segmentation model is retrained for the next iteration.

[0218] The new training dataset D' is represented as:

[0219]

[0220] Define the number of iterations as k, the annotation file of the kth iteration as GT', and prompt (k) represent the prompt obtained after the kth iteration, and repeat the above process for D'. The recursive expression is:

[0221] GT (k+1) = M SAM2 (I, prompt (k) ),

[0222] The original manual annotation GT is constant and can provide a stable reference as a benchmark. The SAM2 output result is dynamically changing, and by comparing it with the constant GT, the improvement trend of the model in the optimization process can be reflected.

[0223] Specifically, as shown in Figure 6 , the judgment condition for whether the iteration is ended is:

[0224] The new segmentation result is compared with the original annotation file, and segmentation evaluation indexes are calculated, including mean intersection over union (mIoU), mean Dice coefficient (mDice) and class average pixel accuracy (mPA);

[0225] The mIoU, mDice and mPA obtained after each iteration are saved, and the incremental change rate of the segmentation evaluation indexes of the current iteration relative to the last iteration is calculated:

[0226] ΔmIoU = |mIoU n -mIoU n-1 |,

[0227] ΔmDice = |mDice n -mDice n-1 |,

[0228] ΔmPA = |mPA n -mPA n-1 |,

[0229] wherein n represents the number of iterations.

[0230] If the change rate Δm of all indexes after a certain iteration is lower than a set threshold σ, it is recorded as one "satisfying the convergence condition". If the condition is satisfied continuously b times, the iteration is terminated. The set threshold σ is set by using the percentile method, and the 90th percentile of Δm in the history iteration is calculated as the threshold:

[0231] σ n = percentile(Δm, 90),

[0232] In this way, large abnormal changes can be filtered out, and only normal fluctuation ranges are concerned. In the embodiment, b is selected as 3.

[0233] In an embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded into a random access memory (RAM) from a storage unit. Various programs and data required for device operation can also be stored in the RAM. The computing unit, the ROM and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0234] A plurality of components in the electronic device are connected to the I / O interface, including an input unit such as a keyboard, a mouse, etc., an output unit such as various types of displays, a speaker, etc., a storage unit such as a magnetic disk, an optical disk, etc., and a communication unit such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0235] The computing unit can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit executes various methods and processes described above, such as the methods of image processing and / or the methods of model training. For example, in some embodiments, the methods of image processing and / or the methods of model training can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via the ROM and / or the communication unit. When the computer program is loaded onto the RAM and executed by the computing unit, one or more steps of the methods of image processing and / or the methods of model training described above can be performed. Alternatively, in other embodiments, the computing unit can be configured, by way of firmware or otherwise, to execute the methods of image processing and / or the methods of model training.

[0236] Program code for carrying out methods of the present disclosure can written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0238] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0239] The above description is merely that of a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for underwater hull fouling annotation optimization based on multi-modal large model guidance, characterized in that, The method comprises the following steps: Step 1) obtaining an existing underwater fouling data set, wherein the underwater fouling data set comprises an original underwater image and an original annotation file; Step 2) performing channel conversion on the original annotation file through mapping to convert a three-channel annotation file into a single channel; Step 3) training a segmentation model by using the annotation file after channel conversion and the original underwater image; Step 4) using the weight file obtained in step 3) and taking the original underwater image as a test image, capturing an output probability score corresponding to each test image at a decoding head of the segmentation model, and converting the output probability score into a fouling probability matrix and a background probability matrix output by the test image; Step 5) generating a suggestion region and a suggestion point set based on the fouling probability matrix and the background probability matrix; Step 6) performing manual point selection and ignoring region discrimination based on the suggestion region and the suggestion point set; Step 7) taking the points selected in step 6) as prompt points for interactive guided updating of a large model, inputting the prompt points and the corresponding original underwater image into the large model to obtain a new segmentation result; Step 8) taking the new segmentation result as a new annotation file, comparing the new annotation file with the original annotation file, and if the comparison result meets a preset condition, ending iteration, taking the new annotation file as a final underwater hull nondestructive annotation output, otherwise, returning to step 3), retraining the segmentation model based on the new annotation file and the original underwater image, and performing next iteration.

2. The method of claim 1, wherein, In step 2), a white region in the annotation file is mapped to label 1, indicating fouling, and a black region is mapped to 0, indicating background.

3. The method of claim 1, wherein, Step 4) comprises the following steps: Step 4-1) capturing a prediction score matrix of each test image at a decoding head of the segmentation model, inputting the prediction score matrix into a Softmax function to convert the prediction score matrix into an output probability score corresponding to each pixel point, and obtaining a total probability matrix of fouling and background; Step 4-2) extracting a fouling probability matrix and a background probability matrix from the total probability matrix through a tensor segmentation statement.

4. The method of claim 1, wherein, Step 5) comprises the following steps: Step 5-1) extracting a pixel point with a probability greater than a first threshold value in the background probability matrix, wherein the pixel point constitutes a background suggestion region; clustering the pixel points in the background suggestion region based on coordinate positions to obtain multiple clusters, taking a center of each cluster as a pixel point representing background, setting the pixel point as label 0, and taking the label 0 as a negative sample, wherein a point set constituted by the negative sample is a background suggestion point set; Step 5-2) extracting a pixel point with a probability greater than a second threshold value in the fouling probability matrix, wherein the pixel point constitutes a fouling suggestion region; clustering the pixel points in the fouling suggestion region based on coordinate positions to obtain multiple clusters, taking a center of each cluster as a pixel point representing background, setting the pixel point as label 1, and taking the label 1 as a positive sample, wherein a point set constituted by the positive sample is a fouling suggestion point set.

5. The method of claim 4, wherein, The first threshold value and the second threshold value are determined based on an adaptive threshold method, and the steps are as follows: The background probability matrix and the stain probability matrix are respectively converted into probability distribution histograms, and the gradient and the difference value of the histogram are calculated, wherein the gradient is the change speed of the density value corresponding to the probability value with respect to the probability value, and the difference value is the change of the density value between adjacent probability values, i.e., the absolute difference between each pair of adjacent probability values; According to the change of the gradient and the difference value, the middle range of the probability distribution is screened; The right end point of the middle range obtained based on the background probability matrix is taken as the first threshold, and the left end point of the middle range obtained based on the stain probability matrix is taken as the second threshold.

6. The method of claim 1, wherein, The step 6) comprises the following steps: Step 6-1) visualizing the suggestion point set on the original underwater image, and discarding the label points in the suggestion point set that do not conform to the actual situation according to the visualization result to obtain a new suggestion point set; Step 6-2) based on the new suggestion point set, manually selecting a first preset number of points in the suggestion area and setting corresponding labels as a first hint point set; Step 6-3) manually screening the ignored area that is not classified as the suggestion area in step 5) but actually belongs to the suggestion area, if the ignored area exists, selecting a second preset number of points in the ignored area and setting corresponding labels as a second hint point set, otherwise, selecting a third preset number of points in the suggestion area, the selected points being different from the points in the first hint point set, and setting corresponding labels as the second hint point set; Step 6-4) combining the first hint point set and the second hint point set as the hint points for the interactive guided update of the large model in step 7).

7. The method of claim 1, wherein, In the step 8), the judgment condition for whether the iteration is ended is: The new segmentation result and the original annotation file are compared to calculate a segmentation evaluation index, and the segmentation evaluation index comprises an average intersection over union, an average Dice coefficient and a class average pixel accuracy; The incremental change rate of the segmentation evaluation index of the current iteration with respect to the last iteration is calculated, if the incremental change rate of each segmentation evaluation index is continuously lower than a third threshold for a preset number of times, the preset condition is met, the iteration is terminated, otherwise, the next iteration is performed.

8. The method of claim 7, wherein, The third threshold is set by using a percentile method and is set as the 90th percentile of the incremental change rate of the segmentation evaluation index in the historical iterations.

9. The method of claim 1, wherein, The segmentation model adopts a Segformer segmentation model.

10. The method of claim 1, wherein, The large model adopts a SAM2 model.

Images