A method and system for automated detection of surface rust and coating defects on a marine vessel

Abstract

This invention provides an automated detection method and system for ship surface corrosion and coating defects, comprising the following steps: defining a scene text class set and generating fine-grained descriptions using a large language model, and constructing positive and negative attribute centers; constructing a dual-stream visual network to extract high-level semantics and fine-grained features respectively, and generating anti-interference text classifier weights based on the attribute centers; inputting the fused visual features and text weights into a decoder, and outputting a prediction logic graph through the encoder; constructing a global affinity matrix based on fine-grained features, and obtaining refined pseudo-labels with edge fitting based on the global affinity matrix; the beneficial effects of this invention are: this invention can utilize the semantic generation capability of a large language model and the strong feature expression capability of a visual basic model to achieve fine-grained localization and segmentation of targets such as metal corrosion and coating defects with only image-level weak supervision annotation, effectively suppressing interference from complex backgrounds such as water surface reflection and quay bridge structures.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to an automated method and system for detecting rust and coating defects on ship surfaces. Background Technology

[0002] As the primary carriers of maritime transport, ships are constantly exposed to the harsh marine environment of high salinity and high humidity, making their hull plating highly susceptible to surface defects such as rust and coating peeling. If these defects are not detected and repaired in a timely manner, they will accelerate the corrosion fatigue of the hull structure, seriously threatening navigational safety and potentially causing marine pollution due to paint peeling. Therefore, regular and accurate surface defect inspection of ship hull plating is a crucial aspect of ship maintenance.

[0003] Current ship surface defect detection technologies are mainly divided into three categories: manual inspection, wall-climbing robot inspection, and inspection based on traditional machine vision. However, all of them have significant limitations in practical applications: traditional manual visual inspection or operation using "spider-man" suspended platforms suffers from high labor intensity, high operational risks, and a high risk of missed detections. While magnetic adsorption or negative pressure adsorption wall-climbing robots, which have emerged in recent years, have replaced manual labor to some extent, their equipment deployment is complex and prone to slippage or detachment. More importantly, the aforementioned contact-based inspection methods usually require ships to be stationed for extended periods, occupying terminal resources and making it impossible to conduct "unobtrusive" inspections while ships are berthed for container loading and unloading operations, severely impacting port turnover efficiency.

[0004] With the development of deep learning technology, automated detection methods based on convolutional neural networks have been widely used. However, these fully supervised learning methods heavily rely on massive amounts of high-quality pixel-level labeled data. Due to the highly irregular shape and fragmented edges of rusted areas on ships, the time and economic costs of manually labeling pixels are extremely high. Furthermore, fully supervised models often experience a significant drop in generalization ability when faced with complex scenes outside the training set.

[0005] To reduce annotation costs, weakly supervised semantic segmentation methods based on image-level labels have gradually become a research hotspot. Recently, methods combining visual-language pre-trained models such as CLIP have demonstrated strong zero-shot or few-shot recognition capabilities. However, directly applying existing CLIP or self-supervised models to ship detection in port quay crane scenarios still faces two major technical challenges: First, insufficient utilization of feature complementarity. CLIP models excel at understanding semantics but have weak spatial localization capabilities; while self-supervised models such as DINO excel at capturing geometric details but lack semantic discriminative power. Existing methods often struggle to efficiently integrate the advantages of these two types of frozen parameter models. Second, weak resistance to environmental interference. In the actual operating environment of port quay cranes, the ship's surface is often accompanied by strong water surface reflection and shadow occlusion. Most existing weakly supervised methods rely on shallow color or texture affinity to generate pseudo-labels. When encountering reflections that cause texture loss or shadows that cause color abrupt changes, the detection results often show breaks, holes, or large-area false alarms, making it difficult to generate connected and complete defect region masks.

[0006] Therefore, there is an urgent need for an automated method for detecting surface defects on ships that can utilize existing quay crane facilities for non-contact inspection, requires no large amount of pixel-level annotation, and can effectively overcome interference from water surface reflection and shadows. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an automated detection method and system for ship surface corrosion and coating defects, which solves the problems of low efficiency of manual inspection, high risk of high-altitude operation, and high dependence on pixel-level annotation and high annotation cost of fully supervised deep learning methods in the existing port quay crane scenario for ship surface defect detection.

[0008] To achieve the above and other related objectives, the present invention provides the following technical solution:

[0009] An automated detection method for rust and coating defects on ship surfaces, applied to port quay crane scenarios, includes the following steps: defining a scene text class set containing foreground defect classes and background interference classes; using a large language model to generate category descriptions based on the scene text class set; and constructing positive and negative attribute centers based on the category descriptions; acquiring images to be detected collected by port monitoring equipment and determining the category to which the images belong; and obtaining high-level semantic features, fine-grained features, and final text classifier weights based on the images to be detected and their categories, respectively.

[0010] The high-level semantic features and fine-grained features are concatenated and fused along the channel dimension. The fused visual features and text classifier weights are then input into the decoder to obtain the prediction logic graph corresponding to the image to be detected. A global affinity matrix is ​​constructed based on the fine-grained features, and the prediction logic graph is subjected to feature propagation and sharpening processing based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected.

[0011] An automated detection system for rust and coating defects on ship surfaces includes: a text class definition module, used to define a scene text class containing foreground defect class and background interference class, generate category descriptions based on the scene text class using a large language model, and construct positive and negative attribute centers based on the category descriptions; and a visual feature extraction module, used to acquire images to be detected collected by port monitoring equipment and determine the category to which the acquired images to be detected belong, and obtain high-level semantic features, fine-grained features, and final text classifier weights based on the images to be detected and their categories.

[0012] The initial prediction module is used to concatenate and fuse the high-level semantic features and fine-grained features along the channel dimension, and input the fused visual features and the text classifier weights into the decoder to obtain the prediction logic graph corresponding to the image to be detected. The defect detection module is used to construct a global affinity matrix based on the fine-grained features, and perform feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected.

[0013] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the automated detection method for rust and coating defects on ship surfaces as described above.

[0014] In one embodiment of the present invention, the definition includes a scene text class set containing foreground defect classes and background interference classes. A large language model is used to generate category descriptions based on the scene text class set, and positive and negative attribute centers are constructed based on the category descriptions. This includes: defining a scene text class set containing foreground defect classes and background interference classes, wherein the foreground defect class is a positive detection category, and the background interference class is a negative interference category; using a large language model to generate detailed text descriptions corresponding to each category, and obtaining a positive description set and a negative description set based on the detailed text descriptions; encoding the positive and negative description sets, and constructing positive and negative attribute centers based on the encoded description sets for subsequent feature space alignment.

[0015] In one embodiment of the present invention, the step of acquiring the image to be detected collected by the port monitoring equipment and determining the category to which the image to be detected belongs, and obtaining high-level semantic features, fine-grained features, and final text classifier weights based on the image to be detected and its category, includes: acquiring the image to be detected collected by the port monitoring equipment and determining the category to which the image to be detected belongs; constructing a dual-stream visual feature extraction network to extract high-level semantic features and fine-grained features of the image to be detected; simultaneously performing attribute search enhancement based on the category to which the image to be detected belongs based on the attribute center, and performing feature orthogonalization suppression based on the negative attribute center to generate the final text classifier weights.

[0016] In one embodiment of the present invention, the construction of the dual-stream visual feature extraction network to extract high-level semantic features and fine-grained features of the image to be detected includes: constructing a dual-stream network containing a first visual encoder and a second visual encoder; dividing the image to be detected into a fixed-size Patch sequence and projecting it; extracting global features containing high-level semantics through the first visual encoder; and extracting fine-grained features containing object boundary information through the parameter-frozen second visual encoder.

[0017] In one embodiment of the present invention, the step of performing attribute search enhancement based on the attribute center to determine the category of the image to be detected, and performing feature orthogonalization suppression based on the negative attribute center to generate the final text classifier weights includes: determining the similarity between the original features of the category of the image to be detected and the positive attribute center, and performing weighted fusion, and obtaining enhanced features based on the weighted fusion result; determining the similarity between the enhanced features and the negative attribute center, and comparing the similarity with a preset threshold, and obtaining the final text classifier weights based on the comparison result.

[0018] In one embodiment of the present invention, the step of concatenating and fusing the high-level semantic features and fine-grained features along the channel dimension, and inputting the fused visual features and text classifier weights into a decoder to obtain a prediction logic graph corresponding to the image to be detected, includes: concatenating and fusing the high-level semantic features and fine-grained features along the channel dimension, and mapping them to a unified feature space through a linear layer to obtain fused visual features; inputting the fused visual features and final text classifier weights into a decoder to obtain an updated category embedding vector, and performing a dot product between the category embedding vector and the fused visual features to obtain a prediction logic graph corresponding to the image to be detected based on the dot product result.

[0019] In one embodiment of the present invention, the step of constructing a global affinity matrix based on the fine-grained features and performing feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected includes: extracting the output features of the last layer from the parameter-frozen second visual encoder and constructing a full-image pixel feature matrix based on the extracted output features of the last layer.

[0020] The cosine similarity of feature vectors between any two pixel positions in the full-image pixel feature matrix is ​​determined, and a global affinity matrix reflecting the correlation probability between pixels is obtained based on the cosine similarity of feature vectors. The prediction logic graph is flattened, and matrix multiplication is performed on the flattened prediction logic graph based on the global affinity matrix. The refined pseudo-labels of edge fitting are obtained based on the matrix multiplication results, which correspond to the defect detection results of the image to be detected.

[0021] In one embodiment of the present invention, the method further includes the following steps: based on a weakly supervised learning paradigm, and according to the prediction logic graph, refined pseudo-labels and global affinity matrix, constructing a total loss function that includes image-level classification loss, semantic consistency segmentation loss and structural affinity loss, and performing end-to-end optimization training on the defect detection model according to the total loss function.

[0022] As described above, the automated detection method and system for rust and coating defects on ship surfaces of the present invention has the following beneficial effects:

[0023] 1. This invention has text semantic enhancement and anti-interference capabilities: This invention introduces a large language model to generate fine-grained descriptions and construct attribute centers. Through the feature orthogonalization mechanism, it explicitly suppresses background noise such as water surface reflection and quay bridge shadows that are common in port scenes, which significantly improves the robustness of the classifier in complex environments.

[0024] 2. Dual-stream visual feature complementarity of the present invention: The present invention combines CLIP’s high-level semantic understanding capability with DINOv3’s fine-grained boundary perception capability, which solves the problem that traditional weak supervision methods can only locate the most discriminative part of the target and cannot completely segment the target (such as the whole rust patch);

[0025] 3. Fine segmentation without pixel-level annotation: This invention utilizes the inherent affinity characteristics of self-supervised visual models to construct a global association matrix and automatically generates high-quality pseudo-labels through feature propagation technology. It achieves near-fully supervised level defect segmentation accuracy using only image-level labels, significantly reducing implementation costs.

[0026] 4. This invention enables end-to-end weakly supervised optimization: This invention proposes a joint loss function that includes classification, segmentation consistency and structural affinity distillation, enabling the model to simultaneously optimize class recognition and boundary localization capabilities in a single training process, avoiding the cumbersome and error accumulation of traditional multi-stage training. Attached Figure Description

[0027] Figure 1 The flowchart shown is a method for automated detection of rust and coating defects on ship surfaces according to the first embodiment of the present invention.

[0028] Figure 2 The diagram shown is an automated detection system for ship surface corrosion and coating defects according to the second embodiment of the present invention.

[0029] Figure 3 The diagram shown is an electronic device according to the third embodiment of the present invention. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0031] The first embodiment of the present invention relates to an automated detection method for rust and coating defects on ship surfaces, specifically the process as follows: Figure 1 As shown, the details are as follows:

[0032] Step 101: Define a scene text class set containing foreground defect class and background interference class, use a large language model to generate category descriptions based on the scene text class set, and construct positive and negative attribute centers based on the category descriptions.

[0033] Specifically, a set of scene text classes is defined, including foreground defect classes and background interference classes. A large language model is used to generate category descriptions, and positive and negative attribute centers are constructed through a text encoder and clustering algorithm.

[0034] More specifically, step S101 includes: S1.1 Defining a scene-related text category set, which is divided into a foreground category and a background category. The foreground category includes positive detection categories such as metal corrosion and coating defects, while the background category includes negative interference categories such as stains, balcony structure occlusion, and water surface reflection; S1.2 Using a large language model to generate detailed text descriptions containing appearance, texture, and color features for each category, thereby forming a positive description set and a negative description set; S1.3 Encoding the description set using the CLIP text encoder and using the K-means algorithm to cluster and generate multiple positive attribute centers and negative attribute centers for subsequent feature space alignment.

[0035] It should be noted that the specific process of step S101 is as follows: S1.1 Define the scene-related text class set It is classified into a positive foreground category. (Including metal corrosion, coating defects, etc.) and negative background categories (Including stains, obstructions from quay bridge structures, water surface reflections, etc.); S1.2 utilizes advanced large language models for each category. Generate a detailed text description containing appearance, texture, and color features, resulting in a foreground class as a positive description set. And the background class is a negative description set S1.3 The description set is encoded using a pre-trained text encoder, and a clustering algorithm is used to generate M positive attribute centers in the feature space. With N negative attribute centers Based on this, a knowledge base containing rich semantic prototypes is constructed using positive and negative attribute centers.

[0036] In practical applications, step S101 is specifically as follows: S1.1 Define a set of scene-related text classes, including: foreground defect classes (positive class set) ): Metal corrosion, coating peeling, coating blistering, coating cracking, intact coating; Background interference (negative category) The system includes 20 detection categories: stains, biological adhesions, safety signs, wood, ropes, quay bridge structure obstructions, welds, rubber scratches, paint runs, exterior panel scratches (without exposed substrate), dust cover, water surface highlights, floating objects, waves, and reflections. S1.2 utilizes a large language model to generate category descriptions. For example, for "rust," it generates descriptions such as "rough patches of reddish-brown or orange," "uneven peeling surface," and "usually accompanied by runs." For "intact coating," it generates descriptions such as "smooth and glossy surface," "uniform color," and "no cracks or bubbles." This foreground category is a positive description set. The background class is a negative description set. S1.3 Utilize a pre-trained CLIP text encoder to... and Encoding yields preliminary text features M positive attribute centers are generated using the k-means algorithm. N negative attribute centers These centers represent typical semantic prototypes of various categories.

[0037] Step 102: Obtain the image to be detected collected by the port monitoring equipment and determine the category to which the image to be detected belongs. Based on the image to be detected and its category, obtain the high-level semantic features, fine-grained features and the final text classifier weights.

[0038] Specifically, a dual-stream visual feature extraction network is constructed to extract high-level semantic features and fine-grained features of the image to be detected. At the same time, attribute centers are used to perform attribute search enhancement for the category to which the image to be detected belongs, and feature orthogonalization suppression is performed based on negative attribute centers to generate the final text classifier weights.

[0039] More specifically, step S102 includes: S2.1, performing attribute search, calculating the similarity between the original features of the input category and the positive attribute center, and weighted fusion; performing feature orthogonalization suppression, calculating the similarity between the features and the negative attribute center, and when the similarity exceeds a preset threshold, using a non-linear activation function and suppression strength parameter to subtract the features to suppress background semantic noise, and obtaining the final text classifier weights; S2.2, constructing a two-stream network containing a first visual encoder and a second visual encoder; dividing the image to be detected into fixed-size patch sequences and projecting them; extracting global features containing high-level semantics through the first visual encoder, and extracting fine-grained features containing object boundary information through the parameter-frozen second visual encoder.

[0040] It should be noted that the specific process of step S102 is as follows: S2.1 On the text side, perform attribute search and orthogonalization suppression: calculate the original features of the input category. The enhanced features are obtained by weighting and fusing the similarity with all positive attribute centers. Further calculations Similarity to negative attribute centers, when the similarity exceeds a preset threshold At that time, a non-linear activation function is used to perform a subtraction operation on the features to remove background semantic noise, thus obtaining the final text classifier weights. S2.2 On the vision side, a two-stream network containing a CLIP visual encoder and a DINOv3 visual encoder is constructed; the image to be detected is segmented into a Patch sequence, and global features containing high-level semantics are extracted through the multi-layer Transformer module of the CLIP visual encoder. Fine-grained features containing object boundary information are extracted using the parameter-frozen DINOv3 visual encoder. .

[0041] In practical applications, the specific process of step S102 is as follows: S2.1 When the "rust" category is input, calculate its original features. The enhanced features are obtained by weighted fusion of cosine similarity with all positive attribute centers. ;calculate Similarity with all negative attribute centers; if the similarity with a certain negative attribute, such as "smooth red paint", exceeds a threshold. Then perform the subtraction operation: , It's a suppression intensity hyperparameter; ReLU ensures that suppression only occurs when the similarity is positive. This refers to the final text classifier weights;

[0042] S2.2 Constructs a two-stream visual feature extraction network containing a first visual encoder and a second visual encoder, and extracts the image to be detected. First, divide the data into N fixed-size patches, where H is the height, W is the width, and the patch size is... Total number of patches The input sequence becomes .

[0043] S2.3 flattens each patch and maps it to the hidden layer dimension D through a learnable linear layer: ,in, It is a projection matrix; a learnable class token is added before the Patch embedding sequence. Add position encoding To preserve spatial information: ;

[0044] The S2.4 first visual encoder first normalizes the input features in each layer through a multi-head attention sublayer, and then fuses the input and attention output through residual connections: Then, the output from the previous step is normalized again through a multilayer perceptron sublayer, and finally summed again through residual connections. After L layers of Transformer Blocks, the final layer of visual features is obtained: Extract the output sequence except Output features corresponding to the N patches outside the token The sequence is rearranged back into a two-dimensional spatial grid to obtain the CLIP feature map: ;

[0045] The processing flow of the S2.5 second visual encoder is similar to CLIP, with the following input vector: After processing through L layers of Transformer blocks, the final layer of features is obtained using DINO's unique attention mechanism: Discard as well Token, extract patch features: .

[0046] Step 103: The high-level semantic features and fine-grained features are concatenated and fused along the channel dimension, and the fused visual features and text classifier weights are input into the decoder. The decoder then obtains the prediction logic graph corresponding to the image to be detected.

[0047] Specifically, the fused visual features and text classifier weights are input into the decoder, and the features are interacted and updated through cross-attention and self-attention mechanisms to output the initial prediction map, which is the prediction logic map.

[0048] More specifically, step S103 includes: S3.1, concatenating CLIP features and DINOv3 features along the channel dimension and mapping them to a unified feature space through a linear layer; S3.2, using the final text classifier weights as learnable queries and the fused visual features as key-value inputs to the decoder; aggregating category-related features in the image through a cross-attention mechanism, simulating co-occurrence relationships between categories through a self-attention mechanism, and obtaining an updated category embedding vector through a feedforward network; S3.3, calculating the dot product of the category embedding vector and the fused visual features to obtain a prediction logic graph.

[0049] It should be noted that the specific process of step S103 is as follows: S3.1 Combine the two visual features and The features are concatenated along the channel dimension and mapped to a unified feature space through a linear projection layer to obtain fused features. S3.2 will fuse features Flatten the values ​​as K and V, and use the weights of the text classifier generated in step S102. The Transformer decoder serves as a learnable query Q-input; in the decoding layer, category-related feature regions in the image are aggregated through a cross-attention mechanism, and the co-occurrence relationship between categories is simulated through a self-attention mechanism. After multiple layers of interaction and updating, the category embedding vector is obtained. S3.3 Calculation With fused visual features The dot product generates an initial prediction logistic graph containing the original class confidence scores. .

[0050] In practical applications, step S103 is specifically as follows: S3.1 The two feature maps are concatenated along the channel dimension: By using a linear projection layer for fusion and dimensionality reduction, heterogeneous features are mapped to a unified feature space. : ,in, It is a learnable weight matrix. It is the GELU activation function. .

[0051] S3.2 Flatten the visual feature map into a sequence format The final text features obtained in step S102 As a learnable query Input decoder, K corresponds to the number of categories, calculate and The similarity is used to aggregate the feature information of each category in the image that belongs to that category: By using self-attention to allow different category query vectors to interact, the co-occurrence relationship of categories can be simulated. The feedforward neural network performs non-linear transformations and dimensionality processing on the query features: After passing through three such Transformer blocks, the final class embedding vector is obtained. .

[0052] S3.3 will finally update the category query vector. With fused visual features Perform the dot product: ,in, Reshape M back Generate a predicted Logits graph. This is the output category logic diagram. Each channel k in the array corresponds to a category, and the pixel value on that channel represents the "raw confidence" that the location belongs to that category.

[0053] Step 104: Construct a global affinity matrix based on the fine-grained features, and perform feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected.

[0054] Specifically, fine-grained visual features are extracted to construct a global affinity matrix, and this matrix is ​​used to perform feature propagation and sharpening on the initial prediction image to generate refined pseudo-labels, which are the defect detection results corresponding to the image to be detected.

[0055] More specifically, step S104 includes: S4.1 Extracting the output features of the last layer from the frozen DINO visual encoder, flattening and transposing them into a feature matrix to represent the deep semantic vectors of all pixels in the image; S4.2 Calculating the cosine similarity of the feature vectors between any two pixel positions in the feature matrix, performing exponential sharpening and row normalization on the similarity matrix to obtain a global affinity matrix that reflects the probability of association between pixels; S4.3 Flattening the prediction logic graph generated in step S103, performing matrix multiplication on it using the global affinity matrix, so that the activation state propagates and diffuses according to pixel similarity, and obtaining refined pseudo-labels with edge fitting after reshaping.

[0056] It should be noted that the specific process of step S104 is as follows: S4.1 Extract the last layer features from the frozen second visual encoder and construct the full-image pixel feature matrix. ; Calculate the positions of any two pixels in the matrix , The cosine similarity between the two pairs of samples is calculated, and the similarity is then subjected to exponential sharpening and normalization to construct a global affinity matrix. S4.2 This affinity matrix The probability weights representing the propagation of semantic activation from one pixel to another are used to represent the prediction logic graph generated in step S103. Flatten out, using this affinity matrix Matrix multiplication is performed on the target data to allow the activation response to spread and propagate based on the semantic similarity between pixels, thereby obtaining refined pseudo-labels with edge-fitting. .

[0057] In practical applications, the specific process of step S104 is as follows: S4.1 Extract the feature map output by the last Transformer block from the frozen DINOv3 visual encoder in step S102 as a reference feature. Its original dimension is ,in, The number of feature channels; Flattening and transposing the matrix in the spatial dimension yields the feature matrix. ,in, This represents the total number of pixels in the image.

[0058] S4.2 Calculate the characteristic matrix Any two pixel positions and To determine the cosine similarity of the eigenvectors between two features, first, L2 normalization is performed on the eigenvectors, and then the dot product is calculated: , where each row in the matrix Representing the Deep feature vectors at each pixel location; similarity matrix Exponential sharpening is performed, and then normalization is applied along the row direction using the Softmax function to obtain the final global affinity matrix. : ,in, , This is the sharpening factor. Indicates from pixel The activation state is propagated to the pixel The probability weights;

[0059] S4.3 The category logic diagram generated in step S103 (dimension is) Similarly, flatten it into a matrix Where K is the number of categories; using the constructed affinity matrix Perform matrix multiplication on the category logic graph and execute feature propagation; refine the pseudo-labels. The calculation formula is as follows: The propagated matrix Reshaping The spatial dimension yields the final refined pseudo-labels.

[0060] In steps 101 to 104, the present invention adopts a weakly supervised learning paradigm, constructs a total loss function that includes image-level classification loss, semantic consistency segmentation loss and structural affinity loss, and performs end-to-end optimization training on the model.

[0061] Specifically, step S105 includes: S5.1, performing weighted global pooling on the predicted logic graph output by the decoder to obtain an image-level predicted vector, calculating the multi-label soft edge loss in combination with the real labels, and using the Log-Sum-Exp pooling function to encourage the model to focus on the overall target region; S5.2, using the refined pseudo-labels generated in step S104 as pseudo-real values ​​to perform pixel-level supervision on the predicted logic graph generated in step S103; supervising the decoder during the calculation process, optimizing only the initial generation network; S5.3, calculating the final affinity matrix corresponding to the fused feature map and the initial affinity matrix corresponding to the DINOv3 reference features; by minimizing the norm distance between the two matrices, transferring the fine-grained association knowledge in the reference features to the fused feature space; S5.4, weighting and summing the above three losses according to the preset balanced weight coefficients, and using the sum as the total loss function for model optimization.

[0062] More specifically, step S105 is as follows: S5.1 Based on the weakly supervised learning paradigm, construct a multi-objective joint loss function. The model is optimized using an image-level multi-label classification loss function. Semantic consistency segmentation loss Distillation loss with structural affinity Weighted composition: S5.2 The initial prediction maps are aggregated into image-level scores using the Log-Sum-Exp pooling function, and then supervised by the ground truth labels to encourage the model to focus on the overall target region; S5.3 Treating the refined pseudo-labels generated in step S104 as pseudo-true values, the initial prediction graph... Pixel-level supervision is performed, and only the initially generated network is optimized using a gradient truncation strategy; S5.4 By minimizing the fusion feature affinity matrix Affinity matrix with reference feature The norm distance between them transfers fine-grained association knowledge from the second visual encoder to the fused feature space.

[0063] In practical applications, step S105 is specifically as follows: S5.1 Based on the initial prediction results generated by the decoder and the pseudo-labels generated by the refining module, calculate the total loss function to supervise image-level category determination and pixel-level semantic consistency; it is composed of a weighted average of image-level multi-label classification loss, semantic consistency segmentation loss, and structural affinity distillation loss. ,in, The primary classification loss is used to establish semantic anchors; The consistency loss is used for boundary calibration. This represents the loss due to affinity distillation, used for characteristic structure constraints. and These are the corresponding balance weight coefficients;

[0064] S5.2 To ensure the model focuses on the overall target rather than the most discriminative local regions, the Log-Sum-Exp pooling function is used to aggregate the prediction maps into image-level scores, and the multi-label soft edge loss is calculated. , ,in, For the converted real labels, This is the hyperparameter for pooling intensity;

[0065] S5.3 Using the refining chart generated in step S104 As the initial prediction for pseudo-truth value supervision : This loss forces the initial prediction network to adaptively learn the fine boundaries discovered by DINOv3;

[0066] S5.4 Affinity matrix calculated using DINOv3 As a source of knowledge, supervision and fusion characteristics Calculated affinity matrix By minimizing the norm distance between the two, fine-grained relational knowledge is transferred to the fused feature space. .

[0067] The second embodiment of the present invention relates to an automated detection system for rust and coating defects on ship surfaces. Please refer to [link to relevant documentation]. Figure 2 ,include:

[0068] The text class definition module is used to define a scene text class that includes foreground defect class and background interference class. The large language model is used to generate category descriptions based on the scene text class, and positive and negative attribute centers are constructed based on the category descriptions.

[0069] The feature extraction module is used to acquire the image to be detected collected by the port monitoring equipment and determine the category to which the image to be detected belongs. Based on the image to be detected and its category, high-level semantic features, fine-grained features and final text classifier weights are obtained respectively.

[0070] The initial prediction module is used to concatenate and fuse high-level semantic features and fine-grained features in the channel dimension, and input the fused visual features and text classifier weights into the decoder. The decoder then obtains the prediction logic graph corresponding to the image to be detected.

[0071] The defect detection module is used to construct a global affinity matrix based on the fine-grained features, and to perform feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix, thereby obtaining the defect detection result corresponding to the image to be detected.

[0072] It is not difficult to see that this embodiment is a system implementation corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0073] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0074] The third embodiment of the present invention relates to an electronic device; please refer to [link / reference]. Figure 3 ,include:

[0075] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described automated detection method for rust and coating defects on ship surfaces.

[0076] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0077] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0078] The fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method embodiments.

[0079] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0080] In summary, this invention can leverage the semantic generation capabilities of large language models and the strong feature representation capabilities of visual foundation models to achieve refined localization and segmentation of targets such as metal corrosion and coating defects with only weak image-level supervision annotation, effectively suppressing interference from complex backgrounds such as water surface reflection and quay bridge structures.

[0081] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. All equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this invention should still be covered by the claims of this invention.

Claims

1. An automated detection method for rust and coating defects on ship surfaces, characterized in that, This method is applied to port quay crane scenarios and includes the following steps: Define a scene text class set containing foreground defect class and background interference class, use a large language model to generate category descriptions based on the scene text class set, and construct positive and negative attribute centers based on the category descriptions; The system acquires images to be detected collected by port monitoring equipment and determines the category to which the images belong. Based on the images to be detected and their categories, it obtains high-level semantic features, fine-grained features, and final text classifier weights. The high-level semantic features and fine-grained features are concatenated and fused along the channel dimension, and the fused visual features and text classifier weights are input into the decoder. The decoder then obtains the prediction logic graph corresponding to the image to be detected. A global affinity matrix is ​​constructed based on the fine-grained features, and the prediction logic graph is subjected to feature propagation and sharpening processing based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected.

2. The automated detection method for rust and coating defects on ship surfaces according to claim 1, characterized in that: The definition includes a set of scene text classes for foreground defects and background interference. A large language model is used to generate category descriptions based on these scene text classes, and positive and negative attribute centers are constructed based on these category descriptions, including: Define a scene text class set that includes foreground defect class and background interference class, wherein the foreground defect class is a positive detection class and the background interference class is a negative interference class; A large language model is used to generate detailed text descriptions for each category, and positive and negative description sets are obtained based on the detailed text descriptions. The positive and negative description sets are encoded, and positive and negative attribute centers are constructed based on the encoded description sets for subsequent feature space alignment.

3. The automated detection method for rust and coating defects on ship surfaces according to claim 1, characterized in that: The process of acquiring the image to be detected collected by the port monitoring equipment and determining the category of the image to be detected, and obtaining high-level semantic features, fine-grained features, and final text classifier weights based on the image to be detected and its category, includes: Acquire images to be detected collected by port monitoring equipment and determine the category to which the images to be detected belong; A dual-stream visual feature extraction network is constructed to extract high-level semantic features and fine-grained features of the image to be detected. At the same time, attribute search enhancement is performed on the category to which the image to be detected belongs based on the attribute center, and feature orthogonalization suppression is performed based on the negative attribute center to generate the final text classifier weights.

4. The automated detection method for rust and coating defects on ship surfaces according to claim 3, characterized in that: The constructed dual-stream visual feature extraction network extracts high-level semantic features and fine-grained features of the image to be detected, including: Construct a two-stream network that includes a first visual encoder and a second visual encoder; The image to be detected is segmented into a fixed-size Patch sequence and projected. Global features containing high-level semantics are extracted by the first visual encoder, and fine-grained features containing object boundary information are extracted by the second visual encoder with parameters frozen.

5. The automated detection method for rust and coating defects on ship surfaces according to claim 3, characterized in that: The step of performing attribute search enhancement based on the attribute center to the category of the image to be detected, and performing feature orthogonalization suppression based on the negative attribute center to generate the final text classifier weights includes: The similarity between the original features of the category to which the image to be detected belongs and the positive attribute centers is determined and weighted fusion is performed. The enhanced features are obtained based on the weighted fusion result. The similarity between the enhanced features and the negative attribute centers is determined, and the similarity is compared with a preset threshold. The final text classifier weights are obtained based on the comparison results.

6. The automated detection method for rust and coating defects on ship surfaces according to claim 1, characterized in that: The step of concatenating and fusing the high-level semantic features and fine-grained features along the channel dimension, and inputting the fused visual features and text classifier weights into the decoder, to obtain the prediction logic graph corresponding to the image to be detected through the decoder, includes: The high-level semantic features and fine-grained features are concatenated and fused along the channel dimension, and then mapped to a unified feature space through a linear layer to obtain the fused visual features. The fused visual features and the final text classifier weights are input into the decoder. The decoder obtains the updated category embedding vector, and the category embedding vector and the fused visual features are multiplied by a dot product. The prediction logic graph corresponding to the image to be detected is obtained based on the dot product result.

7. The automated detection method for rust and coating defects on ship surfaces according to claim 1, characterized in that: The step of constructing a global affinity matrix based on the fine-grained features, and performing feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix to obtain the defect detection result corresponding to the image to be detected, includes: Extract the output features of the last layer from the parameter-frozen second visual encoder, and construct the full-image pixel feature matrix based on the extracted output features of the last layer; Determine the cosine similarity of the feature vectors between any two pixel positions in the full-image pixel feature matrix, and obtain a global affinity matrix reflecting the probability of association between pixels based on the cosine similarity of the feature vectors; The prediction logic graph is flattened, and matrix multiplication is performed on the flattened prediction logic graph according to the global affinity matrix. The refined pseudo-label with edge fitting is obtained according to the matrix multiplication result, which is the defect detection result corresponding to the image to be detected.

8. The automated detection method for rust and coating defects on ship surfaces according to claim 7, characterized in that: It also includes the following steps: Based on the weakly supervised learning paradigm, and constructing a total loss function that includes image-level classification loss, semantic consistency segmentation loss, and structural affinity loss according to the prediction logic graph, refined pseudo-labels, and global affinity matrix, the defect detection model is optimized and trained end-to-end according to the total loss function.

9. An automated detection system for rust and coating defects on ship surfaces, characterized in that: include: The text class definition module is used to define scene text classes that include foreground defect classes and background interference classes. The large language model is used to generate category descriptions based on the scene text classes, and positive and negative attribute centers are constructed based on the category descriptions. The feature extraction module is used to acquire the images to be detected collected by the port monitoring equipment and determine the category to which the collected images to be detected belong. Based on the images to be detected and their categories, high-level semantic features, fine-grained features, and final text classifier weights are obtained respectively. The initial prediction module is used to concatenate and fuse the high-level semantic features and fine-grained features in the channel dimension, and input the fused visual features and the text classifier weights into the decoder, and obtain the prediction logic graph corresponding to the image to be detected through the decoder; The defect detection module is used to construct a global affinity matrix based on the fine-grained features, and to perform feature propagation and sharpening processing on the prediction logic graph based on the global affinity matrix, thereby obtaining the defect detection result corresponding to the image to be detected.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the automated detection method for rust and coating defects on ship surfaces as described in any one of claims 1 to 8.

Images