A method, system and medium for hydraulic metal surface defect segmentation

Abstract

The application provides a hydraulic metal surface defect segmentation method, system and medium, wherein the hydraulic metal surface defect segmentation method comprises the following steps: S1: collecting a hydraulic metal surface original image and completing defect labeling, generating a standardized text prompt corresponding to a defect category based on a unified template, and constructing an image and text paired sample; performing multi-path dynamic feature coding on the original image, performing dimension reduction mapping, dynamic weight generation, bidirectional cross feature fusion and dimension reconstruction, then fusing the original features and the reconstructed features through a gate weighted fusion, and extracting high-dimensional defect features of the image; performing semantic coding on the defect text prompt, generating a text gate weight through adaptive feature adaptation, and completing dimension alignment of the text features and the image features; the application avoids the information homogenization problem easily occurring in single-path feature extraction through lossless flow of defect key information, and reduces the missed detection phenomenon.

Description

Technical Field

[0001] This invention relates to the field of hydraulic metal technology, and in particular to a method, system and medium for segmenting surface defects in hydraulic metal. Background Technology

[0002] Hydraulic metals (steel gates, pressure steel pipes, opening and closing mechanism components, etc.) are load-bearing components of water conservancy projects. They are subjected to high water pressure, water flow erosion, moisture corrosion and alternating load conditions for a long time, and are prone to surface defects such as cracks, rust, peeling and wear, which directly affect the safety of flood control, water supply and power generation of water conservancy projects.

[0003] Currently, defect detection mostly relies on manual visual inspection and conventional non-destructive testing, which suffers from problems such as low efficiency, large subjective judgment errors, easy omission of small and hidden defects, and difficulty in accurately quantifying the defect outline and area. It cannot meet the high-efficiency detection needs of intelligent and routine safe operation and maintenance of water conservancy projects.

[0004] Existing deep learning-based methods for segmenting defects on hydraulic metal surfaces mostly employ a fully supervised training mode, which requires a large amount of manpower to complete pixel-level defect annotation, resulting in high sample costs and long model generalization cycles. Furthermore, single-path feature extraction methods have poor robustness in segmenting defects with varied shapes and low contrast, and are prone to problems such as feature homogenization, blurred segmentation edges, and missed detections when faced with complex background interference. At the same time, they lack flexible semantic guidance mechanisms, and the semantic alignment between segmentation results and defect language descriptions is insufficient, making it difficult to quickly adapt to different defect types according to on-site detection needs.

[0005] To address the aforementioned issues, a method, system, and medium for segmenting defects on the surface of hydraulic metal structures are now designed. Summary of the Invention

[0006] This application provides a method, system, and medium for segmenting defects on the surface of hydraulic metals, in order to solve the problems in related technologies where the segmentation methods for defects on the surface of hydraulic metals rely on a large number of pixel-level annotations, have insufficient robustness in defect feature extraction under complex working conditions, and lack effective semantic guidance and cross-modal semantic alignment mechanisms, making it difficult to balance segmentation accuracy and adaptability to multiple defect types.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a method for segmenting defects on the surface of hydraulic metal structures, comprising the following steps: S1: Collect original images of hydraulic metal surfaces and complete defect annotations. Generate standardized text prompts for corresponding defect categories based on a unified template, and construct image-text pairing samples. S2: Perform multi-path dynamic feature encoding on the original image. After dimensionality reduction mapping, dynamic weight generation, bidirectional cross-feature fusion and dimensionality increase reconstruction, the original features and reconstructed features are fused by gated weighted fusion to extract high-dimensional features of image defects. S3: Perform semantic encoding on defective text prompts, generate text gating weights through adaptive feature adaptation, and fuse the original text features and adapted enhanced features through gating weighting to complete the dimensional alignment of text features and image features; S4: Concatenate learnable mask markers and text prompt markers to construct a decoding input sequence. After mapping the enhanced mask markers to the image channel dimension, perform spatial matching operations with the upsampled image features to output a defect semantic segmentation mask. S5: Construct a weighted multi-task loss function to jointly optimize segmentation accuracy and cross-modal semantic alignment effect, and complete the training of the defect segmentation model.

[0008] Preferably, step S1 includes the following steps: S1.1: Collect images of hydraulic metal surfaces under real service conditions. After screening the original images frame by frame, use professional annotation tools to accurately annotate defect instances to ensure the accuracy and reliability of the dataset. S1.2: Perform data augmentation operations on the selected original hydraulic metal surface images, including rotation, cropping, and horizontal flipping, to simulate the visual appearance of defects under different shooting angles and scales, effectively expanding data diversity. S1.3: For the image dataset of surface defects in hydraulic metals that has been annotated with instances, a corresponding text prompt dataset is constructed simultaneously.

[0009] Preferably, step S2 includes the following steps: S2.1: Input the original image into an image encoder equipped with a dual-path dynamic adapter, wherein the dual-path dynamic adapter is set in parallel at the residual connection position after the multi-head attention layer and the feedforward network of each Transformer block of the image encoder; S2.2: Perform two independent linear dimensionality reduction projection operations on the input image features respectively, and map to obtain two sets of low-dimensional latent features with a channel dimension of one-eighth of the original features; S2.3: The input image features are fed into the dynamic weight generator, which sequentially performs channel linear dimensionality reduction, GELU nonlinear activation, and linear feature enhancement processing, and then outputs two sets of balanced fusion weights that adapt to the low-dimensional latent features along the channel dimension. S2.4: Two sets of fusion weights are used to perform element-wise product modulation on the two low-dimensional latent features respectively, and cross-path addition operation is performed to obtain the modulated low-dimensional features after bidirectional cross-fusion. This symmetrical fusion method can avoid the homogenization of feature information. S2.5: Perform GELU activation and linear upscaling projection on the two modulated low-dimensional features respectively to restore the feature dimensions to the original channel size of the input image features, and obtain two high-dimensional reconstructed features; S2.6: The original image features and the results of the two high-dimensional reconstruction features are spliced ​​along the channel dimension, and after linear transformation, they are evenly distributed along the channel to obtain two sets of gate weights; S2.7: Use two sets of gating weights to perform element-wise weighted mixing of the original image features and the superposition results of the two high-dimensional reconstruction features to obtain the output image features of the current Transformer block.

[0010] Preferably, step S3 includes the following steps: S3.1: Configure several text prompts for each defective image; S3.2: Input the text prompt into the text encoder that has been fine-tuned by the lightweight adapter; S3.3: Perform dimensionality reduction, activation, and dimensionality increase adaptation processing on the input text features in sequence to obtain adapted and increased dimensionality features; concatenate the input text features and the adapted and increased dimensionality features along the channel dimension, and after linear transformation, uniformly divide them along the channel dimension to generate two sets of text gating weights; S3.4: Based on two sets of text gating weights, the input text features and the adapted dimensionality-upgrading features are respectively weighted and modulated element by element. The two weighted results are added together to obtain the output features of the text encoder. S3.5: Perform dimensional transformation on the output features to adapt the text feature dimensions to the image feature dimensions.

[0011] Preferably, step S4 includes the following steps: S4.1: Construct an input tag sequence, which is composed of learnable mask tags and text prompt tags; S4.2: Extract text hints before and after decoding and overlay them to obtain enhanced text tags; calculate the cosine similarity between the enhanced text tags and each mask tag, select the K candidate mask tags with the highest similarity, and perform weighted aggregation on the candidate mask tags to form the mask output tags; S4.3: The mask output marker is processed by 3 layers of MLP mapping to obtain a vector with the same dimension as the feature channel of the upsampled image; the spatial dot product operation is performed between this vector and the upsampled image features to complete the prediction output of the defect mask.

[0012] Preferably, step S5 includes the following steps: S5.1: Select binary cross-entropy loss and contrastive loss as the basic components of the loss function; S5.2: Calculate the text-image contrast loss by comparing and learning the text features with the image features. Perform a softmax operation on the similarity between all image patches and the corresponding text features in each training sample, and calculate the mean after taking the logarithm of the operation results. S5.3: Calculate the text-mask contrast loss. Compare the text fusion label obtained by adding the text features before decoding and the text labels after decoding with the mask output label. Perform softmax operation on the similarity of all text labels in each training sample, and calculate the mean after taking the logarithm of the operation result. S5.4: Build a training environment based on a deep learning framework, set the batch size, and complete model training according to the set maximum iteration cycle; S5.5: Input the original hydraulic metal image to be detected and the matching text prompt into the pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation result that better matches the text prompt.

[0013] A surface defect segmentation system for hydraulic metal structures, comprising: The image acquisition module acquires the original hydraulic metal image; The text prompt acquisition module automatically generates text prompt descriptions based on the corresponding category of defects on the surface of hydraulic metal structures. The model building module is used to build defect segmentation models; The defect segmentation model includes: an image encoder fine-tuned by a dual-path dynamic adapter, a text encoder fine-tuned by a lightweight adapter, and a novel lightweight mask decoder that is more suitable for semantic segmentation. The defect segmentation module is used to input the original hydraulic metal image and text prompts into a pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation results that are more consistent with the text prompts.

[0014] Preferably, the dual-path dynamic adapter is embedded in parallel at the residual connections after the multi-head attention layer and feedforward network of each Transformer block in the image encoder; The dual-path dynamic adapter includes two independent dimension reduction projection layers, a dynamic weight generator, a bidirectional cross-fusion unit, a dimension increase projection layer, and a GRC-gated residual fusion unit. The input image features are mapped into two low-dimensional latent features through two independent dimension reduction projection layers; The dynamic weight generator dynamically generates a set of fusion weights for low-dimensional latent features based on the input image features; The bidirectional cross-fusion unit utilizes fusion weights to perform bidirectional cross-modulation and cross-path weighted summation on two low-dimensional latent features; The upscaling projection layer reconstructs the modulated low-dimensional features to the original dimension, resulting in two high-dimensional reconstructed features. The GRC gated residual fusion unit generates gate weights based on the original input features and two high-dimensional reconstructed features, and obtains the output features of the Transformer block by weighted mixing of the gate weights.

[0015] Preferably, the dynamic weight generator sequentially performs channel dimensionality reduction through a linear layer, a GELU activation layer, and a linear layer to enhance the learning ability of the dynamic weights, and finally outputs two sets of fused weights by uniformly dividing along the channel dimension. The GRC gated residual fusion unit generates gated weights through channel splicing and linear layers, preserving the identity mapping path of the original features for stable optimization, while allowing dual-path high-dimensional reconstructed features to contribute stronger feature updates in key regions.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for segmenting defects on the surface of hydraulic metal structures.

[0017] This invention provides a method, system, and medium for segmenting defects on the surface of hydraulic metal structures, with the following advantages: By embedding dual-path dynamic adapters in parallel at the residual connections of each Transformer block in the image encoder, and through dual-path dimensionality reduction projection, dynamic weight generation, bidirectional cross-feature fusion, and gated residual weighting, the lossless flow of key defect information is achieved, avoiding the information homogenization problem that easily occurs in single-path feature extraction. This enhances the segmentation accuracy and edge clarity of complex backgrounds, low contrast, and morphologically varied hydraulic metal defects, and reduces missed detections. At the same time, it adapts to a lightweight fine-tuning training mode, reducing the manpower cost of fully supervised pixel-level annotation and the model generalization cycle.

[0018] By setting a gated residual adaptation structure after the feedforward network of the text encoder to replace the traditional residual connection, the integrity and stability of the defect semantic description are prioritized during the model fine-tuning process. This provides reliable linguistic and semantic guidance and constraints for defect segmentation, enabling the model to adapt to different defect types according to on-site detection needs.

[0019] By training the model using a weighted multi-task objective function that integrates binary cross-entropy loss, text-image contrast loss, and text-mask contrast loss, the model achieves deep text-visual semantic alignment at both the image and mask levels, while ensuring the accuracy of defect mask segmentation through segmentation loss. This enhances the semantic matching between the segmentation results and the defect language description, thereby improving the interpretability and scene adaptability of the segmentation results. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of the structure provided for an embodiment of this application; Figure 2 This is a schematic diagram of the adapter structure of the image encoder provided in an embodiment of this application; Figure 3 This is a schematic diagram of the adapter structure of the text encoder provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the mask decoder provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] This application provides a method, system, and medium for segmenting defects on the surface of hydraulic metals. It can solve the problems in related technologies where the segmentation method for defects on the surface of hydraulic metals relies on a large number of pixel-level annotations, has insufficient robustness in defect feature extraction under complex working conditions, and lacks effective semantic guidance and cross-modal semantic alignment mechanisms, making it difficult to balance segmentation accuracy and adaptability to multiple defect types.

[0024] like Figure 1-2 As shown, a method for segmenting defects on the surface of hydraulic metal structures includes the following steps: S1: Collect original images of hydraulic metal surfaces and complete defect annotations. Generate standardized text prompts for corresponding defect categories based on a unified template, and construct image-text pairing samples. S2: Perform multi-path dynamic feature encoding on the original image. After dimensionality reduction mapping, dynamic weight generation, bidirectional cross-feature fusion and dimensionality increase reconstruction, the original features and reconstructed features are fused by gated weighted fusion to extract high-dimensional features of image defects. S3: Perform semantic encoding on defective text prompts, generate text gating weights through adaptive feature adaptation, and fuse the original text features and adapted enhanced features through gating weighting to complete the dimensional alignment of text features and image features; S4: Concatenate learnable mask markers and text prompt markers to construct a decoding input sequence. After mapping the enhanced mask markers to the image channel dimension, perform spatial matching operations with the upsampled image features to output a defect semantic segmentation mask. S5: Construct a weighted multi-task loss function to jointly optimize segmentation accuracy and cross-modal semantic alignment effect, and complete the training of the defect segmentation model.

[0025] In one embodiment, step S1 includes the following steps: S1.1: Collect images of hydraulic metal surfaces under real service conditions. After screening the original images frame by frame, use professional annotation tools to accurately annotate defect instances to ensure the accuracy and reliability of the dataset. S1.2: Perform data augmentation operations on the selected original hydraulic metal surface images, including rotation, cropping, and horizontal flipping, to simulate the visual appearance of defects under different shooting angles and scales, effectively expanding data diversity: S1.3: For the image dataset of surface defects in hydraulic metals that has been annotated with instances, a corresponding text prompt dataset is constructed simultaneously.

[0026] Specifically, firstly, images of hydraulic metal surfaces in real service environments are collected. After screening the original images frame by frame, professional annotation tools are used to accurately annotate defect instances to ensure the accuracy and reliability of the dataset. Subsequently, in order to improve the generalization ability and robustness of the model, further steps are taken.

[0027] Based on the Albumentations data augmentation library, synchronous data augmentation operations are performed on the selected original hydraulic metal surface images and their corresponding pixel-level labeled masks. Specifically, the original image and the labeled mask are used as paired samples to input the data augmentation process. During the training phase, geometric transformations such as horizontal flipping, random scaling, and random cropping are performed on the two according to the same set of random augmentation parameters. Horizontal flipping mirrors the image and mask along the same direction with a set probability. Random scaling changes the size of the image and mask according to the same scaling ratio. Random cropping extracts the image area and its corresponding mask area according to the same cropping window. This ensures that the augmented image and the pixel-level labeled mask are consistent in spatial position and defect area, simulating the visual appearance of defects under different shooting angles and scales, and effectively expanding data diversity. Finally, the enhanced images are standardized to unify the image resolution, providing solid data support for subsequent intelligent segmentation and detection of defects on hydraulic metal surfaces. For the dataset of hydraulic metal surface defect images with completed defect instance annotations, corresponding text prompt data is constructed simultaneously. First, the various defect categories covered in the dataset are sorted out. Then, based on the actual defect type annotated in each image, a fixed template Aphotoof[CLS] is used to generate corresponding text prompts. The [CLS] placeholder in the template is replaced with the professional terminology of the defect type corresponding to the image (crack, patch, peeling, "wear", etc.) to ensure that each data-enhanced hydraulic metal surface defect image is matched with one or more text prompts that accurately describe its defect category.

[0028] If an image contains both "crack" and "patch" defect categories, then the corresponding text prompts for that image will be two: "Aphotoofcrack" and "Aphotoofpatch.". These text prompts, after being processed by the text encoder, are concatenated with mask markers and input into the mask decoder for interactive decoding. The text features obtained by the text encoder and the image features are used together to calculate the text-image contrast loss, achieving alignment between image visual information and text semantic information, and supervising the fine-tuning training of the text encoder and image encoder, providing reliable semantic supervision signals for the subsequent training of the defect semantic segmentation model guided by text prompts.

[0029] During the training phase, multiple text prompts corresponding to a single image are input into the text encoder to generate independent text features. Each text feature participates in the calculation of text-image contrast loss and text-mask contrast loss. The loss value is the average of the results of multiple text prompts, which jointly supervise the training of the encoder and decoder. Multiple text prompts are concatenated with mask markers in sequence and input into the mask decoder to guide the generation of defect masks of the corresponding categories.

[0030] In one embodiment, step S2 includes the following steps: S2.1: Input the original image into an image encoder equipped with a dual-path dynamic adapter, wherein the dual-path dynamic adapter is set in parallel at the residual connection position after the multi-head attention layer and the feedforward network of each Transformer block of the image encoder; S2.2: Perform two independent linear dimensionality reduction projection operations on the input image features respectively, and map to obtain two sets of low-dimensional latent features with a channel dimension of one-eighth of the original features; S2.3: The input image features are fed into the dynamic weight generator, which sequentially performs channel linear dimensionality reduction, GELU nonlinear activation, and linear feature enhancement processing, and then outputs two sets of balanced fusion weights that adapt to the low-dimensional latent features along the channel dimension. S2.4: Two sets of fusion weights are used to perform element-wise product modulation on the two low-dimensional latent features respectively, and cross-path addition operation is performed to obtain the modulated low-dimensional features after bidirectional cross-fusion. This symmetrical fusion method can avoid the homogenization of feature information. S2.5: Perform GELU activation and linear upscaling projection on the two modulated low-dimensional features respectively to restore the feature dimensions to the original channel size of the input image features, and obtain two high-dimensional reconstructed features; S2.6: The original image features and the results of the two high-dimensional reconstruction features are spliced ​​along the channel dimension, and after linear transformation, they are evenly distributed along the channel to obtain two sets of gate weights; S2.7: The original image features and the superposition results of the two high-dimensional reconstruction features are respectively weighted and mixed element-wise to obtain the output image features of the current Transformer block; The weighted mixing process preserves the original feature identity mapping path while strengthening the feature update of the key defect region.

[0031] Specifically, such as Figure 2 As shown, in step S2.1, the original image is input to an image encoder finely tuned by a dual-path dynamic adapter, which is embedded in parallel at the residual connections after the multi-head attention layer and feedforward network of each Transformer block in the image encoder.

[0032] The image encoder uses a pre-trained visual Transformer backbone network, with a dual-path dynamic adapter embedded in the residual branches of each Transformer block in the backbone network, to achieve adaptive fine-tuning of defect features without changing the original parameters of the backbone.

[0033] The image features input in step S2.2 First, it is projected through two independent dimensionality reduction layers. and These are mapped to two low-dimensional latent features respectively. and ; ; ; in, and These are all linear layers used for dimensionality reduction projection.

[0034] Image features in step S2.3 It is also fed into a dynamic weight generator (DWG), which calculates weights based on the features. Dynamically generate a set of fusion weights for low-dimensional latent features and ; DWG features As input, the channel dimensionality is first reduced through a linear layer, mapping the features to a low-dimensional space with C / 4 channels, which also contains a GELU activation layer in the middle; The DWG structure balances the scale of introduced parameters with the ability to acquire global context information. It can dynamically generate a set of balanced weights with minimal parameter cost, providing a more reliable control signal for the subsequent fusion of dual low-dimensional potential features.

[0035] Then, a linear layer is applied to enhance the learning ability of the dynamic weights; finally, the weights are uniformly divided along the channel dimension to form the weights. and The process expression is: ; in, It is a linear layer with an output dimension of C / 4; It is the activation layer; It is a linear layer with consistent input and output dimensions; This indicates a uniform segmentation operation along the channel dimension.

[0036] DWG can dynamically generate a set of balanced weights with minimal parameter cost, providing a basis for subsequent two-dimensional low-dimensional latent features. and Integration provides more reliable control signals.

[0037] In step S2.4, the generated dynamic weights are utilized. and Low-dimensional latent features guiding two paths and Bidirectional cross-fusion is performed, where each weight is used to modulate not just a single latent feature, but every latent feature, and a cross-path weighted sum is obtained, resulting in the following calculation formula: ; ; In the formula, It is the Hadamard product; and These are modulated low-dimensional latent features.

[0038] for It not only preserves the past Modulated main information, and also supplemented Useful information.

[0039] It also fuses dual-path information in a symmetrical manner. This modulation method avoids the problem of information homogenization that may occur during the fusion process, and lays a more robust and expressive feature foundation for subsequent up-dimensional projection and gated residual connection.

[0040] Modulated features in step S2.5 and Each through a dimension-up projection layer and Reconstructing back to the original dimension yields two high-dimensional reconstructed features. and The expression is: ; ; In the formula, and These are all linear layers used for updimensional projection.

[0041] In step S2.6, the sum of the dual-path high-dimensional reconstructed features and the original input features are used. As input, gated weights are generated by concatenating channels and using a linear layer. and The formula is as follows: ; In the formula, This indicates a splicing operation along the channel dimension.

[0042] In step S2.7, the original features and reconstructed features are weighted and mixed using gating weights to obtain the output features of the Transformer block. The formula is as follows: ; Gated residual processing not only preserves the identity mapping path of the original features for stable optimization, but also allows dual-path high-dimensional reconstructed features to contribute stronger feature updates in key regions.

[0043] like Figure 3 As shown, in one embodiment, step S3 includes the following steps: S3.1: Configure several text prompts for each defective image; S3.2: Input the text prompt into the text encoder that has been fine-tuned by the lightweight adapter; The text encoder is based on a pre-trained language Transformer backbone network. Lightweight adapters are embedded after the feedforward network of each Transformer layer of the backbone network to ensure semantic stability through adaptation and fine-tuning.

[0044] S3.3: Perform dimensionality reduction, activation, and dimensionality increase adaptation processing on the input text features in sequence to obtain adapted and increased dimensionality features; concatenate the input text features and the adapted and increased dimensionality features along the channel dimension, and after linear transformation, uniformly divide them along the channel dimension to generate two sets of text gating weights; S3.4: Based on two sets of text gating weights, the input text features and the adapted dimensionality-upgrading features are respectively weighted and modulated element by element. The two weighted results are added together to obtain the output features of the text encoder. S3.5: Perform dimensional transformation on the output features to adapt the text feature dimensions to the image feature dimensions.

[0045] Specifically, each defect image is accompanied by several text prompts. Each prompt directly describes the type of defect present in the image and uses a concise and uniform template: Aphotoof[CLS]., where [CLS] is replaced with the specific defect type (such as patch, crack).

[0046] The text prompts are fed into a text encoder finely tuned by a lightweight adapter, which is uniformly embedded after the feedforward network.

[0047] Given the input text features of a certain transformer layer of a text encoder, First, features are obtained through a standard adapter of dimensionality reduction-activation-upgrading. ; Subsequently, gated weights are generated through channel splicing and linear layers. and Where L is the length of the text sequence. The formula is as follows: ; Output features of a text encoder It can be calculated as: ; Finally, a lightweight two-layer MLP is used to perform dimensionality transformation on the text output features of the text encoder to adapt to the image feature dimensions.

[0048] like Figure 4 As shown, in one embodiment, step S4 includes the following steps: S4.1: Construct an input tag sequence, which is composed of learnable mask tags and text prompt tags; S4.2: Extract text hints before and after decoding and overlay them to obtain enhanced text tags; calculate the cosine similarity between the enhanced text tags and each mask tag, and select the K candidate mask tags with the highest similarity; perform weighted aggregation on the candidate mask tags to form a mask output tag with stronger category discrimination ability.

[0049] S4.3: The mask output marker is processed by a 3-layer MLP mapping to obtain a vector with the same dimension as the upsampled image feature channel; the spatial dot product operation is performed between this vector and the upsampled image features to complete the prediction output of the defect mask.

[0050] Specifically, the defect semantic segmentation mask is predicted by a mask decoder, which includes two Transformer layers, two transposed convolutional layers, a cross-attention layer, and an MLP.

[0051] The input tag sequence for constructing the mask decoder is based on the concatenation of learnable mask tags and text prompt tags; where the mask tags are a set of learnable tag sequences, denoted as . ,in The number of mask markers, The dimension of the mask marker. Text hint markers are composed of text semantic features; When multiple text prompts are input, they are concatenated with the same set of mask marks according to their categories to form multiple input sequences. Each sequence is then decoded to generate a defect mask for the corresponding category, and the outputs of each mask set are independent of each other.

[0052] The token enhancement mechanism based on similarity filtering extracts the input text hints and the output text hints from the mask decoder. The extracted text hints are then summed element-wise to obtain the enhanced text hints. Next, the cosine similarity between the enhanced text hints and each mask hint is calculated, and the K candidate mask hints with the highest similarity are selected. Finally, the candidate mask hints are weighted and aggregated. The similarity of the candidate mask hints, after temperature scaling and Softmax normalization, is used as the weight to perform a weighted summation on the K candidate mask hints, forming a mask output hint with better category discrimination capabilities. ; Two-stage transposed convolutional layers are used to upsample the image features updated by the two Transformer layers step by step to obtain upsampled image features that match the target segmentation resolution. Mask output marker After three layers of MLP mapping, a vector with the same channel dimension as the upsampled image features is obtained. Then, the spatial dimension of the upsampled image features is flattened, and the mask prediction vector and the flattened image features are multiplied by matrix in the channel dimension to calculate the dot product response between each mask marker and the image features at each spatial location. Finally, the last dimension of the response result is restored to a 2D space, thereby realizing the mask prediction of the defect region.

[0053] In one embodiment, step S5 includes the following steps: S5.1: Select binary cross-entropy loss and contrastive loss as the basic components of the loss function; S5.2: Calculate the text-image contrast loss by comparing and learning the text features with the image features. Perform a softmax operation on the similarity between all image patches and the corresponding text features in each training sample, and calculate the mean after taking the logarithm of the operation results. S5.3: Calculate the text-mask contrast loss. Compare the text fusion label obtained by adding the text features before decoding and the text labels after decoding with the mask output label. Perform softmax operation on the similarity of all text labels in each training sample, and calculate the mean after taking the logarithm of the operation result. S5.4: Build a training environment based on a deep learning framework, set the batch size, and complete model training according to the set maximum iteration cycle; S5.5: Input the original image of the hydraulic metal to be detected and the matching text prompt into the pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation result that better matches the text prompt.

[0054] Specifically, by selecting binary cross-entropy loss and contrastive loss as the basic components of the loss function, the total loss expression is as follows: ; In the formula, the binary cross-entropy loss It is mainly used to enhance the segmentation ability of the model; It is the contrast loss between text and image; It is the contrast loss between text and mask; , and It is a weighting factor that weighs the contribution of each loss component to the overall training objective, and its value range is... .

[0055] Calculate the contrast loss between text and image The text features (output of the text encoder) are compared and learned with the image features (output of the image encoder), and its expression is: ; In the formula, It is the number of training samples; It represents the number of patches in the image; achieving high-quality language-visual cross-modal alignment in the field of defect segmentation.

[0056] To enhance the semantic consistency between the segmentation results and the linguistic description, the text-mask contrast loss is calculated. Its expression is: ; In the formula, The text fusion marker is obtained by adding the text features before decoding to the text marker after decoding; It represents the number of text tags.

[0057] The text features before decoding and the text tags after decoding have the same dimension. The two are added element by element to obtain the text fusion tag. M is the number of sequence tags corresponding to a single text prompt. The loss is calculated by classifying the samples and then taking the mean.

[0058] Built on the PyTorch platform with a batch size of 4, the native parameters (multi-head attention layer and feedforward network layer parameters) of the Transformer backbones of the image encoder and text encoder were frozen during training. Only the corresponding adapter component parameters were updated, while the mask decoder underwent full parameter fine-tuning. 100 randomly initialized mask tokens were used for the model to predict defects in semantic segmentation. The optimizer used was the AdamW optimizer with an initial learning rate of 0.0002, a weight decay coefficient of 0, and a momentum parameter of... and The values ​​were set to 0.9 and 0.999 respectively; a cosine annealing learning rate scheduler was used, with a maximum iteration period of 500 epochs and a minimum learning rate of [missing information]. Complete model training.

[0059] During the training phase, defect category names are automatically generated based on the category indices corresponding to the ground truth mask values ​​in the dataset. These defect category names are then automatically filled into a preset text prompt template (Aphotoof[CLS].) to generate the final text prompts for the dataset. During the testing phase, defect category names are manually entered based on the required defect categories, allowing for flexible segmentation of different defect categories. Furthermore, the system supports simultaneous input of multiple text prompts from different categories, and the model synchronously outputs multi-category defect segmentation masks that correspond one-to-one with each text prompt, eliminating the need to retrain the model.

[0060] The original image of the hydraulic metal to be detected and the matching text prompt are input into a pre-trained defect segmentation model, which outputs a hydraulic metal surface defect segmentation result that better matches the text prompt.

[0061] In one embodiment, a hydraulic metal surface defect segmentation system includes: The image acquisition module acquires the original hydraulic metal image; The text prompt acquisition module automatically generates text prompt descriptions based on the corresponding category of surface defects in hydraulic metal structures; The model building module is used to build defect segmentation models; The defect segmentation model includes: a dual-path dynamic adapter fine-tuned image encoder, a lightweight adapter fine-tuned text encoder, and a novel lightweight mask decoder that is more suitable for semantic segmentation. The defect segmentation module is used to input the original hydraulic metal image and text prompts into a pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation results that are more consistent with the text prompts.

[0062] In one embodiment, the dual-path dynamic adapter is embedded in parallel at the residual connections after the multi-head attention layer and feedforward network of each Transformer block in the image encoder. The dual-path dynamic adapter includes two independent dimension reduction projection layers, a dynamic weight generator (DWG), a bidirectional cross-fusion unit, a dimension up projection layer, and a GRC-gated residual fusion unit. The input image features are mapped into two low-dimensional latent features through two independent dimension reduction projection layers; The Dynamic Weight Generator (DWG) dynamically generates a set of fusion weights for low-dimensional latent features based on the input image features; The bidirectional cross-fusion unit utilizes fusion weights to perform bidirectional cross-modulation and cross-path weighted summation on two low-dimensional latent features; The upscaling projection layer reconstructs the modulated low-dimensional features to the original dimension, resulting in two high-dimensional reconstructed features. The GRC gated residual fusion unit generates gate weights based on the original input features and two high-dimensional reconstructed features, and obtains the output features of the Transformer block by weighted mixing of the gate weights.

[0063] In one embodiment, the Dynamic Weight Generator (DWG) sequentially performs channel dimensionality reduction through a linear layer, a GELU activation layer, and a linear layer to enhance the learning ability of the dynamic weights, and finally outputs two sets of fused weights by uniformly dividing along the channel dimension. The GRC gated residual fusion unit generates gated weights through channel splicing and linear layers, preserving the identity mapping path of the original features for stable optimization, while allowing dual-path high-dimensional reconstructed features to contribute stronger feature updates in key regions.

[0064] In one embodiment, after the lightweight adapter is uniformly embedded in the feedforward network of the text encoder, the lightweight adapter includes a dimensionality reduction-activation-upgrading adaptation unit, a text gating generation unit, and a dimensionality transformation unit. The dimensionality reduction-activation-upgrading adaptation unit adapts the input text features to obtain upgrading features; The text gating generation unit generates text gating weights by concatenating channels and using a linear layer. It then performs gating weighted fusion on the original text features and the upgraded features to obtain the output features of the text encoder. The dimension transformation unit uses a lightweight two-layer MLP to perform dimension transformation on the text output features to adapt to the image feature dimensions.

[0065] In one embodiment, the mask decoder comprises two Transformer layers, two transposed convolutional layers, a cross-attention layer, and an MLP; the label sequence input to the mask decoder is composed of learnable mask labels and text cue labels concatenated together. The cross-attention layer is set between the two Transformer layers. It uses the mask label sequence as the query and the multi-scale defect features output by the image encoder as the key and value. Through attention calculation, it guides the mask labels to spatially align with the image defect region, thereby enhancing the mask's ability to focus on defect features.

[0066] The mask decoder includes a token enhancement mechanism based on similarity filtering. It extracts text hint tokens before and after decoding and adds them together to obtain the enhanced text tokens. It calculates the cosine similarity between the enhanced text tokens and each mask token, selects the K candidate mask tokens with the highest similarity and aggregates them with weight to form a mask output token with better category discrimination ability. The mask output marker is mapped to a vector with the same dimension as the feature channel of the upsampled image after passing through 3 layers of MLP. The defect mask is predicted by the spatial dot product operation between this vector and the features of the upsampled image.

[0067] In one embodiment, the model further includes a training optimization module, which is used to construct a multi-task loss function that integrates cross-entropy loss and contrastive loss to train the model. The multi-task loss function uses binary cross-entropy loss, text-image contrast loss, and text-mask contrast loss as its basic components, and through... , and Three weighting factors weigh the contribution of each loss component to the overall training objective; during training, the parameters of the image encoder and text encoder are frozen, and only the adapter component and mask decoder are fine-tuned.

[0068] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for segmenting defects on the surface of hydraulic metal structures.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0074] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0075] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for segmenting defects on the surface of hydraulic metal structures, characterized in that, Includes the following steps: S1: Collect original images of hydraulic metal surfaces and complete defect annotations. Generate standardized text prompts for corresponding defect categories based on a unified template, and construct image-text pairing samples. S2: Perform multi-path dynamic feature encoding on the original image. After dimensionality reduction mapping, dynamic weight generation, bidirectional cross-feature fusion and dimensionality increase reconstruction, the original features and reconstructed features are fused by gated weighted fusion to extract high-dimensional features of image defects. S3: Perform semantic encoding on defective text prompts, generate text gating weights through adaptive feature adaptation, and fuse the original text features and adapted enhanced features through gating weighting to complete the dimensional alignment of text features and image features; S4: Concatenate learnable mask markers and text prompt markers to construct a decoding input sequence. After mapping the enhanced mask markers to the image channel dimension, perform spatial matching operations with the upsampled image features to output a defect semantic segmentation mask. S5: Construct a weighted multi-task loss function to jointly optimize segmentation accuracy and cross-modal semantic alignment effect, and complete the training of the defect segmentation model.

2. The method for segmenting surface defects of hydraulic metal structures according to claim 1, characterized in that: Step S1 includes the following steps: S1.1: Collect images of hydraulic metal surfaces under real service conditions. After screening the original images frame by frame, use professional annotation tools to accurately annotate defect instances to ensure the accuracy and reliability of the dataset. S1.2: Perform data augmentation operations on the selected original hydraulic metal surface images, including rotation, cropping, and horizontal flipping, to simulate the visual appearance of defects under different shooting angles and scales, effectively expanding data diversity. S1.3: For the image dataset of surface defects in hydraulic metals that has been annotated with instances, a corresponding text prompt dataset is constructed simultaneously.

3. The method for segmenting surface defects of hydraulic metal structures according to claim 1, characterized in that: Step S2 includes the following steps: S2.1: Input the original image into an image encoder equipped with a dual-path dynamic adapter, wherein the dual-path dynamic adapter is set in parallel at the residual connection position after the multi-head attention layer and the feedforward network of each Transformer block of the image encoder; S2.2: Perform two independent linear dimensionality reduction projection operations on the input image features respectively, and map to obtain two sets of low-dimensional latent features with a channel dimension of one-eighth of the original features; S2.3: The input image features are fed into the dynamic weight generator, which sequentially performs channel linear dimensionality reduction, GELU nonlinear activation, and linear feature enhancement processing, and then outputs two sets of balanced fusion weights that adapt to the low-dimensional latent features along the channel dimension. S2.4: Two sets of fusion weights are used to perform element-wise product modulation on the two low-dimensional latent features respectively, and cross-path addition operation is performed to obtain the modulated low-dimensional features after bidirectional cross-fusion. This symmetrical fusion method can avoid the homogenization of feature information. S2.5: Perform GELU activation and linear upscaling projection on the two modulated low-dimensional features respectively to restore the feature dimensions to the original channel size of the input image features, and obtain two high-dimensional reconstructed features; S2.6: The original image features and the results of the two high-dimensional reconstruction features are spliced ​​along the channel dimension, and after linear transformation, they are evenly distributed along the channel to obtain two sets of gate weights; S2.7: Use two sets of gating weights to perform element-wise weighted mixing of the original image features and the superposition results of the two high-dimensional reconstruction features to obtain the output image features of the current Transformer block.

4. The method for segmenting defects on the surface of hydraulic metal structures according to claim 1, characterized in that: Step S3 includes the following steps: S3.1: Configure several text prompts for each defective image; S3.2: Input the text prompt into the text encoder that has been fine-tuned by the lightweight adapter; S3.3: Perform dimensionality reduction, activation, and dimensionality increase adaptation processing on the input text features in sequence to obtain adapted and increased dimensionality features; concatenate the input text features and the adapted and increased dimensionality features along the channel dimension, and after linear transformation, uniformly divide them along the channel dimension to generate two sets of text gating weights; S3.4: Based on two sets of text gating weights, the input text features and the adapted dimensionality-upgrading features are respectively weighted and modulated element by element. The two weighted results are added together to obtain the output features of the text encoder. S3.5: Perform dimensional transformation on the output features to adapt the text feature dimensions to the image feature dimensions.

5. The method for segmenting defects on the surface of hydraulic metal structures according to claim 1, characterized in that: Step S4 includes the following steps: S4.1: Construct an input tag sequence, which is composed of learnable mask tags and text prompt tags; S4.2: Extract text hints before and after decoding and overlay them to obtain enhanced text tags; calculate the cosine similarity between the enhanced text tags and each mask tag, select the K candidate mask tags with the highest similarity, and perform weighted aggregation on the candidate mask tags to form the mask output tags; S4.3: The mask output marker is processed by 3 layers of MLP mapping to obtain a vector with the same dimension as the feature channel of the upsampled image; the spatial dot product operation is performed between this vector and the upsampled image features to complete the prediction output of the defect mask.

6. The method for segmenting surface defects of hydraulic metal structures according to claim 1, characterized in that: Step S5 includes the following steps: S5.1: Select binary cross-entropy loss and contrastive loss as the basic components of the loss function; S5.2: Calculate the text-image contrast loss by comparing and learning the text features with the image features. Perform a softmax operation on the similarity between all image patches and the corresponding text features in each training sample, and calculate the mean after taking the logarithm of the operation results. S5.3: Calculate the text-mask contrast loss. Compare the text fusion label obtained by adding the text features before decoding and the text labels after decoding with the mask output label. Perform softmax operation on the similarity of all text labels in each training sample, and calculate the mean after taking the logarithm of the operation result. S5.4: Build a training environment based on a deep learning framework, set the batch size, and complete model training according to the set maximum iteration cycle; S5.5: Input the original hydraulic metal image to be detected and the matching text prompt into the pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation result that better matches the text prompt.

7. A system for segmenting surface defects of hydraulic metal, used to implement the method for segmenting surface defects of hydraulic metal as described in any one of claims 1-6, characterized in that, include: The image acquisition module acquires the original hydraulic metal image; The text prompt acquisition module automatically generates text prompt descriptions based on the corresponding category of defects on the surface of hydraulic metal structures. The model building module is used to build defect segmentation models; The defect segmentation model includes: an image encoder fine-tuned by a dual-path dynamic adapter, a text encoder fine-tuned by a lightweight adapter, and a novel lightweight mask decoder that is more suitable for semantic segmentation. The defect segmentation module is used to input the original hydraulic metal image and text prompts into a pre-trained defect segmentation model, and output the hydraulic metal surface defect segmentation results that are more consistent with the text prompts.

8. A surface defect segmentation system for hydraulic metal structures according to claim 7, characterized in that: The dual-path dynamic adapter is embedded in parallel at the residual connections after the multi-head attention layer and feedforward network of each Transformer block in the image encoder. The dual-path dynamic adapter includes two independent dimension reduction projection layers, a dynamic weight generator, a bidirectional cross-fusion unit, a dimension increase projection layer, and a GRC-gated residual fusion unit. The input image features are mapped into two low-dimensional latent features through two independent dimension reduction projection layers; The dynamic weight generator dynamically generates a set of fusion weights for low-dimensional latent features based on the input image features; The bidirectional cross-fusion unit utilizes fusion weights to perform bidirectional cross-modulation and cross-path weighted summation on two low-dimensional latent features; The upscaling projection layer reconstructs the modulated low-dimensional features to the original dimension, resulting in two high-dimensional reconstructed features. The GRC gated residual fusion unit generates gate weights based on the original input features and two high-dimensional reconstructed features, and obtains the output features of the Transformer block by weighted mixing of the gate weights.

9. A surface defect segmentation system for hydraulic metal structures according to claim 8, characterized in that: The dynamic weight generator sequentially performs channel dimensionality reduction through a linear layer, a GELU activation layer, and a linear layer to enhance the learning ability of the dynamic weights. Finally, it evenly divides and outputs two sets of fused weights along the channel dimension. The GRC gated residual fusion unit generates gated weights through channel splicing and linear layers, preserving the identity mapping path of the original features for stable optimization, while allowing dual-path high-dimensional reconstructed features to contribute stronger feature updates in key regions.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements a method for segmenting defects on the surface of hydraulic metal as described in any one of claims 1 to 6.

Images