Method and system for segmentation of inner surface defects of steel pipes fusing multi-modal information

Abstract

The application provides a steel pipe inner surface defect region segmentation method and system fusing multi-modal information, comprising: simultaneously acquiring a steel pipe inner surface defect RGB image and a depth map, and performing preprocessing and labeling; constructing a support data set and a query data set; a support and query image feature extractor extracts multi-modal feature extraction backbone networks respectively to extract support image multi-scale aggregation features F S , support image multi-modal semantic features I S , query image multi-scale aggregation features F Q , and query image multi-modal semantic features I Q ; F S , I S , F Q , and I Q are input into a multi-feature fusioner, a support image semantic guiding module uses F S and M S to generate class guiding features F A , a similar prior feature generation module uses I S , I Q , and M S to generate similar prior features F M , F A , F M , and F Q are spliced in the channel dimension, fusion feature maps are output, and a multi-modal decoder is used to obtain defect region segmentation output. The application can segment the steel pipe inner surface defect region.

Description

Technical Field

[0001] This invention relates to the field of defect region segmentation technology on the inner surface of steel pipes, and in particular to a method and system for segmenting defect regions on the inner surface of steel pipes that integrates multimodal information. Background Technology

[0002] Steel pipes, as a key industrial material, are widely used in oil and gas transportation, high-pressure pipelines, energy equipment, and special manufacturing. The quality of their inner surface directly affects the mechanical properties, sealing performance, and service life of the products. In actual production, the inner wall of steel pipes often develops various defects such as cracks, scratches, oxide scale, and pits due to rolling, heat treatment, or transportation. These defects are usually small in size, complex in shape, and unevenly distributed, making them difficult to accurately identify with the naked eye or low-resolution imaging methods, which seriously restricts the stability and consistency of product quality control.

[0003] Currently, the inspection of the inner surface of steel pipes mainly relies on manual visual inspection or traditional monocular vision imaging devices. Manual visual inspection has a limited field of view, strong subjectivity, and a high rate of missed detection. It is not well adapted to small-diameter pipes. Monocular vision imaging devices often suffer from light spot interference, blurred edges, and geometric distortion under the conditions of high reflectivity and large curvature inside steel pipes, which affects the accuracy of defect identification.

[0004] Most existing methods segment defects based on two-dimensional images, failing to obtain three-dimensional depth information. Since defect depth is a crucial indicator of the severity of pipe defects, the lack of this information significantly impacts the model's ability to identify defects, increasing the risk of missegmentation and missed segmentation.

[0005] Existing deep learning segmentation models rely on a large number of high-quality pixel-level masks for training. However, in industrial settings, samples are scarce and labeling is difficult, limiting the model's ability to generalize to real-world tasks. Furthermore, traditional models are mostly trained on closed categories, failing to effectively identify novel defect categories and lacking segmentation generalization capabilities under cross-category and limited sample conditions. Summary of the Invention

[0006] To address the technical problems existing in the prior art, the present invention provides a method and system for segmenting defect regions on the inner surface of steel pipes by integrating multimodal information. The technical solution is as follows:

[0007] On the one hand, a method for segmenting defect regions on the inner surface of steel pipes by fusing multimodal information is provided, the method comprising:

[0008] S1. The main body of the detection device, including a hexagonal prism multimodal probe, simultaneously acquires RGB images and depth maps of defects on the inner surface of the steel pipe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotations are performed, forming a one-to-one correspondence three-dimensional data set with the RGB images and depth maps.

[0009] S2. Using the aforementioned three-element data set, construct a support dataset and a query dataset for training the few-shot segmentation model. The support dataset includes datasets supporting RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ;

[0010] S3. Input the supported RGB image and supported depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network to extract the multi-scale aggregated features F of the supporting image respectively. S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ;

[0011] S4, the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature F A The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map.

[0012] S5. The fused feature map is decoded by a multimodal decoder to generate a pixel-level prediction mask corresponding to the query image, thereby achieving the final defect region segmentation output.

[0013] Optionally, the main body of the detection device includes a high-strength telescopic mechanism, a soft connection buffer structure, and a hexagonal prism-shaped multimodal probe;

[0014] The telescopic mechanism uses a high-strength telescopic rod, which can be adapted to steel pipes of different specifications and lengths.

[0015] The multimodal probe is flexibly connected to a telescopic rod via a strong spring, providing shock resistance and self-adaptation capabilities. The probe is shaped like a regular hexagonal prism, with six industrial-grade cameras and corresponding line structured light lasers evenly distributed on its six faces, forming a 360° all-view perception capability. When the device is working, driven by the control system, the probe moves forward along the steel pipe axis. The six cameras respectively capture images of the inner wall within a 60° central angle range. At the same time, the line array laser projects a grating onto the inner wall surface, and obtains the depth information of the inner wall by recording the changes in reflected light.

[0016] Optionally, the data processor unfolds and stitches the original RGB images in six directions according to a preset geometric relationship to form a two-dimensional unfolded image with the vertical dimension being the circumference of the inner diameter of the steel pipe and the horizontal dimension being the depth of advancement.

[0017] The depth images acquired simultaneously are also aligned and spatially stitched using corresponding relationships to ensure a perfect match with the RGB image;

[0018] Then, image preprocessing is performed:

[0019] Enhancement operations are performed on the stitched RGB image;

[0020] Meanwhile, the stitched depth image will be resized and normalized, and channel duplication will be performed to meet the input format requirements for subsequent model training.

[0021] Optionally, the support dataset selects high-frequency category images from the defect samples on the inner surface of steel pipes as the main support image source, and also integrates general metal defect image samples from public datasets to form a training sample subset that is cross-material and cross-morphology, which is used to provide rich category priors during the training phase.

[0022] The query dataset is derived from defect images collected on a real steel pipe production line.

[0023] Optionally, the multimodal feature extraction backbone network is a multi-layered feature extraction architecture, including four feature extraction layers and corresponding attention feature enhancement modules. The middle and lower layers are responsible for extracting fine-grained local features, while the higher layers focus on capturing broader semantic information. Through this hierarchical design, features at different levels can be effectively distinguished, while maintaining modality specificity during the fusion process.

[0024] Each feature extraction layer contains a multi-layer convolutional structure. The expressive power of the features is gradually enhanced by stacking convolutional layers. The backbone network receives RGB images and depth images as joint inputs and performs feature extraction through a parallel dual-branch structure. The RGB branch focuses on capturing global semantic information and texture features, while the depth branch focuses on modeling local geometric changes and spatial details. After the outputs of each feature extraction layer, the two branches are fused through an attention feature enhancement module. This achieves semantic alignment and expression enhancement while maintaining modality specificity. To highlight structural details and discriminative ability, the backbone network outputs fused features in the middle layers and then concatenates them. Through information integration and feature dimensionality reduction, multi-scale aggregated features of the model are generated. At the same time, the fused features output in the high-level feature extraction stages retain their modality fusion form and serve as multimodal semantic features.

[0025] Optionally, the attention feature enhancement module receives RGB modal features and deep modal features output from each layer of the multimodal feature extraction backbone network, and performs structure-aware enhancement and context-dependent modeling on the two types of modal features respectively through a differential depth convolution structure, thereby improving the expressive ability of the fused features for complex defect regions. The whole module consists of two parallel paths.

[0026] The deep modal path includes:

[0027] Based on differential convolutional attention, the input depth feature map is first compressed through a 1×1×C / 8 convolution operation to reduce computational complexity.

[0028] The compressed features are then fed into a differential depthwise convolution to achieve adaptive modeling of local geometry. This differential depthwise convolution dynamically adjusts the convolution weights by constructing a differential convolution kernel. Its core calculation method is as follows: for any convolution center position P and its neighboring offset position P+δ i Each convolution weight of the differential convolution kernel The calculation formula is:

[0029]

[0030] Where X(P) represents the value of the input feature map at pixel position P, δ i The relative offset position is represented by |·|, which represents the L1 norm. The exp(·) function is used to construct an exponential decay factor based on pixel differences, which suppresses the response of geometrically abrupt regions during convolution.

[0031] The output features of the differential depth convolution operation are used to generate an attention map through a 1×1 convolution, and then perform element-wise multiplication with the input features before convolution. The original channel dimension is then restored through a 1×1×C convolution, and finally residual summation is performed with the initial input features to output the enhanced depth output features.

[0032] RGB modal path:

[0033] Differential depthwise dilated convolution is employed, introducing a dilation factor r on top of differential depthwise convolution to significantly expand the receptive field of the convolution while maintaining stable computational resources. This is used to model semantic dependencies between distant pixels in an image. The convolution weight formula is adjusted as follows:

[0034]

[0035] Where r represents the dilation factor of dilated convolution;

[0036] The output after the differential depth dilation convolution operation is subjected to 1×1 convolution, followed by element-wise multiplication, original channel dimension restoration, and residual connection. Finally, it is added element-wise with the depth output feature to obtain the enhanced RGB output feature.

[0037] Optionally, the support graph semantic guidance module extracts and generates class guidance features F by utilizing the annotation information of known defects in the support dataset. A This is used to highlight key information related to the target category in the feature space, enabling the model to selectively focus on regions matching the target category when processing query images. The processing procedure is as follows:

[0038] F S and M S Multiply to calculate the defect features F in the supporting image. D ;

[0039] Defect feature F D The triple pooling component, consisting of global attention pooling, global average pooling, and global max pooling, sums the results element by element and calculates the vector average. Then, it is vector-expanded to restore the original feature size. The formula is as follows:

[0040]

[0041] Where η is the averaging factor, GP(·) represents global average pooling, MP(·) represents global max pooling, and AP(·) represents global attention pooling. The results of the three aggregation operations are averaged to provide more comprehensive global semantic information.

[0042] To further enhance the focus on defect areas, considering the spatial feature differences of different modal information, an attention mechanism is introduced to focus on key defect areas in the image: the global attention pooling result is weighted according to the importance of different regions in the image by calculating attention weight A. The formula for calculating the attention weight is as follows:

[0043] A=σ(W1·ReLU(W2·AP(F S )+b2)+b1)

[0044] Where W1 and W2 are learnable weight matrices, b1 and b2 are bias terms, σ(·) is the sigmoid activation function, and the output A is the result of global attention pooling, AP(F). S Attention maps with the same spatial dimensions;

[0045] Next, the attention map is compared with F. S Add them through residual connection, and then combine them with F after vector expansion. P The final guided feature F is obtained by concatenating the data along the channel dimension. A The calculation formula is as follows:

[0046] F A =Concat((AP(F) S )⊙A)+F S ,ε(F P ))

[0047] Where Concat(a,b) means concatenating a and b along the channel dimension, ⊙ means dot product of two matrices, and ε(·) means vector expansion.

[0048] Optionally, the similarity prior feature generation module calculates and generates similarity prior features F by comparing the multimodal semantic features of the support graph and the query graph. M The process quantifies the semantic similarity between each pixel in the query image and the defective region in the support image, providing prior information based on cross-image association for subsequent segmentation. The process is as follows:

[0049] The I Q and I S After activation function processing, for I S It will also be through its binarized image annotation M S By setting the background region in the supporting image to zero and retaining only the foreground features related to the target category, X is obtained. Q and X S The formula is as follows:

[0050] X Q =ReLU(I Q )

[0051] X S =ReLU(I S )·M S

[0052] Where ReLU(·) represents the ReLU activation function;

[0053] Based on the XQ and X S For each pixel position in the semantic features of the query image Calculate its relationship with all supporting image foreground pixels. The cosine similarity between them is calculated using the following formula:

[0054]

[0055] Next, the maximum similarity value between each pixel in the query image and all foreground pixels in the support images is extracted and used as its pixel similarity prior response value. The announcement is as follows:

[0056]

[0057] After calculating the maximum similarity value for each pixel, the final result is the prior similarity response value of each pixel. The similarity feature C constitutes Q The image is deformed to generate prior features F with the same spatial dimension as the query image. M And use minimum-maximum normalization for F M The pixel values ​​in the image are normalized to a range of [0,1]. The normalization process is achieved using the following formula:

[0058]

[0059] Here, ε represents a minimal constant introduced to prevent division by zero errors.

[0060] Optionally, the multimodal decoder employs a feature refinement module, a convolutional attention module, and a convolutional segmentation head to perform information enhancement and classification prediction on the feature map;

[0061] The feature refinement module includes two cascaded substructures. Each substructure includes a set of normalization layers, a smooth rectified activation function, and a 3×3 convolution operation, which are used to perform nonlinear transformation and local structure modeling on the input multi-channel feature map. The two sets of substructures are connected by skip connections to form a residual enhancement path to improve the continuity and stability of feature expression.

[0062] After the output of the first feature refinement module, a convolutional attention module is further introduced to enhance the feature response capability in the channel dimension and spatial dimension.

[0063] The feature map output by the convolutional attention module is then fed into the second feature refinement module to further extract the local texture and boundary information enhanced by the attention mechanism.

[0064] The feature map output by the second feature refinement module is finally input into the convolutional segmentation head, which consists of two sets of feature refinement modules and a 1×1 convolutional layer. This maps the high-dimensional channel features to a one-dimensional output channel and outputs a defect region segmentation result map with the same spatial dimension as the input image.

[0065] The convolutional attention module includes two parts: channel attention and spatial attention, which improves the final segmentation capability of the multimodal decoder for the target region while maintaining a lightweight design.

[0066] The channel attention obtains global feature information of different channels through global average pooling and global max pooling, and calculates the importance of each channel in combination with a fully connected layer. At the same time, it is activated by sigmoid to strengthen key channels and suppress irrelevant channels. Then it is multiplied with the original input features to obtain the channel features after channel attention weighting.

[0067] The spatial attention feature utilizes the channel features, performs max pooling and average pooling in the channel dimension, concatenates the results, and generates spatial attention features through 7×7 convolution and sigmoid activation to highlight salient regions and reduce background interference.

[0068] Finally, the channel features and spatial attention features are multiplied and fused element-wise, enabling the multimodal decoder to more accurately focus on task-related features and improve overall segmentation performance.

[0069] On the other hand, a steel pipe inner surface defect region segmentation system integrating multimodal information is provided, the system comprising:

[0070] The acquisition and processing module is used to simultaneously acquire RGB images and depth maps of defects on the inner surface of steel pipes through the main body of the detection device including a hexagonal multimodal probe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotation is performed, forming a one-to-one correspondence three-data set with the RGB image and depth map.

[0071] The building module is used to construct a support dataset and a query dataset for training a few-shot segmentation model using the three-element data set. The support dataset includes datasets that support RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ;

[0072] The extraction module is used to input the supported RGB image and supported depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network to extract the multi-scale aggregated features F of the supporting image.S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ;

[0073] The fusion module is used to integrate the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature F A The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map.

[0074] The decoding module is used to decode the fused feature map by the multimodal decoder, generate a pixel-level prediction mask corresponding to the query image, and achieve the final defect region segmentation output.

[0075] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method for segmenting defect regions on the inner surface of a steel pipe by fusing multimodal information.

[0076] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described method for segmenting the defect region on the inner surface of a steel pipe by fusing multimodal information.

[0077] The beneficial effects of the technical solution provided by this invention include at least the following:

[0078] 1. The detection device of the present invention has a flexible structure, good physical scalability and anti-interference ability, and can stably adapt to the internal space of steel pipes of various sizes and complex structures, so as to realize high-resolution, high-coverage all-round data acquisition and provide a reliable perception basis for defect identification.

[0079] 2. This invention introduces a joint input method of depth information and texture image, and designs a feature extraction and enhancement module dedicated to modal fusion in the model architecture, which effectively improves the system's spatial understanding and detail restoration of defect morphology. Especially in areas with abrupt changes in geometric structure or complex lighting conditions, the segmentation results are more refined and accurate.

[0080] 3. This invention adopts a few-sample learning mechanism for scenarios with scarce data, and constructs a strategy that supports graph class guidance and prior similarity modeling. This enables the system to still have strong class transfer ability and segmentation generalization performance even when training samples are insufficient, thus meeting the detection needs of new or very few defect types in actual industry.

[0081] 4. This invention, by constructing multi-scale feature aggregation, differential convolution enhancement, and attention mechanisms, guides information interaction and semantic alignment among multimodal features, effectively enhancing the model's ability to discriminate complex background interference and blurred edge regions, and ensuring the integrity and boundary clarity of the segmentation results. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of a method for segmenting defect regions on the inner surface of a steel pipe that integrates multimodal information, provided by an embodiment of the present invention.

[0084] Figure 2 This is a general block diagram of a method for segmenting the inner surface defect region of a steel pipe that integrates multimodal information, provided in an embodiment of the present invention.

[0085] Figure 3 This is a schematic diagram of the main structure of the detection device provided in an embodiment of the present invention;

[0086] Figure 4 This is a flowchart of the data processor processing procedure provided in an embodiment of the present invention;

[0087] Figure 5 This is a schematic diagram of image geometric relationship stitching provided in an embodiment of the present invention;

[0088] Figure 6 This is a block diagram of the multimodal feature extraction backbone network structure provided in the embodiments of the present invention;

[0089] Figure 7 This is a structural block diagram of the attention enhancement module provided in an embodiment of the present invention;

[0090] Figure 8 This is a structural block diagram of the graph semantic guidance module provided in an embodiment of the present invention;

[0091] Figure 9 This is a block diagram of the multimodal decoder structure provided in an embodiment of the present invention;

[0092] Figure 10 This is a block diagram of the convolutional attention module structure provided in an embodiment of the present invention;

[0093] Figure 11 This is a block diagram of a steel pipe inner surface defect region segmentation system that integrates multimodal information, provided by an embodiment of the present invention.

[0094] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0095] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0096] This invention provides a method for segmenting defect regions on the inner surface of steel pipes by fusing multimodal information. First, a flexible detection device is designed, the main body of which includes a telescopic mechanism and a hexagonal prism-shaped multimodal probe, connected by a high-strength anti-collision spring. The probe simultaneously acquires RGB images and depth maps of the defects on the inner surface of the steel pipe through six linear array cameras and a linear array laser. These images are divided into a support dataset and a query dataset. The support dataset also includes a large amount of publicly available data on metal defects. Both datasets are fed as input to a parameter-shared multimodal feature extraction backbone network to extract multi-scale and multimodal semantic features from the images. The multimodal feature extraction backbone network consists of an RGB image branch and a depth image branch, and uses differential convolution and dilation mechanisms... To enhance the model's ability to express local geometric details and long-distance semantics, a unified multi-scale fused feature map is generated. The extracted features are then fed into a multi-feature fusion module that includes a support graph semantic guidance module and a similar prior feature generation module. The support graph semantic guidance module guides the focus of semantic information through labeled masks in the support graph, uses triple pooling to generate class-guided features, and enhances the response of defect regions through an attention mechanism. The similar prior feature generation module calculates pixel-level similarity based on the multimodal semantic features of the support graph and the query graph to construct similar prior features for the query graph. The fused features are decoded by the multimodal decoder to generate a pixel-level prediction mask corresponding to the query image, achieving the final defect region segmentation output.

[0097] To improve segmentation accuracy under limited sample conditions, the model employs a support-query matching mechanism. This mechanism matches features from the support set with features from the query set, ensuring the query set receives the correct segmentation result. During training, support and query set sample pairs are dynamically constructed, and class-guided features and similarity prior features jointly guide the model to complete mask prediction. For the loss function, pixel-level binary cross-entropy loss is used to supervise and constrain the predicted and ground truth masks.

[0098] This invention provides a method for segmenting defect regions on the inner surface of a steel pipe by integrating multimodal information. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps:

[0099] S1. The main body of the detection device, including a hexagonal prism multimodal probe, simultaneously acquires RGB images and depth maps of defects on the inner surface of the steel pipe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotations are performed, forming a one-to-one correspondence three-dimensional data set with the RGB images and depth maps.

[0100] The detection device of this invention uses a high-strength telescopic mechanism and a hexagonal prism multimodal probe to achieve efficient acquisition of 360° image and depth information of the inner wall of the steel pipe, thereby improving the quality of raw data acquisition.

[0101] Optionally, such as Figure 3 As shown, the main body of the detection device includes a high-strength telescopic mechanism, a soft connection buffer structure, and a hexagonal prism-shaped multimodal probe;

[0102] The telescopic mechanism uses a high-strength telescopic rod, which can be adapted to steel pipes of different specifications and lengths.

[0103] The multimodal probe is flexibly connected to a telescopic rod via a strong spring, providing shock resistance and self-adaptation capabilities. The probe is shaped like a regular hexagonal prism, with six industrial-grade cameras and corresponding line structured light lasers evenly distributed on its six faces, forming a 360° all-view perception capability. When the device is working, driven by the control system, the probe moves forward along the steel pipe axis. The six cameras respectively capture images of the inner wall within a 60° central angle range. At the same time, the line array laser projects a grating onto the inner wall surface, and obtains the depth information of the inner wall by recording the changes in reflected light.

[0104] This invention, through the design of a high-strength telescopic mechanism and a hexagonal prism-shaped multimodal probe, achieves efficient acquisition of 360° image and depth information of the inner wall of a steel pipe, thereby improving the quality of raw data acquisition.

[0105] The data processor in this embodiment of the invention splices and unfolds the original images and depth maps in six directions according to geometric relationships, combines data augmentation and preprocessing operations, and generates corresponding semantic labels to construct a high-quality ternary dataset for multi-class defect identification.

[0106] Optionally, such as Figure 4 As shown, the data processor unfolds and stitches the original RGB images in six directions according to a preset geometric relationship, forming a two-dimensional unfolded image with the vertical dimension representing the circumference of the inner diameter of the steel pipe and the horizontal dimension representing the depth of penetration, as shown. Figure 5 As shown;

[0107] The depth images acquired simultaneously are also aligned and spatially stitched using corresponding relationships to ensure a perfect match with the RGB image;

[0108] Then, image preprocessing is performed:

[0109] Enhancement operations are performed on the stitched RGB images (including data augmentation strategies such as random horizontal mirroring, random scaling, and random cropping to increase sample diversity and improve the robustness of the model to different defect morphologies);

[0110] Meanwhile, the stitched depth image will be resized and normalized, and channel duplication will be performed to meet the input format requirements for subsequent model training.

[0111] Then, pixel-level defect areas are depicted and semantic labels are generated through manual annotation, forming a one-to-one correspondence of three-data sets with the RGB image and the depth image (because the depth information can be understood as another piece of information to assist RGB segmentation, so in this embodiment of the invention, only one annotation is needed on the RGB image, since the image size of the RGB image and the depth image are the same, so the RGB image can be directly annotated).

[0112] The annotation map supports the differentiation of labels for multiple types of defects, which facilitates classification training and evaluation.

[0113] S2. Using the aforementioned three-element data set, construct a support dataset and a query dataset for training the few-shot segmentation model. The support dataset includes datasets supporting RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ;

[0114] Optionally, the support dataset selects high-frequency category images from the defect samples on the inner surface of steel pipes as the main support image source, and also integrates general metal defect image samples from public datasets to form a training sample subset that is cross-material and cross-morphology, which is used to provide rich category priors during the training phase.

[0115] The query dataset is derived from defect images collected on a real steel pipe production line.

[0116] In constructing the support dataset, this invention takes into account the relatively large amount of certain types of defect data in industrial scenarios. Firstly, images of frequently occurring categories (such as rust, scratches, and oxide scale) from the inner surface defect samples of steel pipes are selected as the primary support image source. To enhance the model's cross-domain generalization and feature transfer capabilities, the support dataset also integrates general metal defect image samples from publicly available datasets, including strip steel, aluminum profile, and magnetic ceramic defect data from metal surface defect datasets such as FSSD-12 and Surface Defect-4i. This forms a cross-material, cross-morphology training sample subset to provide rich category priors during the training phase. In contrast, the query dataset primarily consists of defect images (such as cracks, dents, and bulges) collected from real steel pipe production lines.

[0117] Ultimately, each sample in the support set and query set contains an RGB image, a depth image, and a defect mask annotation image.

[0118] S3. Input the supporting RGB image and supporting depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network (whose parameters are completely shared, aiming to ensure that the supporting image and the query image have consistent representation and distribution characteristics in the feature space, thereby improving the accuracy and generalization ability of supporting-query feature matching in few-shot segmentation tasks) to extract the multi-scale aggregated features F of the supporting image respectively. S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ;

[0119] The multimodal feature extraction backbone network of this invention constructs a dual-branch backbone network to extract the RGB and depth map features of defects on the inner surface of the steel pipe, respectively. The feature interaction fusion is achieved through the attention enhancement module, thereby realizing structural detail perception and multimodal feature alignment.

[0120] Optionally, such as Figure 6 As shown, the multimodal feature extraction backbone network is a multi-layered feature extraction architecture, including four feature extraction layers and corresponding attention feature enhancement modules. The middle and lower layers are responsible for extracting fine-grained local features, while the higher layers focus on capturing broader semantic information. Through this hierarchical design, it can not only effectively distinguish features at different levels, but also maintain modality specificity during the fusion process.

[0121] Each feature extraction layer contains a multi-layer convolutional structure. The stacking of these convolutional layers gradually enhances the expressive power of the features. The backbone network receives RGB and depth images as joint input and performs feature extraction through a parallel dual-branch structure. The RGB branch focuses on capturing global semantic information and texture features, while the depth branch focuses on modeling local geometric changes and spatial details. After the outputs of each feature extraction layer, the two branches are fused through an attention feature enhancement module. This achieves semantic alignment and expression enhancement while maintaining modality specificity. To highlight structural details and discriminative ability, the backbone network in the middle layer stage (i.e., F...)... RGB-2 and F RGB-3 The fused features are output separately and then concatenated. Through information integration and feature dimensionality reduction, multi-scale aggregated features of the model are generated. Simultaneously, in the high-level stage of feature extraction (i.e., F...),... RGB-4 The output fusion features retain their modal fusion form and are used as multimodal semantic features.

[0122] The attention enhancement module in this embodiment of the invention integrates and interacts with depth and RGB modal features through differential depthwise convolution and differential dilated convolution, respectively, to improve the ability of multimodal features of defects in the inner wall of steel pipe to express complex defect regions, and to provide support for subsequent mask generation and feature alignment.

[0123] Optionally, such as Figure 7 As shown, the attention feature enhancement module receives the RGB modal features and deep modal features output from each layer of the multimodal feature extraction backbone network, and performs structure-aware enhancement and context-dependent modeling on the two types of modal features respectively through a differential depth convolution structure, thereby improving the expressive ability of the fused features for complex defect regions. The whole module consists of two parallel paths.

[0124] The deep modal path includes:

[0125] Based on differential convolutional attention, the input depth feature map is first compressed through a 1×1×C / 8 convolution operation to reduce computational complexity.

[0126] The compressed features are then fed into a differential depthwise convolution to achieve adaptive modeling of local geometry. This differential depthwise convolution dynamically adjusts the convolution weights by constructing a differential convolution kernel. Its core calculation method is as follows: for any convolution center position P and its neighboring offset position P+δ i Each convolution weight of the differential convolution kernel The calculation formula is:

[0127]

[0128] Where X(P) represents the value of the input feature map at pixel position P, δi The relative offset position is represented by |·|, which represents the L1 norm. The exp(·) function is used to construct an exponential decay factor based on pixel differences, which suppresses the response of geometrically abrupt regions during convolution.

[0129] The output features of the differential depth convolution operation are used to generate an attention map through a 1×1 convolution, and then perform element-wise multiplication with the input features before convolution. The original channel dimension is then restored through a 1×1×C convolution, and finally residual summation is performed with the initial input features to output the enhanced depth output features.

[0130] RGB modal path:

[0131] Differential depthwise dilated convolution is employed, introducing a dilation factor r on top of differential depthwise convolution to significantly expand the receptive field of the convolution while maintaining stable computational resources. This is used to model semantic dependencies between distant pixels in an image. The convolution weight formula is adjusted as follows:

[0132]

[0133] Where r represents the dilation factor of dilated convolution;

[0134] The output after the differential depth dilation convolution operation is subjected to 1×1 convolution, followed by element-wise multiplication, original channel dimension restoration, and residual connection. Finally, it is added element-wise with the depth output feature to obtain the enhanced RGB output feature.

[0135] S4, the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature F A The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map.

[0136] The support graph semantic guidance module of this invention extracts defect features through a triple global pooling method and combines it with attention branch weighting of key defect regions to further refine the defect features in the support image, thereby improving the model's ability to perceive known defects under limited sample conditions.

[0137] Optionally, such as Figure 8 As shown, the support graph semantic guidance module extracts and generates class guidance features F by utilizing the annotation information of known defects in the support dataset. A This is used to highlight key information related to the target category in the feature space, enabling the model to selectively focus on regions matching the target category when processing query images. The processing procedure is as follows:

[0138] F S and M S Multiply to calculate the defect features F in the supporting image. D ;

[0139] Defect feature F D The triple pooling component, consisting of global attention pooling, global average pooling, and global max pooling, sums the results element by element and calculates the vector average. Then, it is vector-expanded to restore the original feature size. The formula is as follows:

[0140]

[0141] Where η is the averaging factor, GP(·) represents global average pooling, MP(·) represents global max pooling, and AP(·) represents global attention pooling. The results of the three aggregation operations are averaged to provide more comprehensive global semantic information.

[0142] To further enhance the focus on defect areas, considering the spatial feature differences of different modal information, an attention mechanism is introduced to focus on key defect areas in the image: the global attention pooling result is weighted according to the importance of different regions in the image by calculating attention weight A. The formula for calculating the attention weight is as follows:

[0143] A=σ(W1·ReLU(W2·AP(F S )+b2)+b1)

[0144] Where W1 and W2 are learnable weight matrices, b1 and b2 are bias terms, σ(·) is the sigmoid activation function, and the output A is the result of global attention pooling, AP(F). S Attention maps with the same spatial dimensions;

[0145] Next, the attention map is compared with F. S Add them through residual connection, and then combine them with F after vector expansion.P The final guided feature F is obtained by concatenating the data along the channel dimension. A The calculation formula is as follows:

[0146] F A =Concat((AP(F) S )⊙A)+F S ,ε(F P ))

[0147] Where Concat(a,b) means concatenating a and b along the channel dimension, ⊙ means dot product of two matrices, and ε(·) means vector expansion.

[0148] Optionally, the similarity prior feature generation module calculates and generates similarity prior features F by comparing the multimodal semantic features of the support graph and the query graph. M The process quantifies the semantic similarity between each pixel in the query image and the defective region in the support image, providing prior information based on cross-image association for subsequent segmentation. The process is as follows:

[0149] The I Q and I S After activation function processing, for I S It will also be through its binarized image annotation M S By setting the background region in the supporting image to zero and retaining only the foreground features related to the target category, X is obtained. Q and X S The formula is as follows:

[0150] X Q =ReLU(I Q )

[0151] X S =ReLU(I S )·M S

[0152] Where ReLU(·) represents the ReLU activation function;

[0153] Based on the X Q and X S For each pixel position in the semantic features of the query image Calculate its relationship with all supporting image foreground pixels. The cosine similarity between them is calculated using the following formula:

[0154]

[0155] Next, the maximum similarity value between each pixel in the query image and all foreground pixels in the support images is extracted and used as its pixel similarity prior response value. The announcement is as follows:

[0156]

[0157] After calculating the maximum similarity value for each pixel, the final result is the prior similarity response value of each pixel. The similarity feature C constitutes Q The image is deformed to generate prior features F with the same spatial dimension as the query image. M And use minimum-maximum normalization for F M The pixel values ​​in the image are normalized to a range of [0,1]. The normalization process is achieved using the following formula:

[0158]

[0159] Here, ε represents a minimal constant introduced to prevent division by zero errors.

[0160] The multi-scale aggregation feature F of the query image in this embodiment of the invention Q It preserves its own texture details and spatial structure information, ensuring that the model not only relies on external category guidance and similarity priors during the fusion process, but also makes full use of the original feature representation of the query graph. The fusion unit combines class-guided features, similarity prior features, and query graph multi-scale aggregated features F. Q The features are concatenated along the channel dimension and encoded, compressed, and interactively modeled through a unified feature aggregation structure. This enables semantic guidance, prior constraints, and complementary fusion of original features, resulting in a fused feature map that is consistent with the query map in the spatial dimension.

[0161] S5. The fused feature map is decoded by a multimodal decoder to generate a pixel-level prediction mask corresponding to the query image, thereby achieving the final defect region segmentation output.

[0162] The multimodal decoder in this embodiment of the invention enhances the fused features through a feature refinement module, channels, and convolutional attention modules, gradually guiding the activation of key regions in the few-sample query map, extracting fine texture and boundary information, and finally outputting a high-precision pixel-level defect prediction map via a convolutional segmentation head.

[0163] Optionally, such as Figure 9 As shown, the multimodal decoder uses a feature refinement module, a convolutional attention module, and a convolutional segmentation head to perform information enhancement and classification prediction on the feature map;

[0164] The feature refinement module includes two cascaded substructures. Each substructure includes a set of normalization layers, a smooth rectified activation function, and a 3×3 convolution operation, which are used to perform nonlinear transformation and local structure modeling on the input multi-channel feature map. The two sets of substructures are connected by skip connections to form a residual enhancement path to improve the continuity and stability of feature expression.

[0165] After the output of the first feature refinement module, a convolutional attention module is further introduced to enhance the feature response capability in the channel dimension and spatial dimension.

[0166] The feature map output by the convolutional attention module is then fed into the second feature refinement module to further extract the local texture and boundary information enhanced by the attention mechanism.

[0167] The feature map output by the second feature refinement module is finally input into the convolutional segmentation head, which consists of two sets of feature refinement modules and a 1×1 convolutional layer. This maps the high-dimensional channel features to a one-dimensional output channel and outputs a defect region segmentation result map with the same spatial dimension as the input image.

[0168] Among them, such as Figure 10 As shown, the convolutional attention module includes two parts: channel attention and spatial attention, which improves the multimodal decoder's final segmentation capability of the target region while maintaining a lightweight design.

[0169] The channel attention obtains global feature information of different channels through global average pooling and global max pooling, and calculates the importance of each channel in combination with a fully connected layer. At the same time, it is activated by sigmoid to strengthen key channels and suppress irrelevant channels. Then it is multiplied with the original input features to obtain the channel features after channel attention weighting.

[0170] The spatial attention feature utilizes the channel features, performs max pooling and average pooling in the channel dimension, concatenates the results, and generates spatial attention features through 7×7 convolution and sigmoid activation to highlight salient regions and reduce background interference.

[0171] Finally, the channel features and spatial attention features are fused element-wise, enabling the multimodal decoder to more accurately focus on task-related features (adaptively highlighting key features and suppressing irrelevant information), thereby improving overall segmentation performance.

[0172] In the training process, this invention employs a few-shot segmentation training strategy based on "support-query". Specifically, in each training round, sample pairs are dynamically sampled from the support and query datasets to construct training task units. This simulates the application scenario of using a small number of examples to guide the segmentation of new samples during the actual inference stage. It supports the joint input of the image and query image to a parameter-shared multimodal feature extraction backbone network to extract multi-scale aggregated features F. S and F Q and multimodal semantic features I S and I Q And maintaining modal consistency, the support graph semantic guidance module then utilizes the annotation mask M of the support graph. S Generate class guiding feature F A The similarity prior feature generation module calculates the pixel-level similarity between the support graph and the query graph in the semantic space to obtain the similarity prior feature F. M Then, the multi-scale aggregation feature F of the query graph. Q Similar prior features F M With class-guided feature F A Feature fusion is performed in a multi-feature fusion unit, and the fusion result is further decoded by a multimodal decoder to output a predicted mask. During training and optimization, a pixel-level binary cross-entropy loss function is used to supervise and constrain the predicted mask and the ground truth mask. The network parameters are updated through backpropagation, thereby gradually improving the model's segmentation accuracy and cross-class generalization ability under conditions with few samples.

[0173] During the inference phase, the model performs defect segmentation on query images collected in real-time from the production line using a small number of labeled support image samples. Specifically, during inference, a small number of typical defect samples are first selected manually or by a pre-defined method as the support set. Each support sample contains an RGB image, a depth image, and corresponding defect mask annotations, and remains fixed throughout the inference process or is updated when necessary. Then, the detection device collects RGB and depth images of the inner surface of the steel pipe as query inputs. The query samples contain only image and depth data without annotations. The support set and query set are respectively input into a multimodal feature extraction backbone network with shared parameters to extract multi-scale aggregated features F. S and F Q and multimodal semantic features I S and I Q Supports the use of icon annotation M in the graph semantic guidance module S Generate class guiding feature F A The similarity prior feature generation module is based on the multimodal semantic features I of the support graph and the query graph. S and I Q And support for icon annotation M S Similarity calculations are performed to obtain the prior similarity features F between the query image and the supporting images.M Then, the multi-scale aggregation feature F of the query graph. Q Similar prior features F M With class-guided feature F A Channel splicing and encoding compression are performed in the multi-feature fusion unit and then fed into the multimodal decoder. After enhancement by the feature refinement module and the convolutional attention module, the convolutional segmentation head outputs a pixel-level defect prediction mask with the same spatial size as the input image. Finally, the generated defect prediction result is superimposed on the original query image to accurately mark the defect area on the inner surface of the steel pipe, realizing high-precision defect segmentation and quality detection in the inference stage.

[0174] like Figure 11 As shown, this embodiment of the invention also provides a steel pipe inner surface defect region segmentation system that integrates multimodal information, the system comprising:

[0175] The acquisition module 1110 is used to simultaneously acquire RGB images and depth maps of defects on the inner surface of steel pipes through the main body of the detection device including a hexagonal multimodal probe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotation is performed, forming a one-to-one corresponding three-data group with the RGB image and the depth map.

[0176] Module 1120 is used to construct a support dataset and a query dataset for training a few-shot segmentation model using the three-element data set. The support dataset includes datasets that support RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ;

[0177] Extraction module 1130 is used to input the supporting RGB image and supporting depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network to extract multi-scale aggregated features F of the supporting image respectively. S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ;

[0178] Fusion module 1140, used to fuse the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature FA The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map.

[0179] The decoding module 1150 is used to decode the fused feature map by the multimodal decoder, generate a pixel-level prediction mask corresponding to the query image, and achieve the final defect region segmentation output.

[0180] The present invention provides a steel pipe inner surface defect region segmentation system that integrates multimodal information. Its functional structure corresponds to the steel pipe inner surface defect region segmentation method that integrates multimodal information provided in the present invention, and will not be described again here.

[0181] Figure 12 This is a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present invention. The electronic device 1200 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1201 and one or more memories 1202. The memory 1202 stores at least one instruction, which is loaded and executed by the processor 1201 to implement the steps of the above-described method for segmenting the defect region on the inner surface of a steel pipe by fusing multimodal information.

[0182] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the above-described method for segmenting the defect region on the inner surface of a steel pipe by fusing multimodal information. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0183] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for segmenting defect regions on the inner surface of a steel pipe by integrating multimodal information, characterized in that, The method includes: S1. The main body of the detection device, including a hexagonal prism multimodal probe, simultaneously acquires RGB images and depth maps of defects on the inner surface of the steel pipe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotations are performed, forming a one-to-one correspondence three-dimensional data set with the RGB images and depth maps. S2. Using the aforementioned three-element data set, construct a support dataset and a query dataset for training the few-shot segmentation model. The support dataset includes datasets supporting RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ; S3. Input the supported RGB image and supported depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network to extract the multi-scale aggregated features F of the supporting image respectively. S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ; S4, the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature F A The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map. S5. The fused feature map is decoded by a multimodal decoder to generate a pixel-level prediction mask corresponding to the query image, thereby achieving the final defect region segmentation output.

2. The method according to claim 1, characterized in that, The main body of the detection device includes a high-strength telescopic mechanism, a soft connection buffer structure, and a hexagonal prism-shaped multimodal probe; The telescopic mechanism uses a high-strength telescopic rod, which can be adapted to steel pipes of different specifications and lengths. The multimodal probe is flexibly connected to a telescopic rod via a strong spring, providing shock resistance and self-adaptation capabilities. The probe is shaped like a regular hexagonal prism, with six industrial-grade cameras and corresponding line structured light lasers evenly distributed on its six faces, forming a 360° all-view perception capability. When the device is working, driven by the control system, the probe moves forward along the steel pipe axis. The six cameras respectively capture images of the inner wall within a 60° central angle range. At the same time, the line array laser projects a grating onto the inner wall surface, and obtains the depth information of the inner wall by recording the changes in reflected light.

3. The method according to claim 2, characterized in that, The data processor unfolds and stitches the original RGB images in six directions according to a preset geometric relationship to form a two-dimensional unfolded image with the vertical dimension being the circumference of the inner diameter of the steel pipe and the horizontal dimension being the depth of advancement. The depth images acquired simultaneously are also aligned and spatially stitched using corresponding relationships to ensure a perfect match with the RGB image; Then, image preprocessing is performed: Enhancement operations are performed on the stitched RGB image; Meanwhile, the stitched depth image will be resized and normalized, and channel duplication will be performed to meet the input format requirements for subsequent model training.

4. The method according to claim 1, characterized in that, The support dataset selects high-frequency category images from the inner surface defect samples of steel pipes as the main support image source, and also integrates general metal defect image samples from public datasets to form a training sample subset that is cross-material and cross-morphology, which is used to provide rich category priors during the training phase. The query dataset is derived from defect images collected on a real steel pipe production line.

5. The method according to claim 1, characterized in that, The multimodal feature extraction backbone network is a multi-layered feature extraction architecture, including four feature extraction layers and corresponding attention feature enhancement modules. The middle and lower layers are responsible for extracting fine-grained local features, while the higher layers focus on capturing broader semantic information. Through this hierarchical design, it can not only effectively distinguish features at different levels, but also maintain modality specificity during the fusion process. Each feature extraction layer contains a multi-layer convolutional structure. The expressive power of the features is gradually enhanced by stacking convolutional layers. The backbone network receives RGB images and depth images as joint inputs and performs feature extraction through a parallel dual-branch structure. The RGB branch focuses on capturing global semantic information and texture features, while the depth branch focuses on modeling local geometric changes and spatial details. After the outputs of each feature extraction layer, the two branches are fused through an attention feature enhancement module. This achieves semantic alignment and expression enhancement while maintaining modality specificity. To highlight structural details and discriminative ability, the backbone network outputs fused features in the middle layers and then concatenates them. Through information integration and feature dimensionality reduction, multi-scale aggregated features of the model are generated. At the same time, the fused features output in the high-level feature extraction stages retain their modality fusion form and serve as multimodal semantic features.

6. The method according to claim 5, characterized in that, The attention feature enhancement module receives RGB modal features and deep modal features output from each layer of the multimodal feature extraction backbone network, and performs structure-aware enhancement and context-dependent modeling on the two types of modal features respectively through a differential depth convolution structure, thereby improving the expressive ability of the fused features for complex defect regions. The whole module consists of two parallel paths. The deep modal path includes: Based on differential convolutional attention, the input depth feature map is first compressed through a 1×1×C / 8 convolution operation to reduce computational complexity. The compressed features are then fed into a differential depthwise convolution to achieve adaptive modeling of local geometry. This differential depthwise convolution dynamically adjusts the convolution weights by constructing a differential convolution kernel. Its core calculation method is as follows: for any convolution center position P and its neighboring offset position P+δ i Each convolution weight of the differential convolution kernel The calculation formula is: Where X(P) represents the value of the input feature map at pixel position P, δ i The relative offset position is represented by |·|, which represents the L1 norm. The exp(·) function is used to construct an exponential decay factor based on pixel differences, which suppresses the response of geometrically abrupt regions during convolution. The output features of the differential depth convolution operation are used to generate an attention map through a 1×1 convolution, and then perform element-wise multiplication with the input features before convolution. The original channel dimension is then restored through a 1×1×C convolution, and finally residual summation is performed with the initial input features to output the enhanced depth output features. RGB modal path: Differential depthwise dilated convolution is employed, introducing a dilation factor r on top of differential depthwise convolution to significantly expand the receptive field of the convolution while maintaining stable computational resources. This is used to model semantic dependencies between distant pixels in an image. The convolution weight formula is adjusted as follows: Where r represents the dilation factor of dilated convolution; The output after the differential depth dilation convolution operation is subjected to 1×1 convolution, followed by element-wise multiplication, original channel dimension restoration, and residual connection. Finally, it is added element-wise with the depth output feature to obtain the enhanced RGB output feature.

7. The method according to claim 1, characterized in that, The support graph semantic guidance module extracts and generates class guidance features F by utilizing the annotation information of known defects in the support dataset. A This is used to highlight key information related to the target category in the feature space, enabling the model to selectively focus on regions matching the target category when processing query images. The processing procedure is as follows: F S and M S Multiply to calculate the defect features F in the supporting image. D ; Defect feature F D The triple pooling component, consisting of global attention pooling, global average pooling, and global max pooling, sums the results element by element and calculates the vector average. Then, it is vector-expanded to restore the original feature size. The formula is as follows: Where η is the averaging factor, GP(·) represents global average pooling, MP(·) represents global max pooling, and AP(·) represents global attention pooling. The results of the three aggregation operations are averaged to provide more comprehensive global semantic information. To further enhance the focus on defect areas, considering the spatial feature differences of different modal information, an attention mechanism is introduced to focus on key defect areas in the image: the global attention pooling result is weighted according to the importance of different regions in the image by calculating attention weight A. The formula for calculating the attention weight is as follows: A=σ(W1·ReLU(W2·AP(F S )+b2)+b1) Where W1 and W2 are learnable weight matrices, b1 and b2 are bias terms, σ(·) is the sigmoid activation function, and the output A is the result of global attention pooling, AP(F). S Attention maps with the same spatial dimensions; Next, the attention map is compared with F. S Add them through residual connection, and then combine them with F after vector expansion. P The final guided feature F is obtained by concatenating the data along the channel dimension. A The calculation formula is as follows: F A =Concat((AP(F S )⊙A)+F S ,ε(F P )) Where Concat(a,b) means concatenating a and b along the channel dimension, ⊙ means dot product of two matrices, and ε(·) means vector expansion.

8. The method according to claim 1, characterized in that, The similarity prior feature generation module calculates and generates similarity prior features F by comparing the multimodal semantic features of the support graph and the query graph. M The process quantifies the semantic similarity between each pixel in the query image and the defective region in the support image, providing prior information based on cross-image association for subsequent segmentation. The process is as follows: The I Q and I S After activation function processing, for I S It will also be through its binarized image annotation M S By setting the background region in the supporting image to zero and retaining only the foreground features related to the target category, X is obtained. Q and X S The formula is as follows: X Q =ReLU(I Q ) X S =ReLU(I S )·M S Where ReLU(·) represents the ReLU activation function; Based on the X Q and X S For each pixel position in the semantic features of the query image Calculate its relationship with all supporting image foreground pixels. The cosine similarity between them is calculated using the following formula: Next, the maximum similarity value between each pixel in the query image and all foreground pixels in the support images is extracted and used as its pixel similarity prior response value. The announcement is as follows: After calculating the maximum similarity value for each pixel, the final result is the prior similarity response value of each pixel. The similarity feature C constitutes Q The image is deformed to generate prior features F with the same spatial dimension as the query image. M And use minimum-maximum normalization for F M The pixel values ​​in the image are normalized to a range of [0,1]. The normalization process is achieved using the following formula: Here, ε represents a minimal constant introduced to prevent division by zero errors.

9. The method according to claim 1, characterized in that, The multimodal decoder employs a feature refinement module, a convolutional attention module, and a convolutional segmentation head to enhance information and perform classification prediction on the feature map; The feature refinement module includes two cascaded substructures. Each substructure includes a set of normalization layers, a smooth rectified activation function, and a 3×3 convolution operation, which are used to perform nonlinear transformation and local structure modeling on the input multi-channel feature map. The two sets of substructures are connected by skip connections to form a residual enhancement path to improve the continuity and stability of feature expression. After the output of the first feature refinement module, a convolutional attention module is further introduced to enhance the feature response capability in the channel dimension and spatial dimension. The feature map output by the convolutional attention module is then fed into the second feature refinement module to further extract the local texture and boundary information enhanced by the attention mechanism. The feature map output by the second feature refinement module is finally input into the convolutional segmentation head, which consists of two sets of feature refinement modules and a 1×1 convolutional layer. This maps the high-dimensional channel features to a one-dimensional output channel and outputs a defect region segmentation result map with the same spatial dimension as the input image. The convolutional attention module includes two parts: channel attention and spatial attention, which improves the final segmentation capability of the multimodal decoder for the target region while maintaining a lightweight design. The channel attention obtains global feature information of different channels through global average pooling and global max pooling, and calculates the importance of each channel in combination with a fully connected layer. At the same time, it is activated by sigmoid to strengthen key channels and suppress irrelevant channels. Then it is multiplied with the original input features to obtain the channel features after channel attention weighting. The spatial attention feature utilizes the channel features, performs max pooling and average pooling in the channel dimension, concatenates the results, and generates spatial attention features through 7×7 convolution and sigmoid activation to highlight salient regions and reduce background interference. Finally, the channel features and spatial attention features are multiplied and fused element-wise, enabling the multimodal decoder to more accurately focus on task-related features and improve overall segmentation performance.

10. A system for segmenting defect regions on the inner surface of a steel pipe by integrating multimodal information, characterized in that, The system includes: The acquisition and processing module is used to simultaneously acquire RGB images and depth maps of defects on the inner surface of steel pipes through the main body of the detection device including a hexagonal multimodal probe. After the RGB images and depth maps are processed by the data processor, pixel-level defect area annotation is performed, forming a one-to-one correspondence three-data set with the RGB image and depth map. The building module is used to construct a support dataset and a query dataset for training a few-shot segmentation model using the three-element data set. The support dataset includes datasets that support RGB images, depth maps, and icon annotations. S The query dataset includes query RGB images, query depth maps, and query icon annotations M. q ; The extraction module is used to input the supported RGB image and supported depth image into the supporting image feature extractor, and input the query RGB image and query depth image into the query image feature extractor. The supporting image feature extractor and the query image feature extractor share a multimodal feature extraction backbone network to extract the multi-scale aggregated features F of the supporting image. S Supports multimodal semantic features of images. S And query image multi-scale aggregation feature F Q Query image multimodal semantic features I Q ; The fusion module is used to integrate the F S I S F Q I Q Input a multi-feature fusion processor, the multi-feature fusion processor's support graph semantic guidance module using the F S and M S Generate class guiding feature F A The multi-feature fusion unit uses the I-type feature generation module to highlight key information related to the target category in the feature space. S I Q and M S Generate similar prior features F M To quantify the semantic similarity between each pixel in the query image and the defective region in the support image, the F... A F M With F Q The features are concatenated along the channel dimension to output a fused feature map. The decoding module is used to decode the fused feature map by the multimodal decoder, generate a pixel-level prediction mask corresponding to the query image, and achieve the final defect region segmentation output.

Images