An RGB-D multi-modal segmentation method and system for surface defects of cold-rolled plate strips

Abstract

The application discloses a cold-rolled plate strip surface defect RGB-D multi-modal segmentation method and system, and belongs to the technical field of metal plate strip surface quality detection and computer vision. In view of the problems of low contrast, complex background texture, diverse defect morphology and insufficient robustness of single modal detection of the cold-rolled plate strip surface defect, the RGB image and the corresponding depth image of the surface of the cold-rolled plate strip to be detected are acquired; the RGB feature and the depth feature are extracted by using a double-flow coding network; the multi-level features are fused by using a double-branch decoding network, one branch of which performs cross-modal layer-by-layer focusing fusion, and the other branch performs multi-scale analytical fusion, and the defect feature expression of different scales is enhanced through a multi-scale feature analytical module; and the plate strip surface defect segmentation result is obtained by combining a hybrid supervision strategy of step-by-step output supervision and deep-shallow double supervision and training a combined loss function.

Description

Technical Field

[0001] This invention belongs to the field of metal sheet and strip surface quality inspection and computer vision technology, specifically relating to an RGB-D multimodal segmentation method and system for surface defects in cold-rolled sheet and strip. Background Technology

[0002] Cold-rolled steel strip is widely used in critical fields such as automobile manufacturing and aerospace. Its surface quality directly determines the performance, appearance, and service life of the products. Existing surface defect detection methods for cold-rolled steel strip mainly include manual inspection and machine vision-based automated inspection, with two-dimensional image detection methods based on single-modal information being more commonly used. However, surface defects in cold-rolled steel strip generally exhibit low contrast, diverse shapes, and complex boundaries, and are easily affected by the production environment and background textures. Single-modal information is insufficient to fully characterize defect features, leading to problems such as missed detections, false detections, and insufficient segmentation accuracy in complex scenarios using existing methods. Summary of the Invention

[0003] To address the problems of low contrast, complex background texture, diverse defect morphologies, and insufficient robustness of single-modal detection in cold-rolled strip surface defects, this invention provides an RGB-D multimodal segmentation method and system for cold-rolled strip surface defects, thereby improving the accuracy, robustness, and adaptability of cold-rolled strip surface defect segmentation.

[0004] To achieve the above objectives, the present invention employs the following technical solutions:

[0005] A method for RGB-D multimodal segmentation of surface defects in cold-rolled steel strip, the method comprising the following steps:

[0006] S1: Construct an RGB-D multimodal segmentation network model for surface defects of cold-rolled strip, the network model including a dual-stream coding network, a dual-branch decoding network and a multi-scale feature parsing module; use an industrial camera and a laser sensor to acquire RGB images of the surface of the cold-rolled strip to be inspected and depth images corresponding to the RGB image space;

[0007] The RGB image and depth image in step S1 correspond to the same surface area of ​​the cold-rolled strip.

[0008] The RGB image is used to characterize the texture, grayscale, edge and defect appearance information of the surface of cold-rolled strip;

[0009] The depth image is used to characterize the geometric undulations, spatial depth, and defect morphology information of the surface of cold-rolled strip.

[0010] The RGB image and the depth image correspond one-to-one in spatial location and are used as dual-modal inputs for surface defect segmentation of cold-rolled strip.

[0011] S2: Input the RGB image and depth image into the dual-stream coding network in the network model respectively to extract RGB multi-level features and depth multi-level features;

[0012] The dual-stream coding network in step S2 includes an RGB coding branch and a depth coding branch; wherein:

[0013] The RGB encoding branch is used to extract features from the RGB image layer by layer and output multi-level RGB features.

[0014] The depth coding branch is used to extract features layer by layer from the depth image and output multi-level depth features;

[0015] The RGB encoding branch and the depth encoding branch are set independently and output separately. Level features, the The level features correspond to different spatial resolutions and different semantic levels according to the network hierarchy from shallow to deep, and are used for cross-modal layer-by-layer focused fusion and multi-scale parsing fusion of subsequent dual-branch decoding networks.

[0016] The dual-stream coding network uses VGG16 pre-trained on ImageNet as the backbone network, which is used to extract features from RGB images and depth images respectively.

[0017] The RGB encoding branch and the depth encoding branch each receive the input image and pass it sequentially through the VGG16 backbone network. Each stage outputs... The features are denoted as RGB multi-level features and deep multi-level features, respectively; where the number of channels in each feature map is denoted as... , The total number of layers is given, with the number of channels in each layer increasing sequentially. The initial weights of the dual-stream coding network use ImageNet pre-trained parameters to improve the convergence speed of network training and the quality of feature extraction. The multi-level features output by the two coding branches correspond one-to-one at each layer, which are then used by the subsequent dual-branch decoding network for cross-modal fusion.

[0018] S3: The RGB multi-level features and deep multi-level features are input into the dual-branch decoding network in the network model for fusion processing; wherein, the first decoding branch performs cross-modal layer-by-layer focusing fusion to obtain cross-modal focusing fusion features; the second decoding branch performs deep semantic extraction and fusion to obtain deep fusion features;

[0019] The dual-branch decoding network in step S3 includes a first decoding branch and a second decoding branch, wherein:

[0020] The first decoding branch is used to perform cross-modal layer-by-layer focusing fusion from deep to shallow on the RGB multi-level features, deep multi-level features, and deep fusion features generated by the second decoding branch to obtain cross-modal focused fusion features.

[0021] The second decoding branch is used to decode the code from the first... The deep semantic features extracted at the encoding level are extracted and fused to obtain deep fused features, which are then used by the subsequent multi-scale feature parsing module for step-by-step parsing and fusion.

[0022] The first and second decoding branches work together to generate fused features for subsequent multi-scale feature parsing modules to process.

[0023] The first decoding branch of the dual-branch decoding network in step S3 includes:

[0024] Feature preprocessing unit: Performs feature preprocessing on RGB multi-level features and depth multi-level features respectively to obtain preprocessed RGB features and preprocessed depth features;

[0025] No. Hierarchical fusion unit: for the first RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are concatenated and fused to obtain the first... Hierarchical cross-modal focused fusion features; among which... This represents the total number of levels.

[0026] Layer-by-layer focusing fusion unit: For the first The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together to obtain the first... Hierarchical cross-modal focused fusion features, among which This is to achieve a gradual, focused integration from deep to shallow layers;

[0027] The specific operation of the first decoding branch is as follows:

[0028] Input RGB multi-level features, deep multi-level features, and deep fusion features, wherein the deep fusion features are generated and provided by the second decoding branch;

[0029] Feature preprocessing is performed on RGB multi-level features and deep multi-level features respectively;

[0030] For the first RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are spliced ​​and fused to generate the first... Hierarchical cross-modal focused fusion features;

[0031] Feature upsampling fusion of remaining layers: For the first layer The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together. ;

[0032] Generate cross-modal focused fusion features at the current level;

[0033] The cross-modal features are transmitted and fused layer by layer from deep to shallow.

[0034] Output cross-modal focusing and fusion features of each layer to enhance the correlation expression between different modal features and the focusing ability on surface defect areas of cold-rolled strip;

[0035] The first decoding branch is used to enhance the RGB multi-level features and deep multi-level features output by the dual-stream coding network and achieve cross-modal fusion representation. First, the RGB multi-level features and deep multi-level features output by each level of the dual-stream coding network are input into the CBR module for feature preprocessing to obtain preprocessed RGB features and preprocessed deep features. This process is represented as follows: ;

[0036] In the formula, Indicates the first RGB features after hierarchical preprocessing Indicates the first Deep features after hierarchical preprocessing; the CBR module consists of convolutional layers, batch normalization layers, and activation function layers;

[0037] Subsequently, the first decoding branch performs layer-by-layer cross-modal fusion in a deep-to-shallow manner; among which, the first The hierarchical features are concatenated and fused with the deep fusion features output from the second decoding branch to obtain the first... Hierarchical cross-modal focusing fusion features; for other levels Then the current number will be... RGB multi-level features after hierarchical preprocessing, deep multi-level features after preprocessing, and the first The hierarchical cross-modal focusing fusion features are concatenated and fused with the upsampled results through the feature processing module to obtain the first... Hierarchical cross-modal focusing and fusion of features, the process of which is represented as follows: ; ; ;

[0038] in, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first feature after feature preprocessing. Hierarchical RGB features Indicates the first feature after feature preprocessing. Hierarchical depth features Indicates deep fusion characteristics, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first The result of hierarchical cross-modal focusing fusion features after feature processing and upsampling. This indicates a splicing operation. Indicates an upsampling operation; Indicates the first feature after feature preprocessing. Level RGB features; Indicates the first feature after feature preprocessing. Level-deep features.

[0039] The second decoding branch of the dual-branch decoding network in step S3 includes:

[0040] Deep semantic extraction unit: for the first Hierarchical RGB features and the first The hierarchical depth features are convolved and activated respectively to obtain the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features;

[0041] Deep semantic fusion unit: for the first Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused to obtain deep fused features;

[0042] The specific operation of the second decoding branch is as follows:

[0043] Enter the first Hierarchical RGB features, first Hierarchical depth features and cross-modal focusing fusion features at each level output by the first decoding branch; for the first Hierarchical RGB features and the first The hierarchical depth features are subjected to convolution and activation processing respectively; for the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused; the deep fused features are output for subsequent multi-scale feature parsing modules to call.

[0044] The second decoding branch is used to perform step-by-step parsing and fusion of defect features and output the corresponding segmentation map; firstly, for the first... Hierarchical RGB features with depth features Each feature is processed separately to obtain the corresponding deep semantic features; then, the obtained deep semantic features are concatenated and fused to obtain the deep fused feature. The process is represented as: ;

[0045] in, express Convolution operation, This indicates an activation operation. Indicates a splicing operation; Indicates the feature mapping process of the first Hierarchical deep RGB features; Indicates the feature mapping process of the first Hierarchical depth features;

[0046] Subsequently, the deep fusion features and the cross-modal focused fusion features output from the first decoding branch are input step by step into the MSFA multi-scale feature parsing module for layer-by-layer parsing and fusion of multi-scale information to obtain multi-scale semantic fusion features; this process is represented as: ; ;

[0047] in, This represents deep semantic fusion features. Indicates the first Multi-scale fusion features of layers This represents the multi-scale feature parsing module.

[0048] After obtaining multi-scale semantic fusion features at various levels Then, it is upsampled to the input resolution, and the corresponding segmentation output image is generated through independent convolution operations. And combine real labels for multi-scale supervision.

[0049] S4: Input the cross-modal focusing fusion features output from the first decoding branch and the deep fusion features output from the second decoding branch into the multi-scale feature parsing module, and perform step-by-step multi-scale parsing and fusion, combining the detail features from the first encoding level with those from the second encoding level. The semantic features at the encoding level are collaboratively enhanced to obtain the fused features required for defect segmentation after fusion.

[0050] The multi-scale feature parsing module in step S4 includes:

[0051] Channel attention enhancement unit: Performs channel attention enhancement on the input high-level semantic features and low-level detail features respectively to obtain enhanced high-level semantic features and enhanced low-level detail features; the high-level semantic features come from the first channel attention enhancement unit of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network;

[0052] Multi-scale semantic extraction unit: Parallel feature extraction is performed on the enhanced high-level semantic features using different convolutional kernel scales to obtain high-level semantic features under different receptive fields;

[0053] Preliminary Feature Fusion Unit: The enhanced low-level detail features, enhanced high-level semantic features, and high-level semantic features under different receptive fields are preliminarily spliced ​​and fused to obtain preliminary fused features;

[0054] Feature weighting enhancement unit: Generates weight information based on the preliminary fused features, and performs weighting processing on the high-level semantic features and low-level detail features before enhancement to obtain weighted high-level semantic features and weighted low-level detail features.

[0055] Fusion Enhancement Output Unit: The weighted high-level semantic features, weighted low-level detail features, and the preliminary fusion features are concatenated and fused again, and multi-scale fusion features are obtained through feature enhancement operations;

[0056] The specific operation of the multi-scale feature parsing module is as follows:

[0057] Input high-level semantic features and low-level detail features; wherein, the high-level semantic features come from the first level of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network;

[0058] Channel attention enhancement is applied to the high-level semantic features and low-level detail features of the input respectively to obtain the enhanced high-level semantic features and low-level detail features;

[0059] Parallel feature extraction was performed on the enhanced high-level semantic features using convolutional operations of different scales to obtain multi-scale high-level semantic features under different receptive fields.

[0060] The enhanced low-level detail features, enhanced high-level semantic features, and multi-scale high-level semantic features are concatenated and fused, and feature processing is performed to obtain the initial fused features.

[0061] Weight information is generated based on the initial fusion features, and the high-level semantic features and low-level detail features before enhancement are weighted to enhance the feature response related to defects.

[0062] The weighted high-level semantic features, weighted low-level detail features, and initial fusion features are concatenated and fused again, and feature processing is performed to obtain secondary fusion features.

[0063] The secondary fusion features are enhanced to highlight the features of the defect area and suppress background interference;

[0064] The enhanced multi-scale fusion features are output for subsequent segmentation result generation;

[0065] First, high-level semantic features and low-level detail features A channel attention mechanism is introduced to enhance important channel features, resulting in enhanced high-level semantic features. and enhanced low-level detail features The process is represented as: ;

[0066] In the formula, Indicates channel attention operation;

[0067] Parallel feature extraction is performed on the enhanced high-level semantic features using convolutional kernels of different scales to obtain high-level semantic features under different receptive fields, and these features are then combined with the enhanced low-level detail features. Preliminary fusion was performed to obtain preliminary fusion characteristics. The process is represented as: ;

[0068] in, This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation; This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation;

[0069] Subsequently, to enhance the expressive power of the initial fusion features, weight information is generated based on them. The high-level semantic features and low-level detail features before enhancement are weighted and enhanced to obtain the relabeled high-level semantic features. Low-level details The process is represented as: ;

[0070] in, This represents the sigmoid activation function. express Average pooling operation;

[0071] Furthermore, the initial fusion characteristics will be... After a The convolution operation yields a preliminary fused feature mapping feature. Then, the relabeled high-level semantic features, low-level detail features, preliminary fusion features, and mapping features of the preliminary fusion features are then analyzed. The two parts are spliced ​​and fused again to obtain the secondary fusion features. The process is represented as: ;

[0072] Finally, the secondary fusion features are enhanced to highlight defective regions and suppress background interference, thereby obtaining multi-scale fusion features. This process is represented as follows: ;

[0073] in, This indicates a feature enhancement operation.

[0074] S5: Output the surface defect segmentation result of cold-rolled strip based on the fusion features.

[0075] The network model is pre-trained before executing steps S1-S5. The training process is as follows: A combined loss function is used to train the network model composed of the dual-stream coding network, the dual-branch decoding network, and the multi-scale feature parsing module, resulting in a trained RGB-D multimodal segmentation network model for surface defects in cold-rolled strip. The combined loss function includes binary cross-entropy loss and cross-union ratio loss. During training, progressive output supervision is implemented on multiple outputs of the dual-branch decoding network, and the output of the first... Dual supervision is implemented for hierarchical output features and first-level output features, and the overall loss function... Represented as: ;

[0076] In the formula, This represents the binary cross-entropy loss function. This represents the intersection-union ratio loss function. This represents the labeled ground truth image of the corresponding defect. This represents the predicted output graph for the corresponding level. This indicates stepwise output supervision, used to perform stepwise supervision on multiple outputs of a dual-branch decoding network; This indicates deep and shallow dual supervision, used for the first layer... Dual supervision is implemented for hierarchical output features and first-level output features.

[0077] A multi-modal RGB-D segmentation system for surface defects of cold-rolled steel strip, the system being used to implement the multi-modal RGB-D segmentation method for surface defects of cold-rolled steel strip, the system comprising the following modules:

[0078] The image acquisition module is used to acquire an RGB image of the surface of the cold-rolled strip to be detected and a depth image corresponding to the RGB image space;

[0079] The dual-stream coding module is used to extract features layer by layer from RGB images and depth images respectively, and output RGB multi-level features and depth multi-level features;

[0080] The dual-branch decoding module is used to perform cross-modal layer-by-layer focusing fusion and step-by-step multi-scale analytical fusion on the RGB multi-level features and deep multi-level features;

[0081] The multi-scale feature parsing module is used to analyze the first-level detail features and the second-level feature parsing module. Hierarchical semantic features are collaboratively enhanced to output multi-scale fused features;

[0082] The segmentation output module is used to output the segmentation results of surface defects in cold-rolled strip based on multi-scale fusion features.

[0083] The present invention also provides an electronic device, comprising:

[0084] Memory, used to store computer programs;

[0085] A processor, connected to the memory, is used to call and execute the computer program to implement the RGB-D multimodal segmentation method for surface defects of cold-rolled strip.

[0086] Compared with the prior art, the present invention has the following advantages:

[0087] This invention uses RGB and depth images of the cold-rolled strip surface as dual-modal inputs, employs a dual-stream coding network to extract multi-level features under different modalities, and achieves cross-modal layer-by-layer focusing fusion and level-by-level parsing fusion through a dual-branch decoding network. Furthermore, it combines the MSFA multi-scale feature parsing module to collaboratively enhance high-level semantic features and low-level detail features, achieving pixel-level segmentation of surface defects in cold-rolled strip. Compared to defect detection methods based on single-modal information, this invention can more fully integrate the texture, grayscale, edge, appearance, geometric undulations, spatial depth, and defect morphology information of the strip surface, improving the feature representation ability for low-contrast, complex-boundary, and diverse-shaped defects. Simultaneously, through level-by-level output supervision, deep and shallow layer dual supervision, and combined loss functions, the segmentation network is jointly trained, further improving the accuracy, stability, and robustness of the segmentation results, making it more suitable for cold-rolled strip surface defect detection scenarios. Attached Figure Description

[0088] Figure 1 This is a flowchart of the RGB-D multimodal segmentation method for surface defects of cold-rolled strip provided in this embodiment of the invention;

[0089] Figure 2This is a schematic diagram of a multimodal detection platform for the surface of cold-rolled strip provided in an embodiment of the present invention; wherein, 1-main operating table; 2-roller conveyor system; 3-multimodal detection system; 31-line laser sensor; 32-industrial camera; 4-cold-rolled strip to be detected;

[0090] Figure 3 This is a diagram of the RGB-D multimodal segmentation network model of surface defects in cold-rolled strip provided in this embodiment of the invention.

[0091] Figure 4 This is a schematic diagram of the first decoding branch process provided in an embodiment of the present invention;

[0092] Figure 5 This is a schematic diagram of the collaborative process between the second decoding branch and the multi-scale feature parsing module provided in an embodiment of the present invention;

[0093] Figure 6 This is an architecture diagram of the Multi-Scale Feature Analysis (MSFA) module provided in an embodiment of the present invention;

[0094] Figure 7 This is a schematic diagram of the multi-scale feature parsing module (MSFA) provided in an embodiment of the present invention;

[0095] Figure 8 This is a schematic diagram of the RGB-D multimodal segmentation system for surface defects of cold-rolled strip provided in an embodiment of the present invention;

[0096] Figure 9 This is a visual comparison of the segmentation network provided in this embodiment of the invention with five state-of-the-art methods in several challenging and flawed scenarios on the TYUT SPSDI-DMT dataset. Detailed Implementation

[0097] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0098] Figure 1 This is a flowchart of the RGB-D multimodal segmentation method for surface defects of cold-rolled strip provided in this embodiment of the invention. Figure 2 This is a schematic diagram of a multimodal detection platform for the surface of cold-rolled strip provided in an embodiment of the present invention. The main control console 1 is used to control the start of the roller conveyor system 2 so that the cold-rolled strip 4 to be detected is conveyed to the detection field of view of the multimodal detection system 3 via the roller conveyor system 2. When the cold-rolled strip 4 enters the detection field of view, the main control console 1 sends a control signal to control the industrial camera 32 and the laser sensor 31 to synchronously collect data and obtain RGB images and depth images of the surface of the cold-rolled strip, respectively.

[0099] The multimodal inspection system 3 includes an industrial camera 32 and a line laser sensor 31. The industrial camera 32 is used to acquire RGB images of the strip surface in real time under preset focal length and exposure parameters to ensure the stability of optical imaging quality. The line laser sensor 31 is used to scan along the transverse direction of the strip and acquire three-dimensional point cloud information of the strip surface through line laser projection and structured light measurement. The industrial camera 32 and the line laser sensor 31 are synchronously triggered by the main control console 1 to ensure that the acquired RGB images and three-dimensional point cloud information are consistent in time and space.

[0100] To comprehensively evaluate the generalization ability and robustness of the method described in this invention, this embodiment conducts validation based on the TYUT SPSDI-DMT dataset. In this dataset, each defect sample includes a 2D RGB image, a 2D RGB defect-annotated ground truth image, and a depth map generated from 3D point cloud data, forming a total of 8295 pairs of multimodal image samples. These multimodal image samples are divided into a training set and a test set at an 8:2 ratio, with 6636 pairs in the training set and 1659 pairs in the test set, ensuring a relatively balanced distribution of various defect samples between the two sets. Through the above data partitioning and corresponding data augmentation processing, a data foundation can be provided for verifying the robustness and segmentation performance of the segmentation network described in this invention under complex vibration environments.

[0101] Furthermore, to enhance the generalization ability of the network model and its adaptability to complex production environments, this embodiment performs the following online data augmentation processing on the training set samples during the training phase:

[0102] Randomly flip the image horizontally with a probability of 0.5, and perform the same flipping operation on both the RGB image and the depth image to maintain spatial correspondence;

[0103] Random rotation, with a rotation angle range of ±10°, is used to simulate the angular deviation that may occur during the conveyor transport of cold-rolled strip on the roller conveyor.

[0104] Random brightness and contrast adjustment: The brightness and contrast of the RGB image are randomly adjusted within a range of 0.9 to 1.1 times that of the original image to simulate imaging changes under different lighting conditions;

[0105] Multi-scale random cropping involves randomly scaling the input image to 0.75 to 1.25 times its original size, then cropping or padding to a fixed resolution of 256×256 pixels.

[0106] A method for RGB-D multimodal segmentation of surface defects in cold-rolled steel strip, the method comprising the following steps:

[0107] S1: Construct an RGB-D multimodal segmentation network model for surface defects of cold-rolled strip, the network model including a dual-stream coding network, a dual-branch decoding network and a multi-scale feature parsing module; use an industrial camera and a laser sensor to acquire RGB images of the surface of the cold-rolled strip to be inspected and depth images corresponding to the RGB image space;

[0108] The RGB image and depth image in step S1 correspond to the same surface area of ​​the cold-rolled strip.

[0109] The RGB image is used to characterize the texture, grayscale, edge and defect appearance information of the surface of cold-rolled strip;

[0110] The depth image is used to characterize the geometric undulations, spatial depth, and defect morphology information of the surface of cold-rolled strip.

[0111] The RGB image and the depth image correspond one-to-one in spatial location and are used as dual-modal inputs for surface defect segmentation of cold-rolled strip.

[0112] S2: Input the RGB image and depth image into the dual-stream coding network in the network model respectively to extract RGB multi-level features and depth multi-level features;

[0113] The dual-stream coding network in step S2 includes an RGB coding branch and a depth coding branch; wherein:

[0114] The RGB encoding branch is used to extract features from the RGB image layer by layer and output multi-level RGB features.

[0115] The depth coding branch is used to extract features layer by layer from the depth image and output multi-level depth features;

[0116] The RGB encoding branch and the depth encoding branch are set independently and output separately. Level features, the The level features correspond to different spatial resolutions and different semantic levels according to the network hierarchy from shallow to deep, and are used for cross-modal layer-by-layer focused fusion and multi-scale parsing fusion of subsequent dual-branch decoding networks.

[0117] The dual-stream coding network uses VGG16 pre-trained on ImageNet as the backbone network, which is used to extract features from RGB images and depth images respectively.

[0118] The RGB encoding branch and the depth encoding branch each receive the input image and pass it sequentially through the VGG16 backbone network. Each stage outputs... The features are denoted as RGB multi-level features and deep multi-level features, respectively; where the number of channels in each feature map is denoted as... , The total number of layers is given, with the number of channels in each layer increasing sequentially. The initial weights of the dual-stream coding network use ImageNet pre-trained parameters to improve the convergence speed of network training and the quality of feature extraction. The multi-level features output by the two coding branches correspond one-to-one at each layer, which are then used by the subsequent dual-branch decoding network for cross-modal fusion.

[0119] Furthermore, for the depth coding branch, since the depth image is a single-channel image, while the input layer of the VGG16 backbone network expects a three-channel input, this embodiment copies the single-channel depth image along the channel dimension to a three-channel image before inputting it into the depth coding branch. Thus, the depth coding branch can directly reuse ImageNet pre-trained weights without modifying the first convolutional layer structure, while ensuring that the RGB coding branch and the depth coding branch have consistent representation capabilities in the initial stage of feature extraction.

[0120] S3: The RGB multi-level features and deep multi-level features are input into the dual-branch decoding network in the network model for fusion processing; wherein, the first decoding branch performs cross-modal layer-by-layer focusing fusion to obtain cross-modal focusing fusion features; the second decoding branch performs deep semantic extraction and fusion to obtain deep fusion features;

[0121] The dual-branch decoding network in step S3 includes a first decoding branch and a second decoding branch, wherein:

[0122] The first decoding branch is used to perform cross-modal layer-by-layer focusing fusion from deep to shallow on the RGB multi-level features, deep multi-level features, and deep fusion features generated by the second decoding branch to obtain cross-modal focused fusion features.

[0123] The second decoding branch is used to decode the code from the first... The deep semantic features extracted at the encoding level are extracted and fused to obtain deep fused features, which are then used by the subsequent multi-scale feature parsing module for step-by-step parsing and fusion.

[0124] The first and second decoding branches work together to generate fused features for subsequent multi-scale feature parsing modules to process.

[0125] The first decoding branch of the dual-branch decoding network in step S3 includes:

[0126] Feature preprocessing unit: Performs feature preprocessing on RGB multi-level features and depth multi-level features respectively to obtain preprocessed RGB features and preprocessed depth features;

[0127] No. Hierarchical fusion unit: for the first RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are concatenated and fused to obtain the first... Hierarchical cross-modal focusing fusion features; among which... This represents the total number of levels.

[0128] Layer-by-layer focusing fusion unit: For the first The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together to obtain the first... Hierarchical cross-modal focused fusion features, among which To achieve a gradual, focused integration from deep to shallow layers;

[0129] The specific operation of the first decoding branch is as follows:

[0130] Input RGB multi-level features, deep multi-level features, and deep fusion features, wherein the deep fusion features are generated and provided by the second decoding branch;

[0131] Feature preprocessing is performed on RGB multi-level features and deep multi-level features respectively;

[0132] For the RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are spliced ​​and fused to generate the first... Hierarchical cross-modal focused fusion features;

[0133] Feature upsampling fusion of remaining layers: For the first layer The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together. ;

[0134] Generate cross-modal focused fusion features at the current level;

[0135] The cross-modal features are transmitted and fused layer by layer from deep to shallow.

[0136] Output cross-modal focusing and fusion features of each layer to enhance the correlation expression between different modal features and the focusing ability on surface defect areas of cold-rolled strip;

[0137] The first decoding branch is used to enhance the RGB multi-level features and deep multi-level features output by the dual-stream coding network and achieve cross-modal fusion representation. First, the RGB multi-level features and deep multi-level features output by each level of the dual-stream coding network are input into the CBR module for feature preprocessing to obtain preprocessed RGB features and preprocessed deep features. This process is represented as follows: ;

[0138] In the formula, Indicates the first RGB features after hierarchical preprocessing Indicates the first Deep features after hierarchical preprocessing; the CBR module consists of convolutional layers, batch normalization layers, and activation function layers;

[0139] Subsequently, the first decoding branch performs layer-by-layer cross-modal fusion in a deep-to-shallow manner; among which, the first The hierarchical features are concatenated and fused with the deep fusion features output from the second decoding branch to obtain the first... Hierarchical cross-modal focusing fusion features; for other levels Then the current number will be... RGB multi-level features after hierarchical preprocessing, deep multi-level features after preprocessing, and the first The hierarchical cross-modal focusing fusion features are concatenated and fused with the upsampled results through the feature processing module to obtain the first... Hierarchical cross-modal focusing and fusion of features, the process of which is represented as follows: ; ; ;

[0140] in, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first feature after feature preprocessing. Hierarchical RGB features Indicates the first feature after feature preprocessing. Hierarchical depth features Indicates deep fusion characteristics, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first The result of hierarchical cross-modal focusing fusion features after feature processing and upsampling. This indicates a splicing operation. Indicates an upsampling operation; Indicates the first feature after feature preprocessing. Level RGB features; Indicates the first feature after feature preprocessing. Level-deep features.

[0141] The second decoding branch of the dual-branch decoding network in step S3 includes:

[0142] Deep semantic extraction unit: for the first Hierarchical RGB features and the first The hierarchical depth features are convolved and activated respectively to obtain the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features;

[0143] Deep semantic fusion unit: for the first Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused to obtain deep fused features;

[0144] The specific operation of the second decoding branch is as follows:

[0145] Enter the first Hierarchical RGB features, first Hierarchical depth features and cross-modal focusing fusion features at each level output by the first decoding branch; for the first Hierarchical RGB features and the first The hierarchical depth features are subjected to convolution and activation processing respectively; for the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused; deep fused features are output;

[0146] The second decoding branch is used to perform step-by-step parsing and fusion of defect features and output the corresponding segmentation map; firstly, for the first... Hierarchical RGB features with depth features Each feature is processed separately to obtain the corresponding deep semantic features; then, the obtained deep semantic features are concatenated and fused to obtain the deep fused feature. The process is represented as: ;

[0147] in, express Convolution operation, Indicates activation operation. Indicates a splicing operation; Indicates the feature mapping process of the first Hierarchical deep RGB features; Indicates the feature mapping process of the first Hierarchical depth features;

[0148] Subsequently, the deep fusion features and the cross-modal focused fusion features output from the first decoding branch are input step by step into the MSFA multi-scale feature parsing module for layer-by-layer parsing and fusion of multi-scale information to obtain multi-scale semantic fusion features; this process is represented as: ; ;

[0149] in, This represents deep semantic fusion features. Indicates the first Multi-scale fusion features of layers This represents the multi-scale feature parsing module.

[0150] After obtaining multi-scale semantic fusion features at various levels Then, it is upsampled to the input resolution, and the corresponding segmentation output image is generated through independent convolution operations. And combine real labels for multi-scale supervision.

[0151] S4: Input the cross-modal focusing fusion features output from the first decoding branch and the deep fusion features output from the second decoding branch into the multi-scale feature parsing module, and perform step-by-step multi-scale parsing and fusion, combining the detail features from the first encoding level with those from the second encoding level. The semantic features at the encoding level are collaboratively enhanced to obtain the fused features required for defect segmentation after fusion.

[0152] The multi-scale feature parsing module in step S4 includes:

[0153] Channel attention enhancement unit: Performs channel attention enhancement on the input high-level semantic features and low-level detail features respectively to obtain enhanced high-level semantic features and enhanced low-level detail features; the high-level semantic features come from the first channel attention enhancement unit of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network;

[0154] Multi-scale semantic extraction unit: Parallel feature extraction is performed on the enhanced high-level semantic features using different convolutional kernel scales to obtain high-level semantic features under different receptive fields;

[0155] Preliminary Feature Fusion Unit: The enhanced low-level detail features, enhanced high-level semantic features, and high-level semantic features under different receptive fields are preliminarily spliced ​​and fused to obtain preliminary fused features;

[0156] Feature weighting enhancement unit: Generates weight information based on the preliminary fused features, and performs weighting processing on the high-level semantic features and low-level detail features before enhancement to obtain weighted high-level semantic features and weighted low-level detail features.

[0157] Fusion Enhancement Output Unit: The weighted high-level semantic features, weighted low-level detail features, and the preliminary fusion features are concatenated and fused again, and multi-scale fusion features are obtained through feature enhancement operations;

[0158] The specific operation of the multi-scale feature parsing module is as follows:

[0159] Input high-level semantic features and low-level detail features; wherein, the high-level semantic features come from the first level of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network;

[0160] Channel attention enhancement is applied to the input high-level semantic features and low-level detail features respectively to obtain enhanced high-level semantic features and low-level detail features;

[0161] Parallel feature extraction was performed on the enhanced high-level semantic features using convolutional operations of different scales to obtain multi-scale high-level semantic features under different receptive fields.

[0162] The enhanced low-level detail features, enhanced high-level semantic features, and multi-scale high-level semantic features are concatenated and fused, and feature processing is performed to obtain the initial fused features.

[0163] Weight information is generated based on the initial fusion features, and the high-level semantic features and low-level detail features before enhancement are weighted to enhance the feature response related to defects.

[0164] The weighted high-level semantic features, weighted low-level detail features, and initial fusion features are concatenated and fused again, and feature processing is performed to obtain secondary fusion features.

[0165] The secondary fusion features are enhanced to highlight the features of the defect area and suppress background interference;

[0166] The enhanced multi-scale fusion features are output for subsequent segmentation result generation;

[0167] First, high-level semantic features and low-level detail features A channel attention mechanism is introduced to enhance important channel features, resulting in enhanced high-level semantic features. and enhanced low-level detail features The process is represented as: ;

[0168] In the formula, Indicates channel attention operation;

[0169] Parallel feature extraction is performed on the enhanced high-level semantic features using convolutional kernels of different scales to obtain high-level semantic features under different receptive fields, and these features are then combined with the enhanced low-level detail features. Preliminary fusion was performed to obtain preliminary fusion characteristics. The process is represented as: ;

[0170] in, This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation; This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation;

[0171] Subsequently, to enhance the expressive power of the initial fusion features, weight information is generated based on them. The high-level semantic features and low-level detail features before enhancement are weighted and enhanced to obtain the relabeled high-level semantic features. Low-level details The process is represented as: ;

[0172] in, This represents the sigmoid activation function. express Average pooling operation;

[0173] Furthermore, the initial fusion characteristics will be... After a The convolution operation yields a preliminary fused feature mapping feature. Then, the relabeled high-level semantic features, low-level detail features, preliminary fusion features, and mapping features of the preliminary fusion features are then analyzed. The two parts are spliced ​​and fused again to obtain the secondary fusion features. The process is represented as: ;

[0174] Finally, the secondary fusion features are enhanced to highlight defective regions and suppress background interference, thereby obtaining multi-scale fusion features. This process is represented as follows: ;

[0175] in, This indicates a feature enhancement operation.

[0176] S5: Output the surface defect segmentation result of cold-rolled strip based on the fusion features.

[0177] The network model is pre-trained before executing steps S1-S5. The training process is as follows: A combined loss function is used to train the network model composed of the dual-stream coding network, the dual-branch decoding network, and the multi-scale feature parsing module, resulting in a trained RGB-D multimodal segmentation network model for surface defects in cold-rolled strip. The combined loss function includes binary cross-entropy loss and cross-union ratio loss. During training, progressive output supervision is implemented on multiple outputs of the dual-branch decoding network, and the output of the first... Dual supervision is implemented for hierarchical output features and first-level output features, and the overall loss function... Represented as: ;

[0178] In the formula, This represents the binary cross-entropy loss function. This represents the intersection-union ratio loss function. This represents the labeled ground truth image of the corresponding defect. This represents the predicted output graph for the corresponding level. This indicates stepwise output supervision, used to perform stepwise supervision on multiple outputs of a dual-branch decoding network; This indicates deep and shallow dual supervision, used for the first layer... Dual supervision is implemented for hierarchical output features and first-level output features.

[0179] In this embodiment, the specific configuration of the training process is as follows:

[0180] The Adam optimizer was used to optimize the parameters of the network model, with an initial learning rate set to 1×10⁻. 4 The weight decay factor is set to 5×10⁻ 4 ;

[0181] Set the batch size to 8 and the total number of training rounds to 50.

[0182] All input RGB and depth images were resized to 256×256 pixels during both the training and testing phases.

[0183] The learning rate employs a multinomial decay strategy;

[0184] Training is performed on GPUs and implemented using a deep learning framework.

[0185] A multi-modal RGB-D segmentation system for surface defects of cold-rolled steel strip, the system being used to implement the multi-modal RGB-D segmentation method for surface defects of cold-rolled steel strip, the system comprising the following modules:

[0186] The image acquisition module is used to acquire an RGB image of the surface of the cold-rolled strip to be detected and a depth image corresponding to the RGB image space;

[0187] The dual-stream coding module is used to extract features layer by layer from RGB images and depth images respectively, and output RGB multi-level features and depth multi-level features;

[0188] The dual-branch decoding module is used to perform cross-modal layer-by-layer focusing fusion and step-by-step multi-scale analytical fusion on the RGB multi-level features and deep multi-level features;

[0189] The multi-scale feature parsing module is used to analyze the first-level detail features and the second-level feature parsing module. Hierarchical semantic features are collaboratively enhanced to output multi-scale fused features;

[0190] The segmentation output module is used to output the segmentation results of surface defects in cold-rolled strip based on multi-scale fusion features.

[0191] Table 1. Comparison of quantitative performance with five state-of-the-art methods on the TYUT SPSDI-DMT dataset.

[0192]

[0193] As shown in Table 1, Table 1 is a quantitative performance comparison table of the segmentation network provided in this embodiment of the invention with five state-of-the-art methods on the TYUT SPSDI-DMT dataset. Specifically, in this embodiment, in order to quantitatively compare the performance of different multimodal segmentation models, this embodiment selects nine evaluation metrics for evaluation, including S-measure, Mean Absolute Error (MAE), E-measure, F-measure, and Weighted F-measure.

[0194] Among them, S-measure ( , The MAE (Multi-Equation) is used to evaluate the consistency of segmentation results in terms of structural information and region content; a higher value indicates better segmentation. The MAE measures the predicted output map. Compared with the actual labeled image The mean absolute error between the two values ​​is used; the smaller the value, the closer the predicted result is to the true label. E-measure ( The adaptive E-measure is used to comprehensively evaluate the regional accuracy and boundary consistency of the segmentation results. ), meanE-measure ( ) and maximum E-measure ( As evaluation indicators, the higher the value of the above indicators, the better. F-measure ( The F-measure is used to comprehensively reflect precision and recall to evaluate the quality of segmentation results. Therefore, this embodiment uses the adaptive F-measure. ), mean F-measure ( ) and maximum F-measure ( As evaluation indicators, the higher the value of the above indicators, the better. Weighted F-measure ( , Introducing a weighting mechanism on top of the F-measure can more effectively reflect the model's segmentation performance of key regions; the larger the value, the better the model's performance.

[0195] This embodiment compares the segmentation network of the present invention with five existing multimodal models, namely SAINet, MobileSal, SEFFSal, CIRNet_R50, and CIRNet_V16. To ensure the fairness of the comparison results, the output results of each comparison method are provided by the corresponding authors or generated based on publicly available source code with default parameter settings; each model is tested on the TYUT SPSDI-DMT dataset, and all output results are adjusted to the same size as the original input image.

[0196] like Figure 9 As shown, Figure 9 The image shows a visual comparison between the segmentation network provided in this embodiment of the invention and five state-of-the-art methods in several challenging defect scenarios of the TYUT SPSDI-DMT dataset. Specifically, in this embodiment, it includes:

[0197] Five comparison methods are compared with the saliency segmentation output of the segmentation network of this invention on the TYUT SPSDI-DMT dataset. The method of this invention (fourth row) corresponds to the best defect detection result. Specifically, the defects in the first three columns of images have relatively fine structural features; the defects in the fourth and fifth columns of images have low contrast with the background; the sixth and seventh columns of images have strong complex background interference; and the last three columns of images contain multiple defect targets. Compared with other comparison methods, the segmentation network of this invention can obtain more accurate defect segmentation results in the above-mentioned complex scenarios and can generate output images with clearer boundaries and more complete defect regions.

[0198] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A method for RGB-D multimodal segmentation of surface defects in cold-rolled steel strip, characterized in that, The method includes the following steps: S1: Construct an RGB-D multimodal segmentation network model for surface defects of cold-rolled strip, the network model including a dual-stream coding network, a dual-branch decoding network and a multi-scale feature parsing module; use an industrial camera and a laser sensor to acquire RGB images of the surface of the cold-rolled strip to be inspected and depth images corresponding to the RGB image space; S2: Input the RGB image and depth image into the dual-stream coding network in the network model respectively to extract RGB multi-level features and depth multi-level features; S3: Input the RGB multi-level features and deep multi-level features into the dual-branch decoding network in the network model to fuse the RGB multi-level features and deep multi-level features. The first decoding branch performs cross-modal layer-by-layer focusing fusion to obtain cross-modal focusing fused features; the second decoding branch performs multi-scale parsing fusion to obtain multi-scale fused features. S4: Input the multi-scale fused features from the second decoding branch into the multi-scale feature parsing module, and compare the detail features from the first encoding level with those from the second encoding level. The semantic features at the encoding level are collaboratively enhanced to obtain the fused features required for defect segmentation after fusion. S5: Output the surface defect segmentation result of cold-rolled strip based on the fusion features.

2. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 1, characterized in that, The RGB image and depth image in step S1 correspond to the same surface area of ​​the cold-rolled strip. The RGB image is used to characterize the texture, grayscale, edge and defect appearance information of the surface of cold-rolled strip; The depth image is used to characterize the geometric undulations, spatial depth, and defect morphology information of the surface of cold-rolled strip. The RGB image and the depth image correspond one-to-one in spatial location and are used as dual-modal inputs for surface defect segmentation of cold-rolled strip.

3. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 2, characterized in that, The dual-stream coding network in step S2 includes an RGB coding branch and a depth coding branch; wherein: The RGB encoding branch is used to extract features from the RGB image layer by layer and output multi-level RGB features. The depth coding branch is used to extract features layer by layer from the depth image and output multi-level depth features; The RGB encoding branch and the depth encoding branch are set independently and output separately. Level features, the The level features correspond to different spatial resolutions and different semantic levels according to the network hierarchy from shallow to deep, and are used for cross-modal layer-by-layer focused fusion and multi-scale parsing fusion of subsequent dual-branch decoding networks. The dual-stream coding network uses VGG16 pre-trained on ImageNet as the backbone network, which is used to extract features from RGB images and depth images respectively. The RGB encoding branch and the depth encoding branch each receive the input image and pass it sequentially through the VGG16 backbone network. Each stage outputs... The features are denoted as RGB multi-level features and deep multi-level features, respectively; where the number of channels in each feature map is denoted as... , The total number of layers is represented by the number of channels in each layer, which increases sequentially. The initial weights of the dual-stream coding network use ImageNet pre-trained parameters to improve the convergence speed of network training and the quality of feature extraction.

4. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 3, characterized in that, The dual-branch decoding network in step S3 includes a first decoding branch and a second decoding branch, wherein: The first decoding branch is used to perform cross-modal layer-by-layer focusing fusion from deep to shallow on the RGB multi-level features, deep multi-level features, and deep fusion features generated by the second decoding branch to obtain cross-modal focused fusion features. The second decoding branch is used to decode the code from the first... The deep semantic features extracted at the encoding level and the cross-modal focused fusion features are subjected to hierarchical multi-scale parsing and fusion to obtain multi-scale fusion features; The first and second decoding branches work together to generate fused features for subsequent multi-scale feature parsing modules to process.

5. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 4, characterized in that, The first decoding branch of the dual-branch decoding network in step S3 includes: Feature preprocessing unit: Performs feature preprocessing on RGB multi-level features and depth multi-level features respectively to obtain preprocessed RGB features and preprocessed depth features; No. Hierarchical fusion unit: for the first RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are concatenated and fused to obtain the first... Hierarchical cross-modal focused fusion features; among which... This represents the total number of levels. Layer-by-layer focusing fusion unit: For the first The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together to obtain the first... Hierarchical cross-modal focused fusion features, among which This is to achieve a gradual, focused integration from deep to shallow layers; The specific operation of the first decoding branch is as follows: Input RGB multi-level features, deep multi-level features, and deep fusion features, wherein the deep fusion features are generated and provided by the second decoding branch; Feature preprocessing is performed on RGB multi-level features and deep multi-level features respectively; For the first RGB features after hierarchical preprocessing, the first The deep features after hierarchical preprocessing and the deep fusion features are spliced ​​and fused to generate the first... Hierarchical cross-modal focused fusion features; Feature upsampling fusion of remaining layers: For the first layer The hierarchy includes its preprocessed RGB features, preprocessed depth features, and the first layer. The hierarchical cross-modal focusing fusion features are upsampled and then stitched together. ; Generate cross-modal focused fusion features at the current level; The cross-modal features are transmitted and fused layer by layer from deep to shallow. Output cross-modal focusing and fusion features of each layer to enhance the correlation expression between different modal features and the focusing ability on surface defect areas of cold-rolled strip; The first decoding branch is used to enhance the RGB multi-level features and deep multi-level features output by the dual-stream coding network and achieve cross-modal fusion representation. First, the RGB multi-level features output by each level of the dual-stream coding network are... and deep multi-level features The features are input into the CBR module for feature preprocessing, resulting in preprocessed RGB features and preprocessed depth features. This process is represented as follows: ; In the formula, Indicates the first RGB features after hierarchical preprocessing Indicates the first Depth features after hierarchical preprocessing; The CBR module consists of convolutional layers, batch normalization layers, and activation function layers. Subsequently, the first decoding branch performs layer-by-layer cross-modal fusion in a deep-to-shallow manner; among which, the first The hierarchical features are concatenated and fused with the deep modal fusion features output from the second decoding branch to obtain the first... Hierarchical cross-modal focusing fusion features; for other levels Then the current number will be... RGB multi-level features after hierarchical preprocessing, deep multi-level features after preprocessing, and the first The hierarchical cross-modal focusing fusion features are concatenated and fused with the upsampled results through the feature processing module to obtain the first... Hierarchical cross-modal focusing and fusion of features, the process of which is represented as follows: ; ； ； in, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first feature after feature preprocessing. Hierarchical RGB features Indicates the first feature after feature preprocessing. Hierarchical depth features Indicates deep fusion characteristics, Indicates the first Hierarchical cross-modal focused fusion features Indicates the first The result of hierarchical cross-modal focusing fusion features after feature processing and upsampling. This indicates a splicing operation. Indicates an upsampling operation; Indicates the first feature after feature preprocessing. Level RGB features; Indicates the first feature after feature preprocessing. Level-deep features.

6. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 5, characterized in that, The second decoding branch of the dual-branch decoding network in step S3 includes: Deep semantic extraction unit: for the first Hierarchical RGB features and the first The hierarchical depth features are convolved and activated respectively to obtain the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features; Deep semantic fusion unit: for the first Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused to obtain deep semantic fusion features; Step-by-step parsing and fusion unit: The deep semantic fusion features and the cross-modal focusing fusion features output by the first decoding branch are input step-by-step into the multi-scale feature parsing module to obtain multi-scale fusion features; The specific operation of the second decoding branch is as follows: Enter the first Hierarchical RGB features, first Hierarchical depth features and cross-modal focusing fusion features at each level output by the first decoding branch; for the first Hierarchical RGB features and the first The hierarchical depth features are subjected to convolution and activation processing respectively; for the first... Hierarchical RGB semantic features and the first Hierarchical deep semantic features are concatenated and fused; deep fused features are generated, and the deep semantic fused features and the cross-modal focusing fused features output from the first decoding branch are input into the multi-scale feature parsing module step by step; the features are parsed and fused step by step to generate multi-scale fused features; The second decoding branch is used to perform step-by-step parsing and fusion of defect features and output the corresponding segmentation map; firstly, for the first... Hierarchical RGB features with depth features Each feature is processed separately to obtain the corresponding deep semantic features; then, the obtained deep semantic features are concatenated and fused to obtain the deep fused features. The process is represented as: ; in, express Convolution operation, Indicates activation operation. Indicates a splicing operation; Indicates the feature mapping process of the first Hierarchical deep RGB features; Indicates the feature mapping process of the first Hierarchical depth features; Subsequently, the deep fusion features and the cross-modal focused fusion features output from the first decoding branch are input step by step into the MSFA multi-scale feature parsing module for layer-by-layer parsing and fusion of multi-scale information to obtain multi-scale semantic fusion features; this process is represented as: ; ； in, This represents deep semantic fusion features. Indicates the first Multi-scale fusion features of layers This represents the multi-scale feature parsing module; After obtaining multi-scale semantic fusion features at various levels Then, it is upsampled to the input resolution, and the corresponding segmentation output image is generated through independent convolution operations. And combine real labels for multi-scale supervision.

7. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 6, characterized in that, The multi-scale feature parsing module in step S4 includes: Channel attention enhancement unit: Performs channel attention enhancement on the input high-level semantic features and low-level detail features respectively to obtain enhanced high-level semantic features and enhanced low-level detail features; the high-level semantic features come from the first channel attention enhancement unit of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network; Multi-scale semantic extraction unit: Parallel feature extraction is performed on the enhanced high-level semantic features using different convolutional kernel scales to obtain high-level semantic features under different receptive fields; Preliminary Feature Fusion Unit: The enhanced low-level detail features, enhanced high-level semantic features, and high-level semantic features under different receptive fields are preliminarily spliced ​​and fused to obtain preliminary fused features; Feature weighting enhancement unit: Generates weight information based on the preliminary fused features, and performs weighting processing on the high-level semantic features and low-level detail features before enhancement to obtain weighted high-level semantic features and weighted low-level detail features. Fusion Enhancement Output Unit: The weighted high-level semantic features, weighted low-level detail features, and the preliminary fusion features are concatenated and fused again, and multi-scale fusion features are obtained through feature enhancement operations; The specific operation of the multi-scale feature parsing module is as follows: Input high-level semantic features and low-level detail features; wherein, the high-level semantic features come from the first level of the two-stream coding network. The coding level, wherein the low-level detail features are derived from the first coding level of the two-stream coding network; Channel attention enhancement is applied to the high-level semantic features and low-level detail features of the input respectively to obtain the enhanced high-level semantic features and low-level detail features; Parallel feature extraction was performed on the enhanced high-level semantic features using convolutional operations of different scales to obtain multi-scale high-level semantic features under different receptive fields. The enhanced low-level detail features, enhanced high-level semantic features, and multi-scale high-level semantic features are concatenated and fused, and feature processing is performed to obtain the initial fused features. Weight information is generated based on the initial fusion features, and the high-level semantic features and low-level detail features before enhancement are weighted to enhance the feature response related to defects. The weighted high-level semantic features, weighted low-level detail features, and initial fusion features are concatenated and fused again, and feature processing is performed to obtain secondary fusion features. The secondary fusion features are enhanced to highlight the features of the defect area and suppress background interference; The enhanced multi-scale fusion features are output for subsequent segmentation result generation; First, high-level semantic features and low-level detail features A channel attention mechanism is introduced to enhance important channel features, resulting in enhanced high-level semantic features. and enhanced low-level detail features The process is represented as: ; In the formula, Indicates channel attention operation; Parallel feature extraction is performed on the enhanced high-level semantic features using convolutional kernels of different scales to obtain high-level semantic features under different receptive fields, and these features are then combined with the enhanced low-level detail features. Preliminary fusion was performed to obtain preliminary fusion characteristics. The process is represented as: ; in, This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation; This indicates the enhanced high-level semantic features. go through High-level semantic features after convolution operation; Subsequently, to enhance the expressive power of the initial fusion features, weight information is generated based on them. The high-level semantic features and low-level detail features before enhancement are weighted and enhanced to obtain the relabeled high-level semantic features. Low-level details The process is represented as: ; in, This represents the sigmoid activation function. express Average pooling operation; Furthermore, the initial fusion characteristics will be... After a The convolution operation yields a preliminary fused feature mapping feature. Subsequently, the recalibrated high-level semantic features, low-level detail features, preliminary fusion features, and mapping features of the preliminary fusion features are then analyzed. The two parts are spliced ​​and fused again to obtain the secondary fusion features. The process is represented as: ; Finally, the secondary fusion features are enhanced to highlight defective regions and suppress background interference, thereby obtaining multi-scale fusion features. This process is represented as follows: ; in, This indicates a feature enhancement operation.

8. The RGB-D multimodal segmentation method for surface defects of cold-rolled strip according to claim 7, characterized in that, The network model is pre-trained before executing steps S1-S5. The training process is as follows: the network model consisting of the dual-stream coding network, the dual-branch decoding network, and the multi-scale feature parsing module is trained using a combined loss function to obtain the trained RGB-D multimodal segmentation network model for surface defects of cold-rolled strip. The combined loss function includes binary cross-entropy loss and cross-union ratio loss. During training, step-by-step output supervision is implemented on multiple outputs of the dual-branch decoding network, and the first output is supervised step-by-step. Dual supervision is implemented for hierarchical output features and first-level output features, and the overall loss function... Represented as: ; In the formula, This represents the binary cross-entropy loss function. This represents the intersection-union ratio loss function. This represents the labeled ground truth image of the corresponding defect. This represents the predicted output graph for the corresponding level. This indicates stepwise output supervision, used to perform stepwise supervision on multiple outputs of a dual-branch decoding network; This indicates deep and shallow dual supervision, used for the first layer... Dual supervision is implemented for hierarchical output features and first-level output features.

9. A multi-modal RGB-D segmentation system for surface defects in cold-rolled steel strip, characterized in that, The system is used to implement the method according to any one of claims 1-8, and the system includes the following modules: The image acquisition module is used to acquire an RGB image of the surface of the cold-rolled strip to be detected and a depth image corresponding to the RGB image space; The dual-stream coding module is used to extract features layer by layer from RGB images and depth images respectively, and output RGB multi-level features and depth multi-level features; The dual-branch decoding module is used to perform cross-modal layer-by-layer focusing fusion and step-by-step multi-scale analytical fusion on the RGB multi-level features and deep multi-level features; The multi-scale feature parsing module is used to analyze the first-level detail features and the second-level feature parsing module. Hierarchical semantic features are collaboratively enhanced to output multi-scale fused features; The segmentation output module is used to output the segmentation results of surface defects in cold-rolled strip based on multi-scale fusion features.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, connected to the memory, is configured to invoke and execute the computer program to implement the method according to any one of claims 1-8.

Images