Ship rust classification method based on direction perception multi-scale feature fusion

Abstract

The invention provides a ship rust classification method based on direction perception multi-scale feature fusion. The ship rust classification method comprises the following steps: S1, shooting a ship image to obtain a training data set; step S2, a classification model is constructed, the classification model comprises a feature extraction network with multi-level inverted residual blocks and a feature pyramid network for realizing multi-level feature fusion, the feature extraction network adopts a cooperation module in a deep-layer large-channel-number inverted residual block and adopts an SE attention module in a shallow-layer small-channel-number inverted residual block, and the deep-layer large-channel-number inverted residual block and the shallow-layer small-channel-number inverted residual block are combined; the cooperation module comprises a direction branch, a space attention branch and a channel attention branch which are connected, and the feature pyramid network fuses the multi-level features extracted by the feature extraction network; and S3, training the classification model by using the training data set. The method gives consideration to the classification precision and speed.

Description

Technical Field

[0001] This invention relates to the field of ship maintenance technology, and in particular to a ship rust classification method based on direction-aware multi-scale feature fusion. Background Technology

[0002] Ships are exposed to high salinity and high humidity marine environments for extended periods. Their hull surfaces are susceptible to both electrochemical and microbial corrosion, leading to varying degrees of rust. Rust weakens the structural integrity of the hull, necessitating rust detection. Traditional methods rely on visual inspection combined with ultrasonic thickness gauges, but these methods suffer from low efficiency, subjective bias in human judgment, and safety hazards.

[0003] In recent years, the application of computer vision-based automated inspection methods has gradually increased, but applying existing inspection methods to ship rust classification faces the following problems:

[0004] 1. Model deployment constraints: Existing methods use models such as ResNet34 with 2.5G FLOPs, which have a large computational load and result in inference latency exceeding 500ms, failing to meet real-time requirements;

[0005] 2. Rust features are complex: Rust has diverse morphologies and large scale variations, making it difficult for existing attention mechanisms to effectively capture multi-scale spatial features.

[0006] 3. Insufficient utilization of features: The rusted area will spread along the structural direction of welds, rivets and other structures. Its texture is highly directional and the edges are blurred. Existing methods have failed to effectively integrate the shallow detailed features and deep semantic features of the network, resulting in insufficient ability to recognize the edges of blurred rust. Summary of the Invention

[0007] The main objective of this invention is to propose a ship rust classification method based on direction-aware multi-scale feature fusion, which balances classification accuracy and speed.

[0008] This invention is achieved through the following technical solution:

[0009] A method for classifying ship rust based on direction-aware multi-scale feature fusion includes the following steps:

[0010] Step S1: Take images of the ship to obtain the training dataset;

[0011] Step S2: Construct a classification model. The classification model includes a feature extraction network with multi-level inverted residual blocks and a feature pyramid network that realizes multi-level feature fusion. The feature extraction network uses a collaborative module in deep, large-channel-number inverted residual blocks and an SE attention module in shallow, small-channel-number inverted residual blocks. The collaborative module includes directional branches, spatial attention branches, and channel attention branches. The feature pyramid network fuses the multi-level features extracted by the feature extraction network.

[0012] Step S3: Train the classification model using the training dataset.

[0013] Furthermore, in step S1, the captured ship images are adjusted to the same size, and the adjusted ship images are randomly rotated and flipped to obtain a training dataset.

[0014] Furthermore, in step S2, the feature extraction network is a MobileNetV3 network, and the inverted residual block adopts a combination of depthwise convolution, pointwise convolution, and standard convolution. The inverted residual block is configured with different numbers of input channels, extended channels, and output channels. When the number of extended channels is greater than the set M1, it is a deep inverted residual block with a large number of channels; otherwise, it is a shallow inverted residual block with a small number of channels.

[0015] Furthermore, in step S2, the directional branch extracts the directional feature map through grouped convolution. Where X is the input feature map of the collaborative module, For activation function, For batch normalization operations, For directional convolution weights, This indicates convolution calculation.

[0016] Furthermore, the spatial attention branch first uses a 1×1 convolution to convert the orientation feature map... Compressed into a single-channel feature map Then, for single-channel feature maps Parallel execution of dual-channel pooling yields max pooling results. and average pooling results Finally, calculate the spatial weights. ,in, This represents a convolution operation with a kernel size of 1×1. Indicates single-channel characteristics The midpoint (i,j) is the local neighborhood of the center. Indicates the number of pixels in the neighborhood. For the Sigmoid function, For convolution operations with a kernel size of 5×5, , For splicing operations, Refers to the single-channel feature map The set of max pooling results at each point above. Refers to the single-channel feature map The set consisting of the average pooling results of all points.

[0017] Furthermore, in step S2, the output of the collaborative module is represented as follows: , Here are the channel weights for the channel attention branch, where... This is the channel descriptor after global average pooling. H is the length of the input feature map X, and W is the width of the input feature map X. Given the pixel at position (i,j) in the c-th channel of the input feature map X, and The weights of two 1×1 convolutional layers, To truncate the ReLU activation function, This indicates element-wise multiplication.

[0018] Furthermore, in step S2, the feature pyramid network extracts the shallow feature map F of the output Y of the collaborative module. s Mid-layer feature map F m and deep feature map F d The spatial dimensions of the middle-layer feature map and the deep-layer feature map are respectively sampled from the shallow-layer feature map. Figure 1 The fusion features are then obtained by splicing the components together. ,in, For channel-dimensional splicing, This is the upsampling result of the mid-layer feature map. This is the result of upsampling the deep feature map. , Indicates an upsampling operation. express Space dimensions.

[0019] Furthermore, in step S2, the classification model further includes an average pooling layer connected to the output of the feature pyramid network and a fully connected layer connected to the average pooling layer.

[0020] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0021] The classification model constructed in this invention includes a feature extraction network with multi-level inverted residual blocks and a feature pyramid network that realizes multi-level feature fusion. The feature extraction network uses a collaborative module in deep, large-channel-number inverted residual blocks and an SE attention module in shallow, small-channel-number inverted residual blocks. The collaborative module includes a directional branch, a spatial attention branch, and a channel attention branch. The feature pyramid network fuses the multi-level features extracted by the feature extraction network. The collaborative module effectively captures rust texture features from different directions and combines spatial attention and channel attention information for collaborative enhancement, overcoming the shortcomings of existing technologies that are insensitive to linear and irregular rust features. The feature pyramid network explicitly combines shallow detail features, mid-level structural features, and deep semantic features, enabling the model to simultaneously perceive the local texture and global distribution of rust, significantly improving the classification accuracy for complex shapes and blurred edge rust. It has strong adaptability to changes in rust scale, effectively recognizing both large-area uniform rust and small-scale point rust, reducing false positives and false negatives. Through a hierarchical attention mechanism, a lightweight SE attention module is used in the shallow layer, while a more powerful collaborative module is used only in the deep, computationally intensive areas, achieving a balance between performance and efficiency. Furthermore, the feature fusion process involves only simple upsampling and stitching operations, resulting in low computational overhead. Attached Figure Description

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 This is a flowchart of the present invention.

[0024] Figure 2 This is a schematic diagram of the classification model of the present invention.

[0025] Figure 3 This is a schematic diagram of the collaborative module of the present invention.

[0026] Figure 4 This is a comparison chart of the parameters of the collaborative module of the present invention and the SE attention module of the prior art.

[0027] Figure 5 This is a comparison chart of the inference speed of the collaborative module of this invention and the SE attention module of the prior art.

[0028] Figure 6 This paper describes the performance differences between the collaborative module of this invention and the SE attention module of the prior art in ship corrosion detection tasks.

[0029] Figure 7 This is a diagram illustrating the classification results of the present invention. Detailed Implementation

[0030] The present invention will be further described below through specific embodiments.

[0031] like Figure 1 As shown, the ship rust classification method based on direction-aware multi-scale feature fusion includes the following steps:

[0032] Step S1: Take images of the ship to obtain the training dataset;

[0033] The captured ship images were resized to the same size of 224×224 pixels. The resized ship images were then randomly rotated and flipped to enhance the data. The enhanced data was then divided into training dataset, validation set, and test set.

[0034] Step S2: Construct a classification model. The classification model includes a feature extraction network with multi-level inverted residual blocks, a feature pyramid network that realizes multi-level feature fusion, an average pooling layer connected to the output of the feature pyramid network, and a fully connected layer connected to the average pooling layer. The feature extraction network uses a collaborative module in deep, large-channel-number inverted residual blocks and an SE attention module in shallow, small-channel-number inverted residual blocks. The collaborative module includes a directional branch, a spatial attention branch, and a channel attention branch. The feature pyramid network fuses the multi-level features extracted by the feature extraction network.

[0035] The directional branch is responsible for extracting the directional texture features of rust, solving the problem of irregular morphology of rust spreading along the hull structure. In image processing, directional textures are commonly found in structures such as edges and stripes, while ship rust often spreads along the direction of welds or rivets. Therefore, directional perception can effectively improve feature discrimination. The spatial attention branch generates a spatial weight map, highlighting the spatial location of the rusted area and enhancing the salience of blurred edge areas, solving challenges such as blurred edges, low contrast, and noise interference in ship rust detection. The channel attention branch generates channel weight vectors, strengthening feature channels related to rust, such as color and texture, and suppressing background interference.

[0036] The overall structure diagram of the classification model is as follows: Figure 2As shown. The feature extraction network is the MobileNetV3 network, which contains a multi-level inverse residual block structure. Each inverse residual block uses a combination of depthwise convolution, pointwise convolution, and standard convolution. Inverse residual blocks are configured with different numbers of input channels, expanded channels, and output channels. Deep inverse residual blocks with a large number of channels use a collaborative module, while shallow inverse residual blocks with a small number of channels use an SE attention module. When the number of expanded channels is greater than a set threshold M=288, it is a deep inverse residual block with a large number of channels; otherwise, it is a shallow inverse residual block with a small number of channels, thus achieving adaptive attention mechanism selection. The SE attention module used in the shallow inverse residual blocks with a small number of channels is a current technology. The MobileNetV3 network consists of multiple stacked inverse residual blocks, with convolutional layers and fully connected layers at the beginning and end. The training dataset is input into the MobileNetV3 network and first passes through an expansion layer consisting of existing 1×1 convolutions, batch normalization (BN), and a hardswish activation function to increase the number of channels to the predetermined expanded number. Subsequently, the feature map is efficiently extracted spatially using a 3×3 depthwise separable convolution (which includes depthwise convolution and pointwise convolution), and then batch normalization and activation processing are performed again.

[0037] Subsequently, the network selects either a collaborative module or an SE attention module based on the number of expanded channels. The output of the collaborative module or the SE attention module is then processed by existing 1×1 convolutions, residual connections, etc., to obtain the output feature map Y' of the MobileNetV3 network.

[0038] like Figure 3 As shown, the directional branch of the collaborative module uses grouped convolution to group the input feature map X by channel and independently extract directional feature maps (i.e., gradient features in different directions). The specific formula is expressed as follows: ,in, The input feature map X has a dimension of directional feature map. , For the number of channels, Given the length of the input feature map X, The input feature map X is the width, and D is the number of directions. The activation function is defined as follows: , Activation functions introduce nonlinear transformations to enhance feature representation capabilities. For batch normalization operations, The weights are the directional convolution weights (i.e., the convolution kernel, with a size of 3×3). This indicates convolution calculation. In this embodiment, the number of groups for grouped convolution is set to the number of directions of the rust, D=2, corresponding to the 45° and 135° directions respectively. The number of directions, D, is adjusted as a hyperparameter to balance direction sensitivity and computational efficiency.

[0039] The spatial attention branch first uses a 1×1 convolution to convert the orientation feature map... Compressed into a single-channel feature map To reduce computational dimensionality and retain core information, , This represents a convolution operation with a kernel size of 1×1;

[0040] Then, the single-channel feature map... Parallel execution of dual-channel pooling yields max pooling results. and average pooling results , Represented by single-channel feature map The midpoint (i,j) is the local neighborhood of the center. Indicates the number of pixels in the neighborhood;

[0041] Finally, calculate the spatial weights. , For convolution operations with a kernel size of 5×5, the kernel size provides a sufficient receptive field to cover irregular patches of rust while maintaining a lightweight design. The Sigmoid function is defined as follows: Ensure that the spatial weight values ​​are between [0,1]. , For splicing operations, , Refers to the single-channel feature map The set consisting of the maximum pooling values ​​of all points above. Refers to the single-channel feature map The set of average pooling values ​​at each point is composed of max pooling, which focuses on local salient features such as high gradient edges of rust cracks, while average pooling preserves the global smooth context. The fusion of the two can enhance the model's adaptability to multi-scale changes in rust morphology.

[0042] The channel attention branch first performs global average pooling on the ship image to obtain the pooling results for each channel. Then calculate the channel weights. ,in, This is the channel descriptor after global average pooling. Let (i,j) be the pixel at position (i,j) in the c-th channel of the input feature map X. and The weights of the two 1×1 convolutional layers are used for dimensionality reduction and dimensionality enhancement of the input to reduce computational cost. To truncate the ReLU activation function, it is defined as follows: The above weights obtain their optimal values ​​during training.

[0043] Global average pooling is used to compress the spatial information of each channel of the ship image into a global statistic. An activation function is then employed. This avoids the problem of numerical instability caused by excessively large activation values, and the limited output range results in less accuracy loss during quantization, making it more suitable for resource-constrained embedded platforms such as ship rust removal robots.

[0044] Based on the above branches, the output of the collaboration module is expressed as follows: , This indicates element-wise multiplication.

[0045] The feature pyramid network first extracts shallow feature maps of the output Y' of the feature extraction network. Mid-layer feature map and deep feature maps Then, the spatial dimensions of the middle-layer feature map and the deep-layer feature map are respectively sampled onto the shallow-layer feature map. Figure 1 The fusion features are then obtained by splicing the components together. ,in, For channel-dimensional concatenation, the three feature maps are stacked in the channel direction to form a feature map with more channels but the same spatial size. The number of channels in the concatenated feature map is the sum of the number of channels in each input feature map.

[0046] Feature map size alignment is achieved through bilinear interpolation: This is the upsampling result of the mid-layer feature map, and its target size is the shallow-layer feature map. Size, This is the result of upsampling the deep feature map, which is The result was obtained after a second upsampling. The target size was obtained after two upsampling operations and is a shallow feature. Size, Indicates the target space size. Indicates an upsampling operation. This indicates bilinear interpolation calculation.

[0047] Step S3: Train the classification model using the training dataset.

[0048] In this embodiment, the Adam optimizer is used for training, and the learning rate is set to 1e. -4 The batch size was set to 32, and the training lasted for 50 epochs.

[0049] This invention improves the MobileNetV3 network by using a cooperative module in deep, high-channel-count inverse residual blocks and an SE attention module in shallow, low-channel-count inverse residual blocks, whereas existing MobileNetV3 networks use the SE attention module exclusively. Figures 4 to 6The collaborative module will be compared with the SE attention module used in the existing MobileNetV3 network.

[0050] like Figure 4 As shown, as the number of input channels increases from 32 to 256, the parameter reduction advantage of the collaborative module gradually expands, with the parameter reduction rate significantly increasing from 11.7% to 25.8%. This data indicates that the collaborative module has superior parameter efficiency in scenarios with a large number of channels, and its structural design can effectively control the growth rate of parameter size, achieving more significant parameter optimization effects in deep networks.

[0051] like Figure 5 As shown, the single-frame inference time of the collaborative module is 0.402 milliseconds, and the inference speed reaches 2487.6 FPS. Although slightly lower than the SE attention module, this performance data reflects the computational complexity of the module itself. The multi-branch architecture design adopted by the collaborative module introduces direction-aware convolutional branches and attention mechanisms, and its computational overhead is still controlled within a reasonable range, demonstrating good engineering feasibility.

[0052] like Figure 6 As shown, a comparative analysis using feature activation heatmaps visually demonstrates the performance differences between the collaborative module and the SE attention module in ship corrosion detection. The two modules exhibit different behavioral patterns; the collaborative module shows a significant improvement in average activation intensity, indicating enhanced response to corrosion features. In the region of interest visualization (SE module region of interest and OMCA module region of interest), the high-response regions marked by the collaborative module (OMCA module) have clearer and more concentrated outlines, while the response of the SE attention module is relatively dispersed. In the corresponding heatmap, the response peak of the collaborative module is more prominent and focused. This visualization result directly confirms the effectiveness of the collaborative module's structural design from a perceptual perspective; it enhances the model's feature discrimination ability by strengthening the directional focus on the main features of corrosion. The above comparison shows that the improvements to the MobileNetV3 network in this invention not only improve classification accuracy but also make the parameters more lightweight.

[0053] Figure 7 The system classifies the corrosion as "Grade C (Severe Corrosion)" with a confidence level of 97.04%. The probability distribution of all corrosion levels is also listed at the bottom of the interface, which intuitively presents the reliability of the classification results.

[0054] In this invention, the terms "first," "second," and "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. The use of terms such as "upper," "lower," "left," "right," "front," and "rear" to indicate orientation or positional relationships is based on the orientation or positional relationships shown in the accompanying drawings and is only for the convenience of describing the invention, not to indicate or imply that the device referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the scope of protection of this invention. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0055] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0056] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A ship rust classification method based on direction-aware multi-scale feature fusion, characterized in that: The method comprises the following steps: Step S1, shooting a ship image to obtain a training data set; Step S2, constructing a classification model, the classification model comprising a feature extraction network with multi-level inverted residual blocks and a feature pyramid network for realizing multi-level feature fusion, the feature extraction network using a collaborative module in deep large-channel inverted residual blocks and an SE attention module in shallow small-channel inverted residual blocks, the collaborative module comprising connected directional branches, spatial attention branches and channel attention branches, and the feature pyramid network fusing multi-level features extracted by the feature extraction network; Step S3, training the classification model using the training data set.

2. The method of claim 1, wherein the method is a method of ship rust classification based on orientation-aware multi-scale feature fusion. In the step S1, the shot ship image is adjusted to the same size, and the ship image after size adjustment is randomly rotated and flipped to obtain the training data set.

3. The ship rust classification method based on direction perception multi-scale feature fusion according to claim 2, characterized in that: In the step S2, the feature extraction network is a MobileNetV3 network, the inverted residual block adopts a combination of depth convolution, pointwise convolution and standard convolution, and the inverted residual block is configured with different input channel numbers, expansion channel numbers and output channel numbers, wherein when the expansion channel number is greater than a set M1, it is a deep large-channel inverted residual block, otherwise it is a shallow small-channel inverted residual block.

4. The ship rust classification method based on orientation perception multi-scale feature fusion according to claim 1 or 2 or 3, characterized in that: In the step S2, the direction branch extracts a direction feature map through group convolution wherein X is an input feature map of the synergy module, is an activation function, is a batch normalization operation, is a direction convolution weight, denotes a convolution calculation.

5. The method of claim 4, wherein the method is based on directionally aware multi-scale feature fusion for ship rust classification. In step S2, the spatial attention branch first uses a 1×1 convolution to convert the orientation feature map... Compressed into a single-channel feature map Then, the single-channel feature map Parallel execution of dual-channel pooling yields max pooling results. and average pooling results Finally, calculate the spatial weights. ,in, This represents a convolution operation with a kernel size of 1×1. Indicates single-channel characteristics The midpoint (i,j) is the local neighborhood of the center. Indicates the number of pixels in the neighborhood. For the Sigmoid function, For convolution operations with a kernel size of 5×5, , For splicing operations, Refers to the single-channel feature map The set of max pooling results at each point above. Refers to the single-channel feature map The set consisting of the average pooling results of all points.

6. The method of claim 5, wherein the method is based on directionally aware multi-scale feature fusion for ship rust classification. The output of the collaborative module in the step S2 is represented as , is the channel weight of the channel attention branch, wherein is the channel descriptor after global average pooling, , H is the length of the input feature map X, and W is the width of the input feature map X, is the pixel at position (i, j) in the cth channel of the input feature map X, and are the weights of the two 1x1 convolution layers, is the truncated ReLU activation function, represents element-wise multiplication.

7. The method of claim 6, wherein the method is based on directionally aware multi-scale feature fusion for ship rust classification. In the step S2, the feature pyramid network extracts a shallow feature map F s , a middle feature map F m , and a deep feature map F d of the output Y' of the feature extraction network respectively, and fuses the middle feature map and the deep feature map after upsampling the spatial dimensions of the middle feature map and the deep feature map to be consistent with the shallow feature map to obtain a fused feature , wherein, is channel dimension fusion, is the upsampling result of the middle feature map, is the upsampling result of the deep feature map, , denotes an upsampling operation, denotes a spatial dimension.

8. The ship rust classification method based on orientation perception multi-scale feature fusion according to claim 1 or 2 or 3, characterized in that: In the step S2, the classification model further comprises an average pooling layer connected with the output of the feature pyramid network and a fully connected layer connected with the average pooling layer.

Images