Ship rust classification method based on direction perception multi-scale feature fusion

Abstract

The application provides a ship rust classification method based on direction perception multi-scale feature fusion, and comprises the following steps: step S1, shooting a ship image to obtain a training data set; step S2, constructing a classification model, the classification model comprising a feature extraction network with multi-level reverse residual blocks and a feature pyramid network for realizing multi-level feature fusion, the feature extraction network adopting a collaborative module in a deep layer large-channel-number reverse residual block and an SE attention module in a shallow layer small-channel-number reverse residual block, the collaborative module comprising a connected direction branch, a spatial attention branch and a channel attention branch, and the feature pyramid network fusing multi-level features extracted by the feature extraction network; and step S3, training the classification model by using the training data set. The application takes into account both classification accuracy and speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship maintenance, in particular to a ship rust classification method based on direction perception multi-scale feature fusion. BACKGROUND

[0002] The ship is in the marine environment with high salinity and high humidity for a long time, and the ship surface is subjected to the double action of electrochemical corrosion and microbial corrosion, which is easy to produce rust in different degrees. Rust will lead to the attenuation of the structural strength of the ship body, so it is necessary to detect the ship rust. The traditional detection mainly relies on manual visual inspection combined with ultrasonic thickness gauge, but there are problems such as low detection efficiency, subjective deviation of manual judgment, safety hazards and the like.

[0003] In recent years, the application of automatic detection methods based on computer vision has gradually increased, but the existing detection methods face the following problems when applied to ship rust classification:

[0004] 1. Model deployment constraints: The model used by the existing method, such as ResNet34 with 2.5G FLOPs, has a large amount of calculation, which leads to a delay of more than 500ms in inference, and cannot meet the real-time requirements;

[0005] 2. Rust feature complexity: Rust morphology is diverse, and the scale changes greatly. The attention mechanism used by the existing method is difficult to effectively capture multi-scale spatial features.

[0006] 3. Insufficient feature utilization: The rust area will spread along the structure direction such as weld and rivet, and the texture directionality is strong and the edge is blurred. The existing method cannot effectively fuse the shallow detail features and deep semantic features of the network, resulting in insufficient recognition ability for blurred rust edges. SUMMARY

[0007] The main purpose of the present application is to provide a ship rust classification method based on direction perception multi-scale feature fusion, which takes into account the classification accuracy and speed.

[0008] The present application is realized by the following technical solutions:

[0009] The ship rust classification method based on direction perception multi-scale feature fusion comprises the following steps:

[0010] Step S1, photographing a ship image to obtain a training data set;

[0011] Step S2, a classification model is constructed, the classification model includes a feature extraction network with multi-level inverted residual blocks and a feature pyramid network for realizing multi-level feature fusion, the feature extraction network uses a collaborative module in a deep layer large channel number inverted residual block and an SE attention module in a shallow layer small channel number inverted residual block, the collaborative module includes a connected direction branch, a spatial attention branch and a channel attention branch, and the feature pyramid network fuses the multi-level features extracted by the feature extraction network;

[0012] Step S3, the classification model is trained using a training data set.

[0013] Further, in step S1, the photographed ship image is adjusted to the same size, and the ship image after size adjustment is randomly rotated and flipped to obtain a training data set.

[0014] Further, in step S2, the feature extraction network is a MobileNetV3 network, the inverted residual block uses a combination of depth convolution, pointwise convolution and standard convolution, and the inverted residual block is configured with different input channel numbers, expansion channel numbers and output channel numbers. When the expansion channel number is greater than a set M1, it is a deep layer large channel number inverted residual block, otherwise it is a shallow layer small channel number inverted residual block.

[0015] Further, in step S2, in step S2, the direction branch extracts a direction feature map where X is the input feature map of the collaborative module, is an activation function, is a batch normalization operation, is a direction convolution weight, denotes convolution calculation.

[0016] Further, the spatial attention branch first uses a 1x1 convolution to compress the direction feature map into a single-channel feature map Then, the single-channel feature map is executed in parallel with double-channel pooling to obtain a maximum pooling result and an average pooling result Finally, the spatial weight is calculated, where denotes a convolution operation with a kernel size of 1x1, denotes a local neighborhood centered at the midpoint (i,j) in the single-channel feature , denotes the number of pixels in the neighborhood, is a Sigmoid function, is a convolution operation with a kernel size of 5x5, , is a concatenation operation, denotes a set of maximum pooling results of each point in the single-channel feature map denotes a set of maximum pooling results of each point in the single-channel feature map denotes a set of average pooling results of each point in the single-channel feature map denotes a set of average pooling results of each point in the single-channel feature map

[0017] Further, in the step S2, the output of the collaborative module is represented as , is the channel weight of the channel attention branch, wherein is the channel descriptor after global average pooling, , H is the length of the input feature map X, and W is the width of the input feature map X, is the pixel at position (i, j) in the c-th channel of the input feature map X, and is the weight of the two 1x1 convolution layers, is the truncated ReLU activation function, denotes element-wise multiplication.

[0018] Further, in the step S2, the feature pyramid network extracts shallow feature maps F s , middle feature maps F m and deep feature maps F d of the output Y of the collaborative module, respectively, upsamples the spatial dimensions of the middle feature maps and the deep feature maps to the spatial dimensions of the shallow feature Figure One maps, and then splices to obtain the fusion feature , wherein is the channel dimension splicing, is the upsampled result of the middle feature map, is the upsampled result of the deep feature map, , denotes the upsample operation, denotes spatial dimensions.

[0019] Further, in the step S2, the classification model further comprises an average pooling layer connected with the output of the feature pyramid network and a fully connected layer connected with the average pooling layer.

[0020] As can be known from the above description of the present application, compared with the prior art, the present application has the following beneficial effects:

[0021] The classification model constructed in this invention includes a feature extraction network with multi-level inverted residual blocks and a feature pyramid network that realizes multi-level feature fusion. The feature extraction network uses a collaborative module in deep, large-channel-number inverted residual blocks and an SE attention module in shallow, small-channel-number inverted residual blocks. The collaborative module includes a directional branch, a spatial attention branch, and a channel attention branch. The feature pyramid network fuses the multi-level features extracted by the feature extraction network. The collaborative module effectively captures rust texture features from different directions and combines spatial attention and channel attention information for collaborative enhancement, overcoming the shortcomings of existing technologies that are insensitive to linear and irregular rust features. The feature pyramid network explicitly combines shallow detail features, mid-level structural features, and deep semantic features, enabling the model to simultaneously perceive the local texture and global distribution of rust, significantly improving the classification accuracy for complex shapes and blurred edge rust. It has strong adaptability to changes in rust scale, effectively recognizing both large-area uniform rust and small-scale point rust, reducing false positives and false negatives. Through a hierarchical attention mechanism, a lightweight SE attention module is used in the shallow layer, while a more powerful collaborative module is used only in the deep, computationally intensive areas, achieving a balance between performance and efficiency. Furthermore, the feature fusion process involves only simple upsampling and stitching operations, resulting in low computational overhead. Attached Figure Description

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 This is a flowchart of the present invention.

[0024] Figure 2 This is a schematic diagram of the classification model of the present invention.

[0025] Figure 3 This is a schematic diagram of the collaborative module of the present invention.

[0026] Figure 4 This is a comparison chart of the parameters of the collaborative module of the present invention and the SE attention module of the prior art.

[0027] Figure 5 This is a comparison chart of the inference speed of the collaborative module of this invention and the SE attention module of the prior art.

[0028] Figure 6 This paper describes the performance differences between the collaborative module of this invention and the SE attention module of the prior art in ship corrosion detection tasks.

[0029] Figure 7 This is a diagram illustrating the classification results of the present invention. Detailed Implementation

[0030] The present invention will be further described below through specific embodiments.

[0031] like Figure 1 As shown, the ship rust classification method based on direction-aware multi-scale feature fusion includes the following steps:

[0032] Step S1: Take images of the ship to obtain the training dataset;

[0033] The captured ship images were resized to the same size of 224×224 pixels. The resized ship images were then randomly rotated and flipped to enhance the data. The enhanced data was then divided into training dataset, validation set, and test set.

[0034] Step S2: Construct a classification model. The classification model includes a feature extraction network with multi-level inverted residual blocks, a feature pyramid network that realizes multi-level feature fusion, an average pooling layer connected to the output of the feature pyramid network, and a fully connected layer connected to the average pooling layer. The feature extraction network uses a collaborative module in deep, large-channel-number inverted residual blocks and an SE attention module in shallow, small-channel-number inverted residual blocks. The collaborative module includes a directional branch, a spatial attention branch, and a channel attention branch. The feature pyramid network fuses the multi-level features extracted by the feature extraction network.

[0035] The directional branch is responsible for extracting the directional texture features of rust, solving the problem of irregular morphology of rust spreading along the hull structure. In image processing, directional textures are commonly found in structures such as edges and stripes, while ship rust often spreads along the direction of welds or rivets. Therefore, directional perception can effectively improve feature discrimination. The spatial attention branch generates a spatial weight map, highlighting the spatial location of the rusted area and enhancing the salience of blurred edge areas, solving challenges such as blurred edges, low contrast, and noise interference in ship rust detection. The channel attention branch generates channel weight vectors, strengthening feature channels related to rust, such as color and texture, and suppressing background interference.

[0036] The overall structure diagram of the classification model is as follows: Figure 2As shown. The feature extraction network is the MobileNetV3 network, which contains a multi-level inverse residual block structure. Each inverse residual block uses a combination of depthwise convolution, pointwise convolution, and standard convolution. Inverse residual blocks are configured with different numbers of input channels, expanded channels, and output channels. Deep inverse residual blocks with a large number of channels use a collaborative module, while shallow inverse residual blocks with a small number of channels use an SE attention module. When the number of expanded channels is greater than a set threshold M=288, it is a deep inverse residual block with a large number of channels; otherwise, it is a shallow inverse residual block with a small number of channels, thus achieving adaptive attention mechanism selection. The SE attention module used in the shallow inverse residual blocks with a small number of channels is a current technology. The MobileNetV3 network consists of multiple stacked inverse residual blocks, with convolutional layers and fully connected layers at the beginning and end. The training dataset is input into the MobileNetV3 network and first passes through an expansion layer consisting of existing 1×1 convolutions, batch normalization (BN), and a hardswish activation function to increase the number of channels to the predetermined expanded number. Subsequently, the feature map is efficiently extracted using a 3×3 depthwise separable convolution (which includes depthwise convolution and pointwise convolution), and then batch normalization and activation processing are performed again.

[0037] Subsequently, the network selects either a collaborative module or an SE attention module based on the number of expanded channels. The output of the collaborative module or the SE attention module is then processed by existing 1×1 convolutions, residual connections, etc., to obtain the output feature map Y' of the MobileNetV3 network.

[0038] like Figure 3 As shown, the directional branch of the collaborative module uses grouped convolution to group the input feature map X by channel and independently extract directional feature maps (i.e., gradient features in different directions). The specific formula is expressed as follows: ,in, The input feature map X has a dimension of directional feature map. , For the number of channels, Given the length of the input feature map X, The input feature map X is the width, and D is the number of directions. The activation function is defined as follows: , Activation functions introduce nonlinear transformations to enhance feature representation capabilities. For batch normalization operations, The weights are the directional convolution weights (i.e., the convolution kernel, with a size of 3×3). This indicates convolution calculation. In this embodiment, the number of groups for grouped convolution is set to the number of directions of the rust, D=2, corresponding to the 45° and 135° directions respectively. The number of directions, D, is adjusted as a hyperparameter to balance direction sensitivity and computational efficiency.

[0039] The spatial attention branch first uses a 1x1 convolution to compress the directional feature map into a single-channel feature map to reduce the computational dimension and preserve the core information, , denotes a convolution operation with a kernel size of 1x1;

[0040] Then, a two-channel pooling is performed on the single-channel feature map in parallel to obtain the maximum pooling result and the average pooling result , denotes a local neighborhood centered at the point (i, j) in the single-channel feature map , denotes the number of pixels in the neighborhood;

[0041] Finally, the spatial weight is calculated, is a convolution operation with a kernel size of 5x5, which can provide sufficient receptive field to cover the irregular patches of rust while maintaining lightweight, is a Sigmoid function defined as to ensure that the spatial weight value is between [0, 1], , is a concatenation operation, , denotes a set composed of the maximum pooling values of each point in the single-channel feature map , denotes a set composed of the average pooling values of each point in the single-channel feature map , which focuses on local significant features such as high gradient edges of rust cracks, while the average pooling preserves the global smooth context. The fusion of the two can enhance the adaptability of the model to multi-scale changes in rust morphology.

[0042] The channel attention branch first performs global average pooling on the ship image to obtain the pooling results of each channel , and then calculates the channel weight , where is the channel descriptor after global average pooling, is the pixel at position (i, j) in the c-th channel of the input feature map X, and are the weights of two 1x1 convolution layers for input dimension reduction and dimension increase to reduce the amount of calculation, is a truncated ReLU activation function defined as . The above weights are obtained in the training.

[0043] The spatial information of each channel of the ship image is compressed into a global statistic by global average pooling. The numerical instability problem caused by excessively large activation values can be avoided, and the limited output range makes the precision loss smaller in the quantization process, which is more suitable for resource-limited embedded platforms such as ship rust removal robots.

[0044] Based on the above branches, the output of the coordination module is represented as , represents element-wise multiplication.

[0045] The feature pyramid network first extracts the shallow feature map Y' of the output of the feature extraction network , the middle layer feature map and the deep layer feature map , and then the spatial size of the middle layer feature map and the deep layer feature map is upsampled to the shallow layer feature Figure One to obtain the fused feature , wherein is the channel dimension splicing, which stacks the three feature maps in the channel direction to form a feature map with more channels but the same spatial size. The channel number of the spliced feature map is the sum of the channel numbers of the input feature maps.

[0046] The feature map size is aligned by bilinear interpolation: is the upsampled result of the middle layer feature map, and the target size is the size of the shallow layer feature , is the upsampled result of the deep layer feature map, which is after the second upsample, that is after two times of upsample, and the target size is the size of the shallow layer feature , represents the target spatial size, represents the upsample operation, represents bilinear interpolation calculation.

[0047] Step S3, training the classification model using the training data set.

[0048] In this embodiment, the Adam optimizer is used for training, the learning rate is set to 1e -4 , the batch size is set to 32, and the training is performed for 50 cycles.

[0049] The MobileNetV3 network is improved in the present application, that is, the coordination module is used in the deep layer large channel number reverse residual block, and the SE attention module is used in the shallow layer small channel number reverse residual block. The existing MobileNetV3 network uses the SE attention module. Figures 4 to 6The collaborative module is compared with the SE attention module used in the existing MobileNetV3 network.

[0050] As shown in Figure 4 With the increase of the number of input channels from 32 to 256, the parameter amount advantage of the collaborative module gradually expands, and the parameter reduction ratio is significantly improved from 11.7% to 25.8%. This data shows that the collaboration has superior parameter efficiency in the large channel number scenario, and its structural design can effectively control the growth rate of the parameter size, achieving more obvious parameter optimization effect in the deep network.

[0051] As shown in Figure 5 The single inference time of the collaborative module is 0.402 milliseconds, and the inference speed reaches 2487.6 FPS. Although it is slightly lower than the SE attention module, this performance data reflects the computational complexity of the module itself. The multi-branch architecture design of the collaborative module introduces a direction-aware convolution branch and an attention mechanism, and its computational overhead is still controlled within a reasonable range, which embodies good engineering realizability.

[0052] As shown in Figure 6 Through visual comparison and analysis of the feature activation heat map, the performance difference between the collaborative module and the SE attention module in the ship corrosion detection task is intuitively displayed. The two modules exhibit different behavior patterns, and the collaborative module shows a significant improvement in average activation intensity, indicating that its response to corrosion features is enhanced. In the attention area visualization graph (SE module attention area and OMCA module attention area), the high response area marked by the collaborative module (OMCA module) is more clear and concentrated, while the response of the SE attention module is relatively dispersed; in the corresponding heat map, the response peak of the collaborative module is more prominent and focused. This visualization result directly confirms the effectiveness of the structural design of the collaborative module from the perception level, which enhances the directional focus on the main features of corrosion and improves the feature discrimination of the model. From the above comparison, it can be seen that the improvement of the MobileNetV3 network in the present application not only improves the classification accuracy, but also makes the parameters more lightweight.

[0053] Figure 7 For the classification result display graph, it is classified as "C-class corrosion (severe corrosion)", and the confidence is as high as 97.04%. The probability distribution of all corrosion levels is also listed below the interface, which intuitively presents the reliability of the classification result.

[0054] In the present application, the terms "first", "second", "third" and the like are used only to distinguish similar objects, and do not have to be used to describe a particular order or sequence, nor can they be understood to indicate or imply relative importance. In the description, the orientation or position relationship indicated by "upper", "lower", "left", "right", "front" and "back" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application, and does not indicate or imply that the device referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the scope of protection of the present application. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0055] In addition, in the description of the present application, "a plurality of" means two or more, unless otherwise specified. The association relationship between the associated objects described by "and / or" indicates that there can be three relationships, for example, A and / or B can represent the three cases of A existing alone, A and B existing together, and B existing alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0056] The above is only a specific embodiment of the present application, but the design concept of the present application is not limited thereto, and any non-essential modification of the present application using this concept shall be deemed to infringe the scope of protection of the present application.

Claims

1. A ship rust classification method based on direction-aware multi-scale feature fusion, characterized in that: The method comprises the following steps: Step S1, shooting a ship image to obtain a training data set; Step S2, constructing a classification model, the classification model comprising a feature extraction network with multi-level inverted residual blocks and a feature pyramid network for realizing multi-level feature fusion, the feature extraction network using a collaborative module in deep large-channel inverted residual blocks and an SE attention module in shallow small-channel inverted residual blocks, the collaborative module comprising connected directional branches, spatial attention branches and channel attention branches, and the feature pyramid network fusing multi-level features extracted by the feature extraction network; Step S3, training the classification model using the training data set; In the step S2, the feature extraction network is a MobileNetV3 network, the inverted residual blocks adopt a combination of depth convolution, point-wise convolution and standard convolution, and the inverted residual blocks are configured with different input channel numbers, expansion channel numbers and output channel numbers, wherein when the expansion channel number is greater than a set M1, it is a deep large-channel inverted residual block, otherwise it is a shallow small-channel inverted residual block. In the step S2, the direction branch extracts a direction feature map through grouped convolution wherein X is an input feature map of the synergy module, is an activation function, is a batch normalization operation, is a direction convolution weight, denotes a convolution calculation; In step S2, the spatial attention branch first uses a 1×1 convolution to convert the orientation feature map... Compressed into a single-channel feature map Then, for single-channel feature maps Parallel execution of dual-channel pooling yields max pooling results. and average pooling results Finally, calculate the spatial weights. ,in, This represents a convolution operation with a kernel size of 1×1. Indicates single-channel characteristics The midpoint (i,j) is the local neighborhood of the center. Indicates the number of pixels in the neighborhood. For the Sigmoid function, For convolution operations with a kernel size of 5×5, , For splicing operations, Refers to the single-channel feature map The set of max pooling results at each point above. Refers to the single-channel feature map The set consisting of the average pooling results of all points.

2. The method of claim 1, wherein the method is a method of ship rust classification based on orientation-aware multi-scale feature fusion. In the step S1, the shot ship image is adjusted to the same size, and the ship image after size adjustment is randomly rotated and flipped to obtain the training data set.

3. The ship rust classification method based on orientation perception multi-scale feature fusion according to claim 2, characterized in that: The output of the collaborative module in the step S2 is represented as , is the channel weight of the channel attention branch, wherein is the channel descriptor after global average pooling, , H is the length of the input feature map X, and W is the width of the input feature map X, is the pixel at position (i, j) in the cth channel of the input feature map X, and are the weights of the two 1x1 convolution layers, is the truncated ReLU activation function, represents element-wise multiplication.

4. The ship rust classification method based on orientation perception multi-scale feature fusion according to claim 3, characterized in that: In the step S2, the feature pyramid network extracts a shallow feature map F s , a middle feature map F m , and a deep feature map F d of the output Y' of the feature extraction network respectively, and fuses the middle feature map and the deep feature map after upsampling the spatial dimensions of the middle feature map and the deep feature map to be consistent with the shallow feature map to obtain a fused feature , wherein, is channel dimension splicing, is the middle feature map upsampling result, is the deep feature map upsampling result, , represents an upsampling operation, represents a spatial dimension.

5. The ship rust classification method based on orientation perception multi-scale feature fusion according to claim 1 or 2 or 3, characterized in that: In the step S2, the classification model further comprises an average pooling layer connected with the output of the feature pyramid network and a fully connected layer connected with the average pooling layer.

Images