Image damage identification method based on multi-scale feature fusion

Abstract

The invention provides an image damage identification method based on multi-scale feature fusion, belongs to the technical field of intelligent detection and computer vision, and can at least partially solve the problems of insufficient utilization of multi-scale features, negative migration in multi-task identification and difficulty in adaptive modeling of different damage types in the prior art. The method comprises the following steps: acquiring image data of a to-be-detected object or two-dimensional feature map data obtained by converting a one-dimensional signal, and preprocessing the image data or the two-dimensional feature map data; inputting the preprocessed data into a trunk feature extraction network of an attention mechanism to extract multi-level features; performing multi-head self-attention enhancement modeling on the high-level features; inputting the multi-level features and the enhanced high-level features into a bidirectional pyramid feature fusion network for multi-scale fusion; and firstly predicting a damage area and then predicting damage description by adopting a cascade multi-task output structure based on the fusion features. According to the invention, damage identification under complex working conditions is realized, and the detection accuracy is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection and computer vision technology, specifically relating to an image damage recognition method based on multi-scale feature fusion. Background Technology

[0002] Image damage recognition is an important technical means to automate the condition monitoring, structural safety assessment, and quality inspection of industrial equipment. Current technologies typically employ convolutional neural networks to extract features from images and complete the damage recognition task through classification or regression methods.

[0003] However, existing technologies still have the following shortcomings: In terms of multi-scale feature utilization, most methods only utilize single-scale or simple feature fusion methods, making it difficult to simultaneously take into account local detailed features and global semantic features, thus limiting their ability to identify small-scale or early-stage damage. Shallow network features contain rich texture and edge information, while deep network features contain abstract semantic information; single-scale features cannot fully express the complete characteristics of damage.

[0004] In multi-task identification, tasks such as damage region, damage type, and damage description often employ parallel prediction methods, ignoring the inherent dependencies between tasks and easily leading to negative transfer between tasks. The damage region localization results are of significant reference value for accurate damage description prediction, and parallel prediction methods cannot utilize these conditional dependencies between tasks.

[0005] Regarding the adaptability to different damage types, different damage types vary significantly in morphological features, spatial distribution, and texture characteristics. Using a unified prediction structure makes it difficult to adapt to multiple damage patterns, affecting the model's generalization ability. Crack damage is distributed linearly, corrosion damage is distributed planarly, and wear damage is distributed point-like. A unified network structure is insufficient for differentiated modeling of different damage types.

[0006] Regarding recognition stability, existing methods lack adaptive decision-making mechanisms for complex working conditions, resulting in insufficient recognition stability under noise interference or class imbalance. Image data collected in industrial fields often contains noise factors such as illumination changes, occlusion, and background interference, and the robustness of existing methods needs to be improved.

[0007] Therefore, how to design an image damage recognition technology that can integrate multi-scale features, reduce multi-task interference, and adaptively model different damage types is a technical problem that urgently needs to be solved in the current industrial inspection field. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art, and to provide an image damage recognition method based on multi-scale feature fusion.

[0009] To achieve the above objectives, the present invention provides an image damage recognition method based on multi-scale feature fusion, comprising: The input data of the object to be detected is acquired and preprocessed to obtain standardized input data. The input data is image data or two-dimensional feature map data obtained by converting a one-dimensional signal through a time-frequency transformation method. The preprocessing includes size normalization processing and pixel normalization processing. The standardized input data is input into the backbone feature extraction network, and the standardized input data is subjected to hierarchical feature extraction through a residual network that introduces a channel attention mechanism to obtain multi-level features including shallow features, middle features and high-level features. The extracted high-level features are modeled with attention enhancement by introducing global contextual information and establishing long-range dependencies through a multi-head self-attention mechanism to obtain enhanced high-level features. The multi-level features and the enhanced high-level features are input into a bidirectional pyramid feature fusion network. The bidirectional fusion of multi-scale features is achieved through top-down and bottom-up paths to obtain multi-scale fused features. Based on the multi-scale fusion features, a cascaded multi-task output structure is adopted. First, the damage area information is predicted to obtain the damage area prediction result. Then, the damage area prediction result is constructed into a condition vector. The condition vector is fused with the multi-scale fusion features to predict the damage description and output the image damage recognition result.

[0010] Furthermore, the time-frequency transformation method includes short-time Fourier transform, wavelet transform, or Hilbert-Huang transform, converting the one-dimensional signal into a two-dimensional time-frequency feature map. The tensor form of the standardized input data is represented as follows: ; in, Indicates the number of channels. and These represent the height and width, respectively.

[0011] Furthermore, the channel attention mechanism is implemented as follows: Perform global average pooling on the feature map output by the residual block to obtain the channel description vector: ; in, and These represent the height and width of the feature map, respectively. Indicates the first Each channel is located in , Eigenvalues ​​at; The channel description vector is input into an activation network consisting of two fully connected layers, and the weight coefficients of each channel are output: ; in, This represents the description values ​​of each channel. The channel description vector is composed of and These represent the weight matrices of the two fully connected layers, Represents the ReLU activation function. This represents the Sigmoid activation function; The weight coefficients are multiplied channel by channel by channel by the original feature map to obtain the feature map after channel attention enhancement.

[0012] Furthermore, the calculation method for the multi-head self-attention mechanism is as follows: The high-level features are mapped to embedded features and flattened into a sequence form, introducing two-dimensional positional encoding; The sequences after introducing position encoding are mapped to query matrices respectively. Key matrix Sum matrix ; Calculate attention output: ; in, The dimension of the key vector. This represents the normalized exponential function; The attention output is reconstructed into a feature map form and then residually fused with the high-level features to obtain the enhanced high-level features.

[0013] Furthermore, the fusion method of the bidirectional pyramid feature fusion network is as follows: The top-down path is calculated as follows: ; in, express Multiply Convolution operation, Indicates an upsampling operation. This represents the input features of the i-th layer. Represents the fused features of the i-th layer; The bottom-up path is calculated as follows: ; in, Indicates step size is of Multiply Convolution operation, Indicates the first Multi-scale fusion features output by the layer; The multi-scale fusion feature output after bidirectional fusion has the same number of levels as the input.

[0014] Furthermore, the condition vector is constructed as follows: The predicted damage area is normalized using an exponential function to obtain the damage area probability vector: ; in, Represents the region prediction score vector. Indicates the number of region categories. and Indicates a region category index; The probability vector of the damaged area is used as the condition vector for subsequent damage description prediction.

[0015] Furthermore, the multi-scale fusion features are subjected to global average pooling and concatenation before being input into the cascaded multi-task output structure to obtain a global feature vector: ; in, This indicates a global average pooling operation. Indicates the first Multi-scale fusion features output by the layer. This indicates the number of feature levels. The global feature vector is used for damage region prediction and damage description prediction.

[0016] Furthermore, the fusion method of the conditional vector and the multi-scale fusion feature includes one of the following: The concatenation and fusion method involves concatenating the conditional vector with the multi-scale fusion feature along the channel dimension to obtain the fusion feature. ; in, Represents the global feature vector. Represents the probability vector of the damaged area. Indicates a splicing operation; Feature modulation fusion method: Scaling parameters are generated based on the condition vector. and offset parameters The multi-scale fusion features are subjected to a channel-by-channel affine transformation to obtain the fusion features: ; in, This indicates element-wise multiplication.

[0017] Furthermore, the network parameters of the backbone feature extraction network, the bidirectional pyramid feature fusion network, and the cascaded multi-task output structure are obtained through training and optimization based on a joint loss function. The joint loss function is calculated as follows: ; in, Indicates the predicted loss in the damaged area. This indicates that the damage description predicts the loss. and The weighting coefficients are represented by the cross-entropy loss function, which is used to calculate both the damage region prediction loss and the damage description prediction loss.

[0018] Furthermore, the method also includes a gated hybrid expert prediction step: A gating network is constructed based on the multi-scale fusion features. The output of the gating network is processed by a normalized exponential function to obtain the gating probability of each damage type. ; in, Indicates the first Gating scores for different damage types Indicates the first Gating scores for different damage types Indicates the number of damage types; For each type of damage, a corresponding expert network is constructed, and the cascaded multi-task output structure is used within each expert network. The outputs of each expert network are weighted and fused based on the gating probabilities: ; in, Indicates the first The output of the expert network yields the final damage area and damage description identification results.

[0019] The beneficial effects of this invention are as follows: This invention achieves full fusion of multi-scale features through a bidirectional pyramid feature fusion network, propagating high-level semantic information from the top-down path and feeding back low-level detailed information from the bottom-up path, effectively improving the ability to identify damage of different sizes and shapes.

[0020] This invention introduces a channel attention mechanism into the backbone network to achieve adaptive allocation of feature learning channels for different damage types. This enables the network to dynamically strengthen feature channels related to specific damages and suppress irrelevant features based on the characteristics of the input data, thereby improving the discriminative ability of feature representation.

[0021] This invention uses a multi-head self-attention mechanism to globally model high-level features, introduces global contextual information, establishes long-range dependencies of damage in the spatial dimension, and improves the recognition accuracy in complex backgrounds.

[0022] This invention employs a cascaded multi-task output structure. First, it predicts the damage area information, and then uses the area prediction result as a conditional input for damage description prediction. It explicitly models the conditional dependency between the damage area and the damage description, reduces negative transfer between multiple tasks, and improves the stability of damage description recognition.

[0023] This invention utilizes a gated hybrid expert structure to construct expert networks for different damage types, enabling different damage types to be adaptively processed by corresponding experts, reducing category confusion, and significantly improving detection accuracy in turbine blade surface damage detection experiments. Attached Figure Description

[0024] Figure 1 This is a flowchart of the neural network algorithm for the image damage recognition method based on multi-scale feature fusion of the present invention; Figure 2 This is a flowchart of the overall method of the present invention; Figure 3 This is a schematic diagram of the multi-scale feature fusion network structure of the present invention; Figure 4 This is a schematic diagram of the cascaded multi-task output structure of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates this application. It should be understood that the specific embodiments described in this specification are merely for explaining this application and are not intended to limit it.

[0026] The image damage recognition method based on multi-scale feature fusion of this invention is implemented on a deep learning platform system. The deep learning platform system adopts an architecture design that combines server-side training and edge deployment. The server side is responsible for core functions such as model training, parameter optimization, and performance evaluation. The trained model is deployed at the edge to perform real-time damage detection tasks.

[0027] The server-side hardware configuration includes compute servers and storage devices. The compute servers utilize a high-performance GPU cluster, with each node configured with an NVIDIA A100 GPU, a 64-core CPU, and 256GB of memory. The storage devices employ a high-speed solid-state storage array with a total capacity of no less than 20TB, used to store training datasets, model parameters, and detection results.

[0028] The overall architecture of the image damage recognition method consists of five core modules: data preprocessing module, backbone feature extraction module, attention enhancement module, multi-scale fusion module, and cascaded prediction module.

[0029] The data preprocessing module is responsible for standardizing the input data. This module supports both image data and one-dimensional signal data as input. For image data, size normalization and pixel normalization are performed directly. For one-dimensional signal data, it is first converted into a two-dimensional feature map using a time-frequency transform method, and then standardized.

[0030] The backbone feature extraction module employs a residual network with a channel attention mechanism to perform hierarchical feature extraction on the standardized input data. This module outputs multi-level feature maps, with shallow features representing texture and edge information, and deep features representing semantic structure information.

[0031] The attention enhancement module performs global modeling of the highest-level features output by the backbone network. This module employs a multi-head self-attention mechanism, introducing global contextual information by modeling the correlations between different positions in the feature sequence, thereby obtaining enhanced features containing long-range dependencies.

[0032] The multi-scale fusion module adopts a bidirectional pyramid feature fusion network structure. This module propagates high-level semantic information through a top-down path and feeds back low-level detailed information through a bottom-up path, thereby achieving full fusion of multi-scale features.

[0033] The cascaded prediction module adopts a cascaded multi-task output structure. This module first predicts the damage area information, and then uses the area prediction results as conditional input for damage description prediction, explicitly modeling the conditional dependencies between tasks.

[0034] Example 1 This embodiment uses turbine blade surface damage detection as an application scenario. Turbine blades are the core components of gas turbines, and they are prone to various forms of damage such as cracks, corrosion, and wear under high temperature and high pressure operating environments. Regular inspection is crucial to ensuring the safe operation of the equipment.

[0035] The system configuration in this embodiment is as follows. The training dataset contains 5000 images of the turbine blade surface, including 2000 normal images and 3000 damaged images. Damage types include cracks, corrosion, and wear. Damage region annotations include three categories: blade root region, blade body region, and blade tip region. Damage description annotations include three categories: minor damage, moderate damage, and severe damage. The image resolution is uniformly 512 by 512 pixels. The training set and test set are divided in an 8:2 ratio.

[0036] See Figure 2 The damage identification method in this embodiment includes the following steps: Step S1: Data preprocessing.

[0037] In this embodiment, image data of the turbine blade surface is acquired first. The image data is collected during maintenance using an industrial camera with a resolution of 1920 x 1080 pixels and a collection distance of 30 cm. A ring-shaped LED light source is used for illumination to ensure uniform lighting.

[0038] Preprocessing is performed on the acquired raw images. Size normalization scales the raw images to a target size of 512 x 512 pixels, using bilinear interpolation to maintain image quality. Pixel normalization scales the image pixel values ​​from integers (0-255) to floating-point numbers (0-1). The normalization formula is: ; in, Represents the original pixel value. This represents the normalized pixel value. This indicates the minimum pixel value is 0. This indicates a maximum pixel value of 255.

[0039] After preprocessing, the standardized input data is obtained, and its tensor form is represented as follows: ; in, In this embodiment, the number of channels is indicated. Corresponding to RGB three channels, and These represent the height and width of the data, respectively, in this embodiment. .

[0040] Step S2: Backbone feature extraction and channel attention enhancement.

[0041] Standardized input data is fed into the backbone feature extraction network. In this embodiment, ResNet50 is used as the backbone network skeleton, and a channel attention mechanism is introduced into the residual blocks. The ResNet50 network contains one convolutional layer, one max pooling layer, and four residual stages, each containing 3, 4, 6, and 3 residual blocks, respectively.

[0042] The backbone network performs hierarchical feature extraction on the input data, extracting multi-level feature maps at the output of each residual stage: ; in, to These represent the feature maps of the five levels from shallow to deep. The spatial resolution is 128 by 128 pixels, and the number of channels is 64. The spatial resolution is 64 by 64 pixels, and the number of channels is 256. The spatial resolution is 32 by 32 pixels, and the number of channels is 512. The spatial resolution is 16 by 16 pixels, and the number of channels is 1024. The spatial resolution is 16 by 16 pixels, and the number of channels is 2048.

[0043] Shallow features and It primarily characterizes the texture and edge information of the damage, including detailed features such as crack direction and corrosion texture patterns. Deep features and It mainly characterizes the semantic structural features of damage, including abstract information such as damage type and damage severity.

[0044] A channel attention mechanism is introduced in each residual block. The calculation process of the channel attention mechanism is as follows: First, a global average pooling operation is performed on the feature map output by the residual block to compress the spatial dimension to 1 x 1, resulting in the channel description vector: ; in, Indicates the first The description value of each channel, Indicates the first Each channel is located in eigenvalues ​​at that location and These represent the height and width of the feature map, respectively.

[0045] Input the channel description vector into the activation network consisting of two fully connected layers, and output the weight coefficients of each channel: ; in, Represents the channel description vector. This represents the weight matrix of the first fully connected layer. Represents the ReLU activation function. This represents the weight matrix of the second fully connected layer. This represents the Sigmoid activation function. This represents the output channel weight coefficient vector. The first fully connected layer compresses the number of channels to one-sixteenth of the original number, and the second fully connected layer restores the number of channels to the original number.

[0046] The channel weight coefficients are multiplied channel by channel of the original feature map to obtain the feature map after channel attention enhancement: ; in, Indicates the first Feature maps enhanced by each channel Indicates the first The weighting coefficients of each channel.

[0047] Through the channel attention mechanism, the network can dynamically adjust the response intensity of different feature channels according to the characteristics of the input data, strengthen the feature channels related to damage, and suppress the irrelevant feature channels related to the background, so as to achieve adaptive allocation of feature learning channels for different damage types.

[0048] Step S3: High-level feature attention enhancement modeling.

[0049] The highest-level features output by the backbone network A multi-head self-attention mechanism is introduced for global feature modeling. This mechanism can capture the correlation between different spatial locations in the feature map and establish the long-range dependency of the damage in the spatial dimension.

[0050] First of all Mapped to embedded features. Let... The dimension is The number of channels is reduced to 256 by a 1-to-1 convolution, resulting in a dimensionality-reduced feature map. .

[0051] The dimensionality-reduced feature maps are flattened into a sequence. Flattening the spatial dimensions yields a sequence of length 256, with each sequence element having a dimension of 256. A learnable two-dimensional positional encoding is introduced to embed positional information into the sequence: ; in, This represents the flattened feature sequence. Represents the position encoding matrix, This represents the sequence after introducing positional encoding.

[0052] The sequence after introducing position encoding is mapped to a query matrix through three linear transformations. Key matrix Sum matrix : ; in, , , These represent the linear transformation weight matrices for the query, key, and value, respectively.

[0053] Calculate the multi-head self-attention output. This embodiment uses 8 attention heads, each with a dimension of 32. The attention calculation formula is: ; in, The dimension of the key vector is represented in this embodiment. , It is used as a scaling factor to prevent the gradient from vanishing due to an excessively large dot product result. This represents the normalization exponential function, used to normalize attention scores into a probability distribution.

[0054] The outputs of the eight attention heads are concatenated and then subjected to a linear transformation to obtain the final attention output: ; in, Indicates the first The output of each attention head, This represents the weight matrix of the output linear transformation.

[0055] Reconstruct the attention output to match the original Same spatial dimensions, and with the original Residual fusion of features: ; in, This indicates the enhanced high-level features. This represents the operation of reconstructing a sequence into a spatial feature map.

[0056] Through a multi-head self-attention mechanism, the network can model the long-range dependency of turbine blade damage in the spatial dimension, enabling damage features at different spatial locations to reference each other, introducing global contextual information, and improving the ability to distinguish damage features in complex backgrounds.

[0057] Step S4: Multi-scale feature fusion.

[0058] See Figure 3 In obtaining enhanced high-level features Afterwards, to as well as The features are then fed into the bidirectional pyramid feature fusion network PANet. PANet achieves full fusion of multi-scale features through a bidirectional path, both top-down and bottom-up.

[0059] The execution process of the top-down path is as follows. First, for... Perform a 1x1 convolution to adjust the number of channels to 256, resulting in... .Will After 2x upsampling and The results after adjusting the number of channels by multiplying by 1 are summed element by element to obtain the following result. And so on, calculating sequentially. , , The formula for calculating the top-down path is: ; in, This represents a 1-to-1 convolution operation. This indicates a 2x upsampling operation.

[0060] The execution process of the bottom-up path is as follows. Perform downsampling using a 3x3 convolution with a stride of 2, and... By adding element by element, we get And so on, calculating sequentially. , , The formula for calculating the bottom-up path is: ; in, This represents a 3x3 convolution operation with a stride of 2. .

[0061] After bidirectional fusion, multi-scale fusion features are output: ; This multi-scale feature fusion method enhances the propagation of high-level semantic information to low-level features through a top-down path and enhances the feedback of low-level detail information to high-level features through a bottom-up path. It effectively enhances the expressive ability of damage edge details and overall structural information, and is particularly suitable for the identification of complex damage morphologies on turbine blade surfaces.

[0062] Step S5: Cascaded multi-task prediction.

[0063] See Figure 4 Global average pooling is performed on the multi-scale fused features, and the results are concatenated to obtain a global feature vector. to Perform global average pooling operations separately to compress the spatial dimension of the feature maps at each level to 1x1, and then concatenate them along the channel dimension: ; in, This indicates a global average pooling operation. This indicates a splicing operation. This represents the global feature vector. Since each level of feature has 256 channels, the concatenated global feature vector has a dimension of 1280.

[0064] Based on global feature vectors Damage identification is performed using a cascaded multi-task output structure. This structure first predicts damage region information, then uses the region prediction results as conditional input for damage description prediction.

[0065] The process of damage region prediction is as follows. The global feature vector is... The input is a region classifier consisting of two fully connected layers. The first fully connected layer maps the feature dimension from 1280 to 256 using the ReLU activation function. The second fully connected layer maps the feature dimension from 256 to 3, corresponding to the three categories: leaf root region, leaf blade region, and leaf tip region. The region classifier outputs the predicted score of the damaged region: ; in, and These represent the first and second fully connected layers, respectively. This represents the region prediction score vector.

[0066] The probability vector of damaged areas is obtained by performing a normalized exponential function operation on the regional prediction scores: ; in, This represents the probability vector of the damaged area, which is used as a conditional vector for subsequent damage description and prediction.

[0067] The damage description prediction process is as follows. The condition vector... With global feature vectors The feature vectors are then fused. This embodiment employs a concatenation fusion method, concatenating the 3D conditional vector with the 1280-dimensional global feature vector along the channel dimension to obtain a 1283-dimensional fused feature vector. ; fuse feature vectors The input is a description classifier consisting of two fully connected layers. The first fully connected layer maps the feature dimension from 1283 to 256 using the ReLU activation function. The second fully connected layer maps the feature dimension from 256 to 3, corresponding to the three categories of minor, moderate, and severe damage. The description classifier outputs the predicted probability of the damage description: ; in, and These represent the first and second fully connected layers describing the classifier, respectively. This represents the damage description prediction probability vector.

[0068] During the model training phase, a joint loss function is used to jointly optimize damage region prediction and damage description prediction. The joint loss function is a weighted sum of the region prediction loss and the description prediction loss: ; in, Indicates the predicted loss in the damaged area. This indicates that the damage description predicts the loss. and This represents the weighting coefficient, in this embodiment Both the regional prediction loss and the descriptive prediction loss are calculated using the cross-entropy loss function: , ; in, The one-hot encoding representing the true label of the region. This represents the one-hot encoding that describes the actual label.

[0069] The model training parameters are set as follows. The optimizer uses the Adam algorithm, with an initial learning rate of 0.0001 and a cosine annealing learning rate scheduling strategy. The batch size is set to 16. The number of training epochs is set to 100. Data augmentation strategies include random horizontal flipping, random rotation of ±15 degrees, and random brightness adjustment of ±0.2.

[0070] Following the above steps, this embodiment was tested on a turbine blade surface damage detection dataset. The test results show that the damage area prediction accuracy reached 97.8%, and the damage description prediction accuracy reached 98.56%. Compared to the benchmark method using parallel prediction, the cascaded prediction method of this invention improves the damage description prediction accuracy by 3.5 percentage points.

[0071] Example 2 This embodiment uses industrial pipeline inner wall damage detection as an application scenario. During the transportation of high-temperature and high-pressure media, industrial pipelines are prone to damage such as corrosion, scaling, and cracks on their inner walls. Regular inspection is crucial for preventing leaks. Compared to turbine blade surface damage detection in Embodiment 1, the image characteristics of pipeline inner wall damage detection are different, but the overall detection process follows the same methodological framework.

[0072] The system configuration in this embodiment is as follows. The training dataset contains 4000 endoscopic images of the pipe's inner wall, including 1500 normal images and 2500 damaged images. Damage types include four categories: corrosion, scaling, cracks, and pits. Damage area annotations include three categories: the top area of ​​the pipe, the sidewall area of ​​the pipe, and the bottom area of ​​the pipe. Damage description annotations include three categories: minor damage, moderate damage, and severe damage. The image resolution is uniformly 384 x 384 pixels. The training set and the test set are divided in an 8:2 ratio.

[0073] The damage identification method in this embodiment is executed sequentially according to steps S1 to S5 as described in Embodiment 1. The specific implementation process of each step is adapted to the characteristics of pipeline inner wall damage detection.

[0074] Step S1: Data preprocessing.

[0075] In this embodiment, endoscopic image data of the inner wall of a pipe is acquired. The image data is collected by moving an industrial endoscope inside the pipe; the endoscope is equipped with a high-definition camera and an LED lighting source. Due to the complex lighting conditions inside the pipe, the acquired raw images may have problems such as uneven lighting and reflections.

[0076] The acquired raw images undergo preprocessing. Size normalization scales the raw images to a target size of 384 x 384 pixels. Pixel normalization scales the image pixel values ​​from an integer range of 0 to 255 to a floating-point range of 0 to 1. Due to the complex lighting conditions of the pipe inner wall images, this embodiment adds histogram equalization processing before pixel normalization to enhance image contrast.

[0077] After preprocessing, the standardized input data is obtained, and its tensor form is represented as follows: ; Step S2: Backbone feature extraction and channel attention enhancement.

[0078] Standardized input data is fed into the backbone feature extraction network. Similar to Example 1, this example uses ResNet50 with a channel attention mechanism as the backbone network.

[0079] The backbone network performs hierarchical feature extraction on the input data, obtaining multi-level feature maps. Since the input image resolution is 384 by 384 pixels, the spatial resolution of each level of feature map is adjusted accordingly. The spatial resolution is 96 by 96 pixels. The spatial resolution is 48 by 48 pixels. The spatial resolution is 24 by 24 pixels. The spatial resolution is 12 by 12 pixels. The spatial resolution is 12 by 12 pixels.

[0080] The channel attention mechanism is implemented in the same way as in Example 1. Through the channel attention mechanism, the network can adaptively enhance feature channels related to damage to the inner wall of the pipe and suppress irrelevant feature channels related to the background texture of the pipe.

[0081] Step S3: High-level feature attention enhancement modeling.

[0082] For the highest layer features A multi-head self-attention mechanism is introduced for global feature modeling. Because... The spatial resolution is 12 by 12 pixels, which, when flattened, results in a sequence of length 144. This embodiment uses 8 attention heads, each with a dimension of 32, consistent with Embodiment 1.

[0083] The calculation process for the multi-head self-attention mechanism is the same as in Example 1. The attention calculation formula is: ; Through a multi-head self-attention mechanism, the network can model the long-range dependence of pipeline inner wall damage in the spatial dimension, enabling damage features at different spatial locations to be cross-referenced.

[0084] Step S4: Multi-scale feature fusion.

[0085] Will to and enhanced high-level features The bidirectional pyramid feature fusion network PANet is input together. The bidirectional fusion process is the same as in Example 1, including a top-down path and a bottom-up path.

[0086] The top-down path propagates high-level semantic information to low-level features step by step. The bottom-up path feeds back low-level detailed information to high-level features step by step. After bidirectional fusion, multi-scale fused features are output. ; This multi-scale feature fusion method effectively enhances the ability to express the details of the damage edges and the overall structural information of the pipeline inner wall, and is particularly suitable for the identification of complex damage morphologies of the pipeline inner wall.

[0087] Step S5: Cascaded multi-task prediction.

[0088] Global average pooling is performed on the multi-scale fused features and then concatenated to obtain the global feature vector. The dimension is 1280.

[0089] Based on global feature vectors Damage identification is performed using a cascaded multi-task output structure. First, damage region information is predicted, and a 3D region probability vector is output. This corresponds to three categories: the top area of ​​the pipe, the side wall area of ​​the pipe, and the bottom area of ​​the pipe.

[0090] region probability vector As condition vector and global feature vector The features are concatenated and fused to obtain a fused feature vector. The dimension is 1283.

[0091] The fused feature vector is input into the descriptive classifier, which outputs a 3-dimensional descriptive probability vector. These correspond to three categories: mild injury, moderate injury, and severe injury.

[0092] The model training parameters were set the same as in Example 1. The optimizer used the Adam algorithm, with an initial learning rate of 0.0001. The batch size was set to 16. The number of training epochs was set to 100.

[0093] Following the above steps, this embodiment was tested on a pipeline inner wall damage detection dataset. The test results showed that the damage area prediction accuracy reached 96.5%, and the damage description prediction accuracy reached 97.2%. Compared to the benchmark method using parallel prediction, the cascaded prediction method of this invention improved the damage description prediction accuracy by 2.8 percentage points.

[0094] Example 3 This embodiment uses bridge structure crack detection as an application scenario and introduces a gated hybrid expert prediction module. During long-term service, bridge structures are susceptible to crack damage due to factors such as vehicle loads, temperature changes, and material aging. Regular inspection is crucial for ensuring the safe operation of bridges. Compared to the previous two embodiments, this embodiment features more diverse damage types and employs a gated hybrid expert structure for differentiated modeling.

[0095] The system configuration in this embodiment is as follows. The training dataset contains 6000 images of the bridge structure surface, including 2000 normal images and 4000 damaged images. Damage types include four categories: transverse cracks, longitudinal cracks, network cracks, and block spalling. Damage area annotations include three categories: bridge deck area, bridge web area, and pier area. Damage description annotations include three categories: micro-cracks, ordinary cracks, and through cracks. The image resolution is uniformly 448 x 448 pixels. The training set and test set are divided in an 8:2 ratio.

[0096] The damage identification method in this embodiment performs data preprocessing, backbone feature extraction, attention enhancement and multi-scale fusion according to steps S1 to S4 of Embodiment 1, and introduces a gated hybrid expert prediction module in step S5.

[0097] The execution process of steps S1 to S4 is the same as in Embodiments 1 and 2, and will not be repeated here. The parameters of each step are adaptively adjusted to suit the characteristics of bridge structure crack detection. The input image resolution is 448 by 448 pixels. The spatial resolution is 112 by 112 pixels. The spatial resolution is 56 by 56 pixels. The spatial resolution is 28 by 28 pixels. The spatial resolution is 14 by 14 pixels. The spatial resolution is 14 by 14 pixels.

[0098] Step S5: Gated hybrid expert multi-task prediction.

[0099] Global average pooling is performed on the multi-scale fused features and then concatenated to obtain the global feature vector. The dimension is 1280.

[0100] Based on global feature vectors A gating network is constructed to predict damage types. The gating network consists of two fully connected layers. The first fully connected layer maps the feature dimension from 1280 to 256 and uses the ReLU activation function. The second fully connected layer maps the feature dimension from 256 to 4, corresponding to four damage types: transverse cracks, longitudinal cracks, mesh cracks, and blocky peeling. The output of the gating network is processed by a normalized exponential function to obtain the gating probability vector. ; in, This represents the fully connected layer transformation of a gated network. Represents the gating probability vector. Indicates the number of damage types, in this embodiment , Indicates the first Gating scores for different damage types.

[0101] For each damage type, a corresponding expert network is constructed. This embodiment constructs four expert networks, corresponding to transverse crack experts, longitudinal crack experts, mesh crack experts, and block spalling experts, respectively. Each expert network employs a cascaded multi-task output structure, identical to the cascaded prediction structure in Embodiment 1.

[0102] The cascaded prediction process for each expert network is as follows. First, based on the global feature vector... Predict damage area information and output a 3D region probability vector. Then, fuse the region probability vector as a conditional vector with the global feature vector to predict damage description information, outputting a 3D description probability vector.

[0103] No. The output of an expert network is represented as follows: ; in, Indicates the first The region prediction probability vector of an expert network Indicates the first A description of the predicted probability vector of an expert network.

[0104] The outputs of each expert network are weighted and fused based on gating probabilities to obtain the final damage area and damage description identification results: , ; in, Indicates the first Gating probabilities for different damage types This represents the final region prediction probability vector. This represents the final predicted probability vector.

[0105] During the model training phase, a joint loss function is used to jointly optimize the region prediction and description prediction of the gating network and each expert network. The joint loss function is a weighted sum of the three losses: ; in, This indicates the damage type prediction loss of the gated network. This represents the predicted loss after merging the regions. This represents the predicted loss after fusion. , , This represents the weighting coefficient, in this embodiment All three loss terms are calculated using the cross-entropy loss function.

[0106] The formula for calculating the loss of damage type prediction in a gated network is as follows: ; in, One-hot encoding representing the true label of the damage type.

[0107] The model training parameters were set as follows. The optimizer used the Adam algorithm, with an initial learning rate of 0.0001 and a cosine annealing learning rate scheduling strategy. The batch size was set to 12. The number of training epochs was set to 120. Data augmentation strategies included random horizontal flipping, random vertical flipping, random rotation by ±30 degrees, and random cropping to 80% to 100% of the original image.

[0108] Following the above steps, this embodiment was tested on a bridge structure crack detection dataset. The test results show that the accuracy rate for damage type prediction reached 96.2%, the accuracy rate for damage area prediction reached 95.8%, and the accuracy rate for damage description prediction reached 97.1%. Compared to the benchmark method using a unified prediction structure, the gated hybrid expert structure of this invention improves the accuracy rate for damage type prediction by 2.1 percentage points and the accuracy rate for damage description prediction by 2.9 percentage points.

[0109] Comparative Experiments: To verify the effectiveness of the method of this invention, comparative experiments were conducted on the datasets of the three embodiments. The comparison methods included: traditional convolutional neural network ResNet50, ResNet50-FPN with feature pyramid network, and ResNet50-SE with attention mechanism.

[0110] Comparison results on the dataset of Example 1: The damage description accuracy of the ResNet50 method is 91.3%, the damage description accuracy of the ResNet50-FPN method is 94.2%, the damage description accuracy of the ResNet50-SE method is 95.1%, and the damage description accuracy of the method of this invention is 98.56%. The accuracy of the method of this invention is improved by 7.26 percentage points compared with the ResNet50 method.

[0111] Comparison results on the dataset of Example 2: The damage description accuracy of the ResNet50 method is 89.8%, the damage description accuracy of the ResNet50-FPN method is 92.5%, the damage description accuracy of the ResNet50-SE method is 93.9%, and the damage description accuracy of the method of the present invention is 97.2%. The accuracy of the method of the present invention is improved by 7.4 percentage points compared with the ResNet50 method.

[0112] Comparison results on the dataset of Example 3: The damage description accuracy of the ResNet50 method is 88.5%, the damage description accuracy of the ResNet50-FPN method is 91.3%, the damage description accuracy of the ResNet50-SE method is 92.8%, and the damage description accuracy of the method of the present invention is 97.1%. The accuracy of the method of the present invention is improved by 8.6 percentage points compared with the ResNet50 method.

[0113] In summary, the embodiments of the present invention have at least the following technical effects: This invention achieves full fusion of multi-scale features through a bidirectional pyramid feature fusion network, effectively improving the ability to identify damage of different sizes and shapes.

[0114] This invention introduces a channel attention mechanism into the backbone network to achieve adaptive allocation of feature learning channels for different damage types, thereby improving the discriminative ability of feature representation.

[0115] This invention uses a multi-head self-attention mechanism to globally model high-level features, introduces global contextual information, and improves recognition accuracy in complex backgrounds.

[0116] This invention employs a cascaded multi-task output structure to explicitly model the conditional dependency between the damage region and the damage description, thereby reducing negative migration between multiple tasks.

[0117] This invention utilizes a gated hybrid expert structure to enable different damage types to be adaptively processed by corresponding experts, thereby reducing category confusion and significantly improving detection accuracy.

[0118] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An image damage recognition method based on multi-scale feature fusion, characterized in that, include: The input data of the object to be detected is acquired and preprocessed to obtain standardized input data. The input data is image data or two-dimensional feature map data obtained by converting a one-dimensional signal through a time-frequency transformation method. The preprocessing includes size normalization processing and pixel normalization processing. The standardized input data is input into the backbone feature extraction network, and the standardized input data is subjected to hierarchical feature extraction through a residual network that introduces a channel attention mechanism to obtain multi-level features including shallow features, middle features and high-level features. The extracted high-level features are modeled with attention enhancement by introducing global contextual information and establishing long-range dependencies through a multi-head self-attention mechanism to obtain enhanced high-level features. The multi-level features and the enhanced high-level features are input into a bidirectional pyramid feature fusion network. The bidirectional fusion of multi-scale features is achieved through top-down and bottom-up paths to obtain multi-scale fused features. Based on the multi-scale fusion features, a cascaded multi-task output structure is adopted. First, the damage area information is predicted to obtain the damage area prediction result. Then, the damage area prediction result is constructed into a condition vector. The condition vector is fused with the multi-scale fusion features to predict the damage description and output the image damage recognition result.

2. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The time-frequency transformation method includes short-time Fourier transform, wavelet transform, or Hilbert-Huang transform, which converts the one-dimensional signal into a two-dimensional time-frequency feature map. The tensor form of the standardized input data is represented as follows: ； in, Indicates the number of channels. and These represent the height and width, respectively.

3. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The channel attention mechanism is implemented as follows: Perform global average pooling on the feature map output by the residual block to obtain the channel description vector: ； in, and These represent the height and width of the feature map, respectively. Indicates the first Each channel is located in , Eigenvalues ​​at; The channel description vector is input into an activation network consisting of two fully connected layers, and the weight coefficients of each channel are output: ； in, This represents the description values ​​of each channel. The channel description vector is composed of and These represent the weight matrices of the two fully connected layers, Represents the ReLU activation function. This represents the Sigmoid activation function; The weight coefficients are multiplied channel by channel by channel by the original feature map to obtain the feature map after channel attention enhancement.

4. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The calculation method for the multi-head self-attention mechanism is as follows: The high-level features are mapped to embedded features and flattened into a sequence form, introducing two-dimensional positional encoding; The sequences after introducing position encoding are mapped to query matrices respectively. Key matrix Sum matrix ; Calculate attention output: ； in, The dimension of the key vector. This represents the normalized exponential function; The attention output is reconstructed into a feature map form and then residually fused with the high-level features to obtain the enhanced high-level features.

5. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The fusion method of the bidirectional pyramid feature fusion network is as follows: The top-down path is calculated as follows: ； in, express Multiply Convolution operation, Indicates an upsampling operation. This represents the input features of the i-th layer. Represents the fused features of the i-th layer; The bottom-up path is calculated as follows: ； in, Indicates step size is of Multiply Convolution operation, Indicates the first Multi-scale fusion features output by the layer; The multi-scale fusion feature output after bidirectional fusion has the same number of levels as the input.

6. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The condition vector is constructed as follows: The predicted damage area is normalized using an exponential function to obtain the damage area probability vector: ； in, Represents the region prediction score vector. Indicates the number of region categories. and Indicates a region category index; The probability vector of the damaged area is used as the condition vector for subsequent damage description prediction.

7. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The multi-scale fusion features are subjected to global average pooling and concatenation before being input into the cascaded multi-task output structure to obtain a global feature vector: ； in, This indicates a global average pooling operation. This represents the multi-scale fused feature output from the i-th layer. This indicates the number of feature levels. The global feature vector is used for damage region prediction and damage description prediction.

8. The image damage recognition method based on multi-scale feature fusion according to claim 7, characterized in that, The fusion method of the conditional vector and the multi-scale fusion feature includes one of the following: The concatenation and fusion method involves concatenating the conditional vector with the multi-scale fusion feature along the channel dimension to obtain the fusion feature. ； in, Represents the global feature vector. Represents the probability vector of the damaged area. Indicates a splicing operation; Feature modulation fusion method: Scaling parameters are generated based on the condition vector. and offset parameters The multi-scale fusion features are subjected to a channel-by-channel affine transformation to obtain the fusion features: ； in, This indicates element-wise multiplication.

9. The image damage recognition method based on multi-scale feature fusion according to claim 1, characterized in that, The network parameters of the backbone feature extraction network, the bidirectional pyramid feature fusion network, and the cascaded multi-task output structure are obtained through training and optimization based on a joint loss function. The joint loss function is calculated as follows: ； in, Indicates the predicted loss in the damaged area. This indicates that the damage description predicts the loss. and The weighting coefficients are represented by the cross-entropy loss function, which is used to calculate both the damage region prediction loss and the damage description prediction loss.

10. The image damage recognition method based on multi-scale feature fusion according to any one of claims 1 to 9, characterized in that, It also includes a gated hybrid expert prediction step: A gating network is constructed based on the multi-scale fusion features. The output of the gating network is processed by a normalized exponential function to obtain the gating probability of each damage type. ； in, Indicates the first Gating scores for different damage types Indicates the first Gating scores for different damage types Indicates the number of damage types; For each type of damage, a corresponding expert network is constructed, and the cascaded multi-task output structure is used within each expert network. The outputs of each expert network are weighted and fused based on the gating probabilities: ； in, Indicates the first The output of the expert network yields the final damage area and damage description identification results.

Images