Floor defect identification method based on double attention mechanism

Abstract

This invention provides a method for identifying floor defects based on a dual attention mechanism. The model consists of two identical feature extractors and a feature classifier. The feature extractors are composed of a convolutional neural network and a channel attention module. First, the convolutional neural network extracts features from the input floor image. Then, the channel attention mechanism adaptively allocates the feature weights for each channel to suppress interference information within the same modality. The feature classifier includes a multimodal data attention module, a feature fusion layer, and a softmax classifier. The multimodal attention mechanism is used to adaptively adjust the feature weights of different modalities to improve the utilization rate of key fault information. After constructing the model, end-to-end model training is performed using a pre-defined training set and the cross-entropy loss function as the optimization objective. In application, the floor image to be identified and its corresponding edge texture map are input into the trained model to achieve the classification and identification of floor defects.

Description

Technical Field

[0001] This invention belongs to the field of industrial inspection and artificial intelligence technology, specifically involving a method for identifying floor defects based on a dual attention mechanism. Background Technology

[0002] During floor grinding, the surface material is gradually removed, exposing potential internal defects. However, current technology struggles to identify these defects automatically and in real-time during the grinding process, leading to a lack of timely warnings. Typically, after grinding, manual inspection and acceptance are required to assess the floor surface condition. However, the grinding area is vast, making manual inspection labor-intensive, inefficient, and prone to missing defects due to personnel experience and subjective factors.

[0003] Despite the rapid development of computer vision and artificial intelligence technologies in recent years, automatic defect recognition technology applied to floor grinding scenarios still has significant limitations. Existing deep learning-based recognition methods mostly use a single RGB image input, failing to fully utilize the structural and textural information exposed on the floor surface during grinding. This results in limited accuracy in recognizing minute defects, early-stage defects, and defects against complex backgrounds. Furthermore, existing methods typically lack effective fusion mechanisms for multimodal features and attention guidance, making it difficult to achieve stable and reliable real-time detection in the complex and ever-changing environment of a grinding site. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for identifying floor defects based on a dual attention mechanism. By embedding channel attention and attention modules into the model, this invention effectively improves the utilization rate of key defect image information.

[0005] A method for identifying floor defects based on a dual attention mechanism, the method specifically includes the following steps: Step 1: Data preparation and preprocessing. Collect ground sample images, perform edge texture processing on them to generate corresponding edge texture images; pair the original RGB images with the edge texture images to construct a labeled dataset, and divide it into training and test sets. Step 2: Model construction. Construct a recognition model containing two parallel feature extractors and one feature classifier. Each feature extractor includes a convolutional neural network and a channel attention module, used for feature extraction and semantic enhancement of input images of different modalities. The feature classifier includes a multimodal attention module, a feature fusion layer, and a classification output layer, used for weighted fusion and classification decision of dual-branch features. Step 3: Model training and optimization. The recognition model is trained end-to-end using the training set. During training, the error between the model output and the true label is calculated using the cross-entropy loss function. The model parameters are optimized using the backpropagation method until the model reaches the preset conditions, and the model weights with the best performance are saved. Step 4: Model evaluation. The model is evaluated using the test set and the saved optimal model weights. The accuracy and loss rate performance metrics are output, and a confusion matrix is ​​generated for error analysis. Step 5: Defect recognition application. For the floor image to be identified, generate its edge texture map according to the method in Step 1. Input the original RGB image and edge texture map into the model trained in Step 3, and realize the classification and recognition of floor defects based on the model output.

[0006] Furthermore, in step 1, the original RGB floor image is acquired using an industrial camera, converted into a grayscale image, and the Sobel gradient magnitude and Laplacian response are calculated. The two are then linearly fused according to empirical coefficients to generate an initial edge texture map, which is then normalized to obtain an edge texture map with the same size as the original RGB image and a pixel value range of [0,1]. All images are size-standardized, and data is augmented using rotation, inversion, and random cropping. The original RGB image and the corresponding edge texture image are used as paired samples and divided into a training set and a test set according to a preset ratio, while maintaining the pairing relationship between the two types of images during the division.

[0007] Furthermore, the two parallel feature extractors take the original RGB image and the corresponding edge texture image as input, respectively; the convolutional neural network of each feature extractor is used to extract the depth features of the input image, and the subsequent channel attention module is used to perform adaptive weight calibration of the channel dimension of the depth features; the multimodal attention module of the feature classifier is used to evaluate the importance of the features output by the two feature extractors and generate weights; the feature fusion layer multiplies the two feature vectors with their corresponding weights and then concatenates them to form a unified fused feature representation; in the classification output layer, the fused features output the predicted probability distribution of each defect category through the Softmax function to generate the defect category recognition result.

[0008] Furthermore, in step 3, the recognition model is trained end-to-end using the training set as input, and the prediction result is obtained through forward propagation. The cross-entropy loss between the prediction result and the real label is calculated. Based on the cross-entropy loss, the model parameters are updated through the backpropagation algorithm. This process is repeated until the model meets the maximum number of iterations, and the optimal model weights are saved during the training process.

[0009] Furthermore, in step 4, the optimal model weights saved in step 3 are loaded to initialize the recognition model; the model is evaluated using a test set, and the accuracy and loss performance indicators are calculated and output; a multi-class confusion matrix is ​​generated to analyze the recognition performance and error distribution of the model on different defect categories.

[0010] Furthermore, in step 5, for the RGB image of the floor to be identified, a corresponding edge texture image is generated according to the method described in step 1; the original RGB image and the edge texture image are used as a pair of inputs and input into the recognition model trained in step 3; the defect category classification result output by the recognition model is obtained to complete the automatic classification and recognition of floor defects.

[0011] This invention provides a method for identifying floor defects based on a dual attention mechanism. It has the following beneficial effects: 1. The dual-branch heterogeneous input architecture lays a solid foundation for the model. By processing the original RGB image and the edge texture image through two parallel channels, it ensures that the apparent color information and deep structural information are separated and complemented at the source, overcoming the inherent limitations of single-modal input.

[0012] 2. The model introduces a hierarchical dual attention mechanism. In the feature extraction stage, the channel attention module performs internal calibration and semantic enhancement on features from the same source; in the decision fusion stage, the multimodal attention module evaluates and weights the importance of features from different sources. The contribution ratio of RGB and edge features is dynamically adjusted according to the content of the specific input sample to achieve accurate classification of floor defects. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the model framework structure in this invention.

[0014] Figure 2 This is a schematic diagram of image enhancement in this invention.

[0015] Figure 3 This is a schematic diagram of the channel attention module in this invention.

[0016] Figure 4 This is a schematic diagram of the multimodal attention mechanism module in this invention.

[0017] Figure 5 This is a schematic diagram illustrating the accuracy in the concrete floor case of this invention.

[0018] Figure 6 This is a schematic diagram of the loss value in the concrete floor case of this invention.

[0019] Figure 7 This is a schematic diagram of the confusion matrix in the concrete floor case of this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see the appendix Figure 1 This invention provides a method for classifying floor defects based on a dual attention mechanism, comprising the following steps: Step 1: Data preparation and preprocessing. Collect ground sample images, perform edge texture processing on them to generate corresponding edge texture images; pair the original RGB images with the edge texture images to construct a labeled dataset, and divide it into training and test sets. Step 2: Model construction. Construct a recognition model containing two parallel feature extractors and one feature classifier. Each feature extractor includes a convolutional neural network and a channel attention module, used for feature extraction and semantic enhancement of input images of different modalities. The feature classifier includes a multimodal attention module, a feature fusion layer, and a classification output layer, used for weighted fusion and classification decision of dual-path features. Step 3: Model training and optimization. The recognition model is trained end-to-end using the training set. During training, the error between the model output and the real label is calculated using the cross-entropy loss function. The model parameters are optimized using the backpropagation method until the model reaches the preset conditions, and the model weights with the best performance are saved.

[0022] Step 4: Model evaluation. The model is evaluated using the test set and the saved optimal model weights. The accuracy and loss rate performance metrics are output, and a confusion matrix is ​​generated for error analysis. Step 5: Defect recognition application. For the floor image to be identified, generate its edge texture map according to the method in Step 1. Input both into the model trained in Step 3, and realize the automatic identification and classification of floor defects based on the model output.

[0023] Specifically, the floor samples in step 1 include: In this specific embodiment of the invention, the model input contains two complementary types of floor images. The first type is the original RGB image, obtained directly by an industrial camera, containing complete color and texture appearance information of the floor surface. The second type is the edge texture image, the acquisition process of which reflects targeted enhancement of defect geometry: First, the RGB image is converted into a grayscale image. Next, the magnitudes of their Sobel gradients are calculated. Response to Laplacian The former effectively captures the intensity and direction of edges, while the latter is extremely sensitive to second-order changes in the image. Therefore, the two are linearly fused using empirical coefficients. An initial texture map is generated, and this fusion strategy takes into account both macroscopic contours and microscopic abrupt changes. Finally, the edge texture map is normalized to obtain an edge texture map with pixel values ​​in the range [0,1]. These two types of images serve as inputs to the dual-branch model, providing a visual information foundation that is both independent and highly complementary for subsequent feature extraction and fusion.

[0024] Furthermore, all images were adjusted to a fixed resolution (e.g., 256×256), and data augmentation was performed using rotation, reversal, and random cropping. See the appendix for details. Figure 2 The original RGB images and their corresponding edge texture images are used as paired samples and divided into training and test sets according to a preset ratio. During the division process, the pairing relationship between the two types of images is strictly maintained, ensuring that each set of original RGB images and its generated edge texture images still belong to the same set after division. This provides a structurally consistent and accurately paired data foundation for subsequent model training and evaluation.

[0025] Specifically, two parallel feature extractors take the original RGB image and the corresponding edge texture image as input, respectively. The convolutional neural network in each feature extractor extracts features independently, outputting two independent feature vectors. After feature extraction in each branch, an attention module is introduced to enhance key features. Its core lies in using adaptive weight allocation to enhance feature responses related to defects and suppress irrelevant background interference, thereby achieving semantic enhancement within features of the same modality. The attention module first aggregates spatial context information through global average pooling and max pooling operations. Its mathematical expression is... ; In the formula, No. m The feature in the first n The first layer k The value of each neuron; Indicates the first Max pooling input values ​​for layer neurons; Indicates the first Global average pooling input values ​​of layer neurons; H It is the width of the pooling region.

[0026] Subsequently, the two pooled one-dimensional vectors are fed into a small multilayer perceptron with shared parameters for processing and fusion, ultimately generating the channel attention weight vector. M C The expression for evaluating the weights is: ; In the formula, Use the Sigmoid activation function; The input features are the output feature sequence after passing through a multilayer perceptron.

[0027] Finally, the weight vector is multiplied by the original feature map and then concatenated to form a unified fused feature representation, achieving adaptive weight allocation for feature channels. See attached diagram for details. Figure 3 As shown.

[0028] Furthermore, the attention-enhanced feature vectors from the two branches are fused. First, a multimodal attention module evaluates the importance of the features output from the two feature extractors and generates weights. The mathematical principle behind this process is shown in the formula. ; In the formula, V 1 and V 2 These are the feature vectors of two modes; and The formula corresponds to the weighted characteristics; and These are the weights corresponding to these two modes; and We can obtain the following from the formula: ; Then, the feature fusion layer multiplies the two feature vectors by their corresponding weights and concatenates them to form a unified fused feature representation. See attached diagram for details. Figure 4 As shown.

[0029] Furthermore, the fused feature representation is input into the classification decision layer for classification. This layer consists of multiple fully connected layers that map the fused features to the defect category space. Finally, the predicted probability of each defect category is output through the Softmax function, generating the defect category identification result.

[0030] Specifically, model training is conducted end-to-end. Through a forward propagation process, the model sequentially undergoes feature extraction, channel attention enhancement, multimodal fusion, and classification decision, ultimately outputting the predicted defect category probability distribution. Based on this prediction result and the true defect category labels in the image, the cross-entropy loss function is calculated, with the following formula: ; in, p(x) The target value; q(x) These are the predicted values ​​output by Softmax.

[0031] Subsequently, based on the cross-entropy loss value, the gradients of each parameter in the model are calculated using the backpropagation algorithm, and optimizers such as Adam are used to update the parameters, continuously improving the model's predictive ability. This process is repeated until the model meets the preset convergence condition or reaches the maximum number of iterations, and the model weights with the best performance on the validation set are saved during training.

[0032] Specifically, after model training is complete, its generalization performance needs to be objectively evaluated. First, the optimal model weight file saved in step 3 is loaded to initialize the recognition model. Then, the model is evaluated using a test set that has never been used in training. Paired images from the test set are input into the model to obtain the predicted categories, which are then compared with the true labels. Accuracy and loss values ​​are automatically calculated and output as performance metrics. Furthermore, a multi-class confusion matrix is ​​generated to analyze the model's recognition performance and error distribution across different defect categories.

[0033] Specifically, when deploying the trained and evaluated model into practical applications, the following process is followed: For the RGB image of the ground surface to be identified collected on-site, firstly, according to the edge texture processing method described in step 1, its corresponding edge texture image is generated. Then, this original RGB image and the generated edge texture image are used as a paired input and directly input into the recognition model trained in step 3. The model automatically performs a series of forward computation processes, including feature extraction, dual attention weighting and fusion, and classification decision. Finally, the system acquires and outputs the defect category recognition result given by the model.

[0034] Taking the task of identifying defects in concrete floor types as an example: Collect raw RGB floor images, including categories such as health, cracks, and potholes, with 40 images for each category; The system automatically generates a corresponding edge texture map for each image and uses data augmentation methods to amplify the samples; The processed image is then fed into a two-branch model with a convolutional network as the backbone for training. Experimental results show that the model performs excellently on the test set, with training and testing accuracy rapidly improving in the early stages of training and eventually stabilizing at around 99%, as shown in the attached figure. Figure 5 As shown in the attached figure; simultaneously, both the training loss and the testing loss continuously decrease and tend to converge with the increase of the number of iterations, as shown in the attached figure. Figure 6 As shown.

[0035] The system can ultimately output the recognition results and intuitively display the predicted probabilities and recognition status of each category in the form of confusion matrices, as shown in the appendix. Figure 7As shown in the diagram, the horizontal axis of the confusion matrix represents the defect category predicted by the model, and the vertical axis represents the actual defect category of the sample. The values ​​on the diagonal of the matrix are all 100%, indicating that the model achieves 100% accuracy in identifying the three floor conditions—"healthy," "cracked," and "pothole"—on the test set, with no class confusion or misclassification. This clearly and quantitatively verifies the superior classification performance and generalization ability of this method.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for identifying floor defects based on a dual attention mechanism, characterized in that: Step 1: Data preparation and preprocessing. Collect ground sample images, perform edge texture processing on them to generate corresponding edge texture images; pair the original RGB images with the edge texture images to construct a labeled dataset, and divide it into training and test sets. Step 2: Model construction. Construct a recognition model containing two parallel feature extractors and one feature classifier. Each feature extractor includes a convolutional neural network and a channel attention module, used for feature extraction and semantic enhancement of input images of different modalities. The feature classifier includes a multimodal attention module, a feature fusion layer, and a classification output layer, used for weighted fusion and classification decision of dual-branch features. Step 3: Model training and optimization, using the training set to perform end-to-end training on the recognition model; During training, the error between the model output and the true label is calculated using the cross-entropy loss function. The model parameters are optimized through backpropagation until the model reaches the preset conditions, and the optimal model weights are saved. Step 4: Model evaluation. The model is evaluated using the test set and the saved optimal model weights. The accuracy and loss rate performance metrics are output, and a confusion matrix is ​​generated for error analysis. Step 5: Defect recognition application. For the floor image to be identified, generate its edge texture map according to the method in Step 1. Input the original RGB image and edge texture map into the model trained in Step 3, and realize the classification and recognition of floor defects based on the model output.

2. The method for identifying floor defects based on a dual attention mechanism according to claim 1, characterized in that: In step 1, an industrial camera is used to acquire the original RGB ground image. After converting it into a grayscale image, the Sobel gradient magnitude and Laplacian response are calculated. The two are then linearly fused according to empirical coefficients to generate an initial edge texture map. After normalization, an edge texture map with the same size as the original RGB image and a pixel value range of [0,1] is obtained. All images are size-normalized, and data is augmented using rotation, inversion, and random cropping. The original RGB image and the corresponding edge texture image are used as paired samples and divided into training set and test set according to a preset ratio, while maintaining the pairing relationship between the two types of images during the division.

3. The method for identifying floor defects based on a dual attention mechanism according to claim 1, characterized in that: Two parallel feature extractors take the original RGB image and the corresponding edge texture image as input, respectively. Each feature extractor's convolutional neural network extracts depth features from the input image, and its subsequent channel attention module performs adaptive weight calibration on the channel dimensions of these depth features. The multimodal attention module of the feature classifier evaluates the importance of the features output by the two feature extractors and generates weights. The feature fusion layer multiplies the two feature vectors by their corresponding weights and then concatenates them to form a unified fused feature representation. In the classification output layer, the fused features output the predicted probability distribution of each defect category through a Softmax function, generating the defect category identification result.

4. The method for identifying floor defects based on a dual attention mechanism according to claim 1, characterized in that: In step 3, the recognition model is trained end-to-end using the training set as input. The prediction result is obtained through forward propagation, and the cross-entropy loss between the prediction result and the real label is calculated. Based on the cross-entropy loss, the model parameters are updated through the backpropagation algorithm. Repeat this process until the model reaches the maximum number of iterations, and save the optimal model weights during training.

5. The method for identifying floor defects based on a dual attention mechanism according to claim 1, characterized in that: In step 4, the optimal model weights saved in step 3 are loaded to initialize the recognition model; the model is evaluated using a test set, and the accuracy and loss performance metrics are calculated and output; a multi-class confusion matrix is ​​generated to analyze the model's recognition performance and error distribution on different defect categories.

6. The method for identifying floor defects based on a dual attention mechanism according to claim 1, characterized in that: In step 5, the RGB image of the ground surface to be identified is used to generate the corresponding edge texture image according to the method described in step 1; the original RGB image and the edge texture image are used as a pair of inputs and input into the recognition model trained in step 3. Obtain the defect category classification results output by the recognition model to complete the automatic classification and recognition of floor defects.

Images