A structural multi-disease recognition method based on a lightweight fully convolutional residual network

Abstract

The application relates to a structural multi-disease recognition method based on a lightweight full convolution residual network, which comprises the following steps: acquiring single-disease images, setting the image size, marking each image sample, and generating a single-disease data set; pre-processing the images in the single-disease data set, generating composite disease sample images of multiple diseases coexisting through cross aliasing, and constructing a multi-disease data set; building a full convolution residual network model, setting hyperparameters, and using an Adam optimization algorithm and a classification cross-entropy loss function to train the network; using the trained full convolution residual network to recognize structural diseases, and realizing fast and accurate recognition of multiple diseases at the same time. Compared with the prior art, the application can efficiently recognize various diseases under the condition that multiple diseases coexist, overcomes the limitation that traditional methods can only recognize single diseases, and has remarkable practical value and good application effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of structural health monitoring and computer deep learning, and particularly relates to a structural multi-disease identification method based on a light full convolution residual network. BACKGROUND

[0002] In large-scale engineering structures such as highways, bridges, high-rise buildings, airport runways, tunnels and other buildings, due to climate, environment, vehicle collision and use, etc., various damages, cracks, corrosion, peeling and other surface diseases often occur on the surface of the structures. The existence of surface diseases will lead to a decrease in the strength, stiffness, durability and other properties of the structure, thereby increasing the risk of accidents of the structure. At the same time, maintaining and repairing surface diseases is also a huge workload and cost, so it is of great significance to discover and deal with surface diseases as soon as possible. Structural surface disease identification is to identify and monitor the occurrence and development of surface diseases, and to obtain effective information and repair in time.

[0003] Traditional structural disease identification methods mainly rely on visual inspection, that is, observing the structure surface by naked eye to identify diseases. Although this method is direct, it is low in efficiency and needs to consume a large amount of time and human resources. With the progress of sensor technology, electronic measurement and image processing and other scientific technologies, structural disease detection technology has been significantly improved, especially in the use of neural network technology to process images, which can significantly improve the identification accuracy of structural disease pictures.

[0004] However, the existing technology has some limitations in structural disease identification. First, these technologies mainly identify single diseases, and lack of in-depth research and effective identification methods for multiple diseases occurring on the structure at the same time, i.e. structural multi-disease. Secondly, the sample of structural multi-disease is relatively small, especially the composite disease, i.e. multiple diseases existing in the same structure area at the same time (for example, both honeycomb disease and cracking exist on the surface of concrete), which makes the traditional method have obvious deficiency in identifying composite disease. In addition, due to the less coexistence of structural multi-disease, it is difficult to obtain related data sets, and the current research and application are mostly concentrated on single disease data sets, which limits the development and application of multi-disease identification technology. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a structural multi-disease identification method based on a light full convolution residual network, which can efficiently identify various diseases in the case of coexistence of multiple diseases, overcome the limitation of traditional methods that can only identify single diseases, and has significant practical value and good application effect.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] The application provides a structural multi-disease identification method based on a lightweight full convolution residual network, comprising the following steps:

[0008] S1: single-disease dataset construction and labeling: obtain single-disease images and set the image size, label each image sample, and form a single-disease dataset;

[0009] S2: multi-disease dataset construction and labeling: preprocess the single-disease images in the single-disease dataset, generate composite disease sample images coexisting with multiple diseases through cross aliasing, and form a multi-disease dataset;

[0010] S3: build a full convolution residual network model, including convolution layers, pooling layers, classification layers, full quality attention blocks, and point-by-point feature selection blocks, set hyperparameters based on the multi-disease dataset, and use the Adam optimization algorithm and the classification cross-entropy loss function for network training;

[0011] S4: perform structural multi-disease identification based on the trained full convolution residual network to realize fast and accurate identification of structural multi-disease.

[0012] Further, in S1, the specific process includes:

[0013] Obtain single-disease images and set the image sample size;

[0014] Label each image sample, including: label the structure surface as 0, label the structure surface as 1, label the structure surface as 2, label the structure as 3, etc., a total of N, and sequentially obtain the result Y corresponding to the input X, wherein each element y_i of Y corresponds to the sample x_i in X, and the value range of y_i is 0 to N-1, and m is the number of samples.

[0015] Further, in S2, the preprocessing includes at least one of scaling, stretching, flipping, rotating, cropping, and adding noise to the single-disease images.

[0016] Further, in S2, in the cross aliasing process, a random linear mixing method is used to obtain a multi-disease network dataset and its label Specifically including:

[0017] Randomly select u single-disease images from the m single-disease images in the single-disease dataset for construction;

[0018] Linearly mix the u single-disease images, and combine the labels of the u single-disease images to obtain a new label:

[0019] Repeat the above process m times to obtain the final and

[0020] Further, in S3, the process of building the full convolutional residual network includes the construction of a convolutional block, specifically including:

[0021] Set the input multi-disease image data format as (C0, H0, W0), and define the multi-disease image output data format as (C, H, W);

[0022] Perform two-dimensional convolution, batch normalization, ReLU activation function, and again two-dimensional convolution and batch normalization operations on a part of the input multi-disease image data;

[0023] Perform only two-dimensional convolution and batch normalization operations on another part of the input multi-disease image data;

[0024] Superimpose the results of the two parts, and perform the same convolution and batch normalization operations again to obtain the convolutional block output.

[0025] Further, in S3, the process of building the full convolutional residual network includes the construction of a point-by-point feature selection block, specifically including:

[0026] Analyze the input feature map, covering the channel number, height, and width of the multi-disease image feature;

[0027] Divide the feature map into three subgroups based on the channel, retain the features in the highest and middle subgroups, and discard the features in the lowest subgroup;

[0028] Perform point-by-point averaging operations on the remaining two subgroups to compress them into an independent subgroup, keeping the spatial dimensions of the feature map unchanged.

[0029] Further, in S3, the process of building the full convolutional residual network includes the construction of a full-quality attention block, specifically including:

[0030] In the channel attention path, generate a channel attention map through a skip layer and a bottleneck structure, where the bottleneck structure contains two dense layers activated by ReLU and sigmoid activation functions, respectively;

[0031] In the spatial attention path, calculate the spatial descriptor along the channel axis, and generate a spatial attention map using convolutional layers and transposed convolutional layers;

[0032] Element-wise multiply the channel and spatial attention maps with the input feature map to achieve importance weighting of the features.

[0033] Further, in S3, the process of building the full convolutional residual network also includes the construction of the FCResNet neural network framework, specifically including:

[0034] Perform two-dimensional convolution, batch normalization, ReLU activation function, and maximum pooling operations on the input of the training network;

[0035] The network structure comprising 4 convolutional blocks is constructed.

[0036] A two-dimensional adaptive average pooling layer, a two-dimensional convolution, a batch normalization, and a Softmax classification layer are added after the 4 convolutional blocks.

[0037] The Softmax classification layer is defined to calculate the probability distribution.

[0038] Further, in S3, the expression of the max-pooling operation is:

[0039]

[0040] wherein is the jth channel of the lth layer; P i l is the ith channel of the pooling operator in the lth layer, I is the number of channels of the l-1th and lth layers, and the plane size of the output feature map is calculated according to the following expression:

[0041] D out =(D in -P pool ) / S pool +1

[0042] wherein P pool is the side length of the pooling operation, S pool is the stride, D in is the input of the feature map of the pooling layer, D out is the output of the feature map of the pooling layer, and the max-pooling is the maximum value of the calculation region.

[0043] The two-dimensional adaptive average pooling layer, the two-dimensional convolution, the batch normalization, and the Softmax classification layer are added after the 4 convolutional blocks.

[0044] The Softmax classification layer is defined as:

[0045]

[0046] wherein, and 0≤softmax(x) i ≤1, softmax(x) i is a probability, k represents the number of categories, and xi represents the input.

[0047] Further, in S3, the definition of the classification cross-entropy loss function is as follows:

[0048]

[0049] wherein, the vector is made to be the yth(i) 1, and the rest are 0; is the predicted probability distribution; m represents the sample number of the training data set, and k represents the number of categories; and respectively y (i) and is an element that is 1 or 0; and represents the model parameters.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] 1) The present application successfully simulates and constructs a complex disease sample through an innovative cross-over technology. This technology superimposes multiple single disease pictures through image processing means to form a simulated complex disease image. The present application effectively solves the problem of the scarcity of complex disease samples, provides sufficient training data for deep learning models, and thus improves the accuracy and generalization ability of the model in identifying complex diseases.

[0052] 2) The FCResNet network of the present application can identify and classify multiple structural diseases including single disease and complex disease. Through the setting of different layers of full convolution residual network, the present application not only improves the accuracy of the traditional residual network, but also optimizes the identification of complex diseases, so that the model can accurately identify multiple diseases occurring simultaneously in structure in practical application, and significantly improves the efficiency and accuracy of structural disease identification.

[0053] 3) The method for constructing a multi-disease data set using image splicing technology proposed by the present application provides a new data source for the training of deep learning models. The present application enhances the model's ability to identify complex disease scenarios by simulating the actual possible complex disease conditions.

[0054] 4) The present application also has the advantages of not being easily disturbed, high efficiency, reducing a large amount of manpower, and improving work efficiency; at the same time, the present method can realize the simultaneous identification of structural multiple diseases; in addition, the present method proposes a method for constructing a multi-disease data set using image splicing technology, which solves the problem of insufficient multi-disease data sets for training. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a structural disease identification method framework based on residual full convolutional neural network (FCResNet);

[0056] Figure 2 is a convolutional block architecture diagram;

[0057] Figure 3 is a point-by-point feature selection block architecture diagram;

[0058] Figure 4 is a full-quality attention block architecture diagram;

[0059] Figure 5 is a residual full convolutional neural network architecture diagram. DETAILED DESCRIPTION

[0060] Overall, the present application discloses a structural multi-disease identification method based on residual full convolutional neural network (FCResNet). The method introduces a multi-scale feature fusion lightweight attention network to solve the challenge of high recognition calculation complexity, and introduces a full quality attention block and a point-by-point feature selection block to identify multi-disease features. The method first labels the directly collected structural surface picture dataset, then expands and enhances the multi-disease samples through image cross aliasing technology, and restructures the network dataset and labels it, then divides the new picture dataset into training set and test set, and then builds FCResNet network for structural multi-disease identification. The training and test set of the method is used to train and verify the accuracy of the network. The method is based on a supervised deep learning network, including convolutional layer, pooling layer, classification layer, full quality attention block and point-by-point feature selection block. The artificial labeled abnormal classification is compared with the prediction output of the test set data of the FCResNet network to judge the excellence of the FCReNet network, and finally the model with good training accuracy can be saved for prediction data and classification. The method can realize fast and accurate identification of structural multi-disease, and overcome the limitation of the prior art that only single disease identification can be performed, and good results have been achieved.

[0061] The present application will be described in detail below in combination with the drawings and specific embodiments. In this technical solution, if the component model, material name, connection structure, control method, algorithm and other features are not explicitly stated, they are considered as common technical features disclosed in the prior art.

[0062] Embodiment 1

[0063] In this embodiment, the structural disease identification method based on full convolution residual network is as shown in the flowchart Figure 1 , including the following steps:

[0064] 1) dataset construction;

[0065] 2) dataset label marking;

[0066] 3) build FCResNet network and set hyperparameters for training;

[0067] 4) structural disease identification based on FCResNet network.

[0068] The process of step 1) is as follows:

[0069] To acquire a single lesion image, set the image size. Let the input data format X be (m,c,h,w), where m represents the number of samples (i.e., the number of input images), c represents the number of channels, h represents the image height, and w represents the image width. Assuming the image length and width are both h, then the data format of X is (m,c,h,h).

[0070] Each single-disease image sample is labeled, assuming there are N types: 0 for intact surface, 1 for surface chipping, 2 for honeycomb-like structures, 3 for cracked structures, and so on, totaling N types. The result Y corresponding to input X is obtained sequentially.

[0071] Y = [y1, y2, ... y i ,…y m ] Τ (1)

[0072] Among them, y i ∈{0,1,…,N-1}, where m is the number of samples.

[0073] The process of step 2) is as follows:

[0074] The original single-disease image dataset obtained in step 1) is used to obtain a multi-disease network dataset by employing a random linear mixture method. and its labels The specific process is as follows:

[0075] ① Randomly select u images from m single-disease images to construct the image. For ease of calculation, u is taken as a square number.

[0076] ② Linearly blend the u individual disease images. The following example uses u=9:

[0077]

[0078] ③ Combine the u individual disease image labels to obtain new labels:

[0079] y new =[y1,y2,…,y9] (3)

[0080] ④ Repeat the above process m times to obtain and The data format is (m,c,3h,3h), and its corresponding label is...

[0081]

[0082] Among them, y i_new The calculation method is shown in process ③.

[0083] The process of step 3) is as follows:

[0084] 3.1 Convolution block construction

[0085] Referring to Figure 2 , a convolution module is constructed, assuming that the input data format of the multi-disease image is (C0, H0, W0), and the output data format of the multi-disease image is (C, H, W), wherein the input and output formats can be the same or different. One part is subjected to two-dimensional convolution, batch normalization, ReLU activation function, two-dimensional convolution and batch normalization operation, and the other part is subjected to two-dimensional convolution and batch normalization operation, the results of the two parts are superimposed, and then the above steps are performed to obtain the output of the convolution block.

[0086] The convolution operation is as follows:

[0087]

[0088] wherein is the jth channel of the lth layer, is the ith channel of the filter j in the lth layer, is the bias of the filter j of the lth layer, and I and J are the number of channels of the l-1th and lth layers.

[0089] When performing convolution operation on an image, the filter uses the input receptive field to perform element-wise multiplication, sums the products in each channel, and then advances with a specified stride, and so on. The size of the output feature map in the plane is calculated by the following formula:

[0090] D out =(D in -C) / S conv +1 (6)

[0091] wherein D in is the side length of the input layer x l-1 , and it is assumed herein that the input layer is a square with a size of D in ×D in in each channel; C is the side length of the filter, and it is assumed herein that each filter is a square with a size of C×C; D out is the side length of the output layer x l ; S conv is the stride, and it is assumed herein that the stride along the height and width directions is S conv .

[0092] The batch normalization operation is to standardize the input data, i.e., to make the sample mean 0 and the standard deviation 1.

[0093] The ReLU activation function has the expression:

[0094]

[0095] 3.2 Point-wise feature selection block construction

[0096] Existing methods generally adopt direct fusion of multi-scale features and assign uniform weights to these features. However, this direct fusion feature extraction method often contains redundant information, which produces a blurring effect in the presentation of key details. Especially in complex real-world environments, due to the presence of a large number of redundant details in the background area, this problem is further exacerbated. Such redundant information not only increases the difficulty of structural multi-disaster recognition, but also can significantly reduce the performance of the recognition system.

[0097] The present application introduces a point-wise feature selection block, which reduces redundant information by eliminating weak features of each scale one by one before fusion. This feature selection module effectively reduces the number of parameters, making the system architecture more efficient and compact. In addition, this method can extract more informative features from relatively fewer feature maps, significantly improving the generalization ability of the FCResNet model.

[0098] The working process of the point-wise feature selection block starts with the analysis of the input feature map, covering three dimensions of the number of channels, height, and width of the feature. Then, the feature map is divided into three subgroups based on the channel, and the features in the highest and middle subgroups are retained while the lowest features in all subgroups are discarded. Next, the remaining two subgroups are subjected to point-wise averaging operation, compressed into an independent subgroup. In this process, the spatial dimension of the feature map remains unchanged, thereby optimizing the feature representation and improving the performance of the system.

[0099] 3.3 Full-quality attention block construction

[0100] Reference Figure 4 The full-quality attention block applies an attention mechanism by dynamically adjusting the importance of the feature map in the channel and spatial dimensions, thereby significantly improving the accuracy of multi-disaster feature recognition. The input and output processing flow of this module consists of two parallel paths.

[0101] In the upper path, i.e., the channel attention path, the channel attention map is generated through a skip layer and a bottleneck structure. The skip layer first generates channel description information of the feature map, which is then input into the bottleneck structure. The bottleneck structure contains two dense layers, activated by ReLU and sigmoid activation functions, respectively. The first dense layer reduces the channel dimension to one-fourth of the original input, while the second dense layer restores to the original dimension, similar to the self-attention mechanism, which precisely focuses on important channel information.

[0102] The lower-level path, the spatial attention path, uses the same process to generate the spatial attention map. First, the spatial descriptor along the channel axis is calculated, which is done by taking the average of the input feature map through a point-by-point jump operation. Next, two convolutional layers with a stride of 2 are used to progressively reduce the spatial dimension, and then two transposed convolutional layers are used to restore it to the original spatial size. In the final stage, the channel and spatial attention maps are multiplied element-wise with the input feature map to weight the importance of the features. Through this dual-path, hierarchical attention mechanism, the full-quality attention block enhances the model's ability to capture important features, enabling it to perform exceptionally well in complex tasks involving the identification of multiple disease features.

[0103] 3.4 Construction of the FCResNet Neural Network Framework

[0104] See Figure 5 The model structure of the FCResNet neural network is constructed. The network input is first subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The network structure then includes four convolutional modules, each of which is shown in Figure 3.1. The expression for the pooling operation is as follows:

[0105]

[0106] in It is the j-th channel of the l-th layer; P i l is the i-th channel of the pooling operator in layer l; I is the number of channels in layers l-1 and l. The expression for calculating the in-plane size of the output feature map is as follows:

[0107] D out =(D in -P pool ) / S pool +1 (9)

[0108] Where P pool It is the side length of the pooling operation (assuming the operator is P). pool ×P pool (square), S pool It is stride length (both height and width are S) pool ), D in D is the feature map input for the pooling layer. out This is the feature map output of the pooling layer. Max pooling calculates the maximum value in the computational region.

[0109] After four convolutional blocks, there is a two-dimensional adaptive average pooling layer, a two-dimensional convolution, a batch normalization layer, and a softmax classification layer; the softmax classification layer is defined by equation (10):

[0110]

[0111] where, and 0≤softmax(x) i ≤1, softmax(x) i is a probability, k represents the number of classes; x i represents the input;

[0112] The network training uses the Adam optimization algorithm; the loss function of network training uses the classification cross-entropy, which is defined as follows:

[0113]

[0114] where, vector According to the label data, the y (i) th element is 1 and the rest are 0; is the predicted probability distribution; m represents the number of samples in the training data set, and k represents the number of classes; and are the 0-1 elements in y (i) and respectively; Θ represents the model parameters;

[0115] 3.5 Save the trained FCResNet model;

[0116] Test the pros and cons of the model on the test set, and save the model with excellent results.

[0117] The process of step 4) is as follows:

[0118] The multi-disease data set and label obtained in step 2) are input into the FCResNet model constructed in step 3) for prediction. The specific process is as follows:

[0119] ① Extract each sample x i and the corresponding label y i from the data set;

[0120] ② The width and height of the x i image sample are 3h, which are much larger than the model input width and height h. Cut the sample by h to obtain 9 prediction samples, and the prediction input format is (9, 3, h, h):

[0121] ③ Input the prediction sample into the model to obtain the prediction result where, is the prediction result,

[0122] ④ Obtain the multi-disease recognition result according to the prediction result.

[0123] The foregoing description of the embodiments has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed. Modifications and variations are possible in light of the above teachings or can be acquired from practice of the application. As well, the description is presented in the context of the preferred embodiments as a number of alternatives. It is not intended to limit the application to the precise form described.

Claims

1. A method for structural multi-disease recognition based on a lightweight fully convolutional residual network, characterized in that, Comprise the following steps: S1: single disease data set construction and labeling: obtain single disease image and set image size, label each image sample, form single disease data set; S2: multi-disease data set construction and labeling: by preprocessing single disease image in single disease data set, by cross aliasing, generate composite disease sample image of multiple diseases coexisting, form multi-disease data set; S3: build full convolution residual network model, including convolution layer, pooling layer, classification layer, full quality attention block, point-by-point feature selection block, based on the multi-disease data set, set hyperparameters, and use Adam optimization algorithm and classification cross entropy loss function for network training; S4: structure multi-disease recognition based on trained full convolution residual network; In S3, the specific process of building full convolution residual network includes: Perform two-dimensional convolution, batch normalization, ReLU activation function, maximum pooling operation on the input of the training network; Construct a network structure containing 4 convolution blocks; Add two-dimensional adaptive average pooling layer, two-dimensional convolution, batch normalization and Softmax classification layer after the 4 convolution blocks; Define the Softmax classification layer to calculate the probability distribution; In S3, the process of building full convolution residual network includes the construction of convolution block, which specifically includes: Set the input multi-disease image data format as (C0, H0, W0), and define the output multi-disease image data format as (C, H, W); Perform two-dimensional convolution, batch normalization, ReLU activation function, two-dimensional convolution and batch normalization operation on part of the input multi-disease image data again; Perform two-dimensional convolution and batch normalization operation on another part of the input multi-disease image data; Superimpose the results of the two parts, and perform the same convolution and batch normalization operation again to obtain the output of the convolution block; In S3, the process of building full convolution residual network includes the construction of point-by-point feature selection block, which specifically includes: Analyze the input multi-disease image feature mapping, covering the channel number, height and width of the multi-disease image feature; Divide the multi-disease feature map into three subgroups based on the channel, retain the features in the highest and middle subgroups, and discard the features in the lowest subgroup; Perform point-by-point average operation on the remaining two subgroups to compress them into an independent subgroup, keeping the spatial dimension of the feature map unchanged; In S3, the process of building full convolution residual network includes the construction of full quality attention block, which specifically includes: In the channel attention path, generate a channel attention map through a skip layer and a bottleneck structure, wherein the bottleneck structure contains two dense layers activated by ReLU and sigmoid activation functions respectively; In the spatial attention path, calculate the spatial descriptor along the channel axis, and generate a spatial attention map using convolution layer and transposed convolution layer; Multiply the channel and spatial attention maps with the input feature map element by element to weight the importance of the features. 2.The structural multi-disease recognition method based on the lightweight full convolution residual network according to claim 1, characterized in that, In S1, the specific process includes: Obtain single disease image, set image sample size; Labeling each single-disease image sample includes: structural surface complete labeling as 0, structural surface block labeling as 1, structural surface honeycomb appearance labeling as 2, structural cracking labeling as 3, a total of N kinds, sequentially obtaining the result corresponding to the input X as Y, wherein each element y_i of Y corresponds to a sample x_i in X, and the value range of y_i is 0 to N-1, m is the number of samples. 3.The structural multi-disease recognition method based on the lightweight full convolution residual network according to claim 1, characterized in that, In S2, the preprocessing includes at least one of scaling, stretching, flipping, rotating, cropping and adding noise on the single-disease image.

4. The structural multi-disease recognition method based on a lightweight fully convolutional residual network according to claim 1, characterized in that, In S2, in the process of cross aliasing, a random linear mixing method is used to obtain a multi-disease network data set and tags thereof , specifically comprising: Randomly selecting u single-disease images from the m single-disease images of the single-disease data set for construction; Linearly mixing the u single-disease images, and combining the labels of the u single-disease images to obtain a new label: The above process is repeated m times to obtain the final and .

5. The structural multi-disease recognition method based on the lightweight full convolution residual network according to claim 1, characterized in that, In S3, the expression of the max-pooling operation is: wherein is the number of channels of the first layer, is the number of channels of the first layer, is the number of channels of the first layer, is the number of channels of the layer and the in-plane dimension of the output feature map is calculated as follows: wherein is the side length of the pooling operation, is the stride, is the input of the feature map of the pooling layer, is the output of the feature map of the pooling layer, the max pooling is the maximum value of the calculation region; After the four convolutional blocks, there are a two-dimensional adaptive average pooling layer, a two-dimensional convolution, a batch normalization and a Softmax classification layer. The Softmax classification layer is defined as: wherein , and , is a probability, k represents the number of classes; represents an input. 6.The structural multi-disease recognition method based on the lightweight full convolution residual network according to claim 1, characterized in that, In S3, the definition of the classification cross-entropy loss function is as follows: where the vector is made to be 1 at its th element and 0 elsewhere according to the label data; is the predicted probability distribution; m represents the number of samples in the training dataset, denotes the number of classes; and are the and non-zero elements in respectively, and denotes the model parameters.

Images