Bridge disease identification method and device, storage medium and equipment

Abstract

The invention discloses a bridge disease identification method and device, a storage medium and equipment, and belongs to the technical field of bridge disease identification in target monitoring, and the method comprises the steps: obtaining a bridge image; inputting the bridge image into a trained bridge disease recognition model to obtain a bridge disease recognition image, and marking diseases and categories thereof included in the bridge image in the bridge disease recognition image; the bridge disease recognition model is obtained through improvement on the basis of a bridge disease semantic segmentation basic model based on an encoder-decoder, and the improvement comprises the following steps: adding a multi-scale feature comparison mechanism in the bridge disease semantic segmentation basic model, and changing convolution mechanisms in the encoder and the decoder into a mapping convolution mechanism; according to the method, the accuracy of bridge disease target identification is improved through a multi-scale feature comparison mechanism, and the real-time performance of the model on bridge disease identification is effectively improved through a mapping convolution mechanism.

Description

Technical Field

[0001] The invention relates to a bridge defect recognition method, device, storage medium and equipment, belonging to the technical field of bridge defect recognition in target monitoring. Background Art

[0002] Bridges are an important part of the transportation field. When bridges are used for a long time, their structural characteristics and usage characteristics determine that they are more likely to have cracks, leaks, damage and other diseases. Especially under the influence of extreme weather and overloaded and overweight vehicles, bridges are prone to bridge pier damage, guardrail damage, expansion joint blockage and bridge surface water and other bridge disease events. In order to effectively avoid the expansion of bridge diseases, daily inspections of bridges are required. In terms of daily inspections of bridges, due to the diversity of bridge diseases and the complexity of the bridge environment, the existing UAV-based bridge disease recognition method is easily disturbed by the surrounding environment of the bridge disease target, and cannot effectively distinguish the disease target from the surrounding environment, resulting in errors, omissions, and incompleteness in the recognition results of the bridge disease target, that is, the recognition accuracy is not high; secondly, the existing UAV-based bridge disease recognition method has a large model architecture and repeated convolution, resulting in a large amount of model calculation, low real-time performance of bridge disease recognition, and cannot meet the requirements of real-time recognition of bridge diseases in actual engineering applications. Summary of the invention

[0003] The purpose of the present invention is to provide a bridge defect identification method, device, storage medium and equipment to solve the problems of low accuracy and low real-time performance of bridge defect identification in the prior art.

[0004] To achieve the above objectives, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a bridge disease identification method, comprising: Get bridge images; Inputting the bridge image into the trained bridge disease recognition model to obtain a bridge disease recognition image, wherein the bridge disease recognition image is labeled with the diseases contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the basic model of bridge disease semantic segmentation based on encoder-decoder, and the improvement includes: adding a multi-scale feature comparison mechanism to the basic model of bridge disease semantic segmentation, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

[0005] Furthermore, the multi-scale feature comparison mechanism performs the following operations on the feature matrix in the bridge defect recognition model: On the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain an optimal feature matrix that best represents the convolution layer, wherein the feature matrix is ​​obtained by extracting features of the bridge image through each channel in the convolution layer of the encoder; The feature matrix fusion operation is performed on convolutional layers of all scales to obtain the optimal feature matrices corresponding to convolutional layers of all scales, and all the optimal feature matrices are merged to form a multi-scale feature pyramid; Performing a cross-scale reduction operation on the multi-scale feature pyramid to obtain contrast information of the bridge disease target and surrounding background information, thereby generating a multi-scale feature contrast matrix; The multi-scale feature contrast matrix is ​​combined with the semantic segmentation features obtained from the original bridge defect semantic segmentation basic model; The fused semantic segmentation features are used to classify whether they belong to a defect target through the classifier in the original bridge defect semantic segmentation basic model.

[0006] Furthermore, on the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain an optimal feature matrix that best represents the convolution layer, including: Get the feature matrices of any two different channels in the convolutional layer from the set of all feature matrices of the convolutional layer and , we can obtain the value of and The two vector matrices with the largest correlation and : (1); in, T represents transpose, argmax represents the argmax function, for and The correlation coefficient between Represents the convolutional layer j Middle Channel i The characteristic matrix of Represents the convolutional layer j Middle Channel i+ The feature matrix of 1, express The corresponding vector matrix is, express The corresponding vector matrix; By formula (2) and pass and Two vector matrices are fused: (2); in Represents the feature matrix after a single fusion; The fused feature matrix Replace the original convolutional layer with all feature matrices in the collection and , and form a new feature set with the remaining feature matrices in the set : (3); exist Re-acquire and As the characteristic matrix in the above formula (1) and , perform feature fusion through formula (1), formula (2), and formula (3), and generate the next new feature set , and the calculation is repeated until all channels are cycled through to obtain the fusion that best represents the convolution layer. n The optimal feature matrix after , n Indicates the number of channels of the convolutional layer.

[0007] Furthermore, the operations performed by the mapping convolution mechanism on the input of the encoder / decoder include: Use half the number of convolution kernels in the original bridge defect semantic segmentation basic model to perform traditional convolution on the encoder / decoder input to obtain a traditional convolution feature map; The traditional convolution feature map is extracted through ordinary convolution to generate an ordinary convolution feature map; The traditional convolution feature map and the ordinary convolution feature map are spliced ​​together to obtain the final feature map; After the input of the encoder / decoder is processed by the mapping convolution mechanism to obtain the final feature map, the encoder / decoder performs subsequent processing on the obtained final feature map.

[0008] Furthermore, the training of the bridge defect recognition model includes: Acquire multiple bridge images to construct a bridge inspection sample set, and pre-process the bridge inspection sample set; Annotate the bridge images in the bridge inspection sample set at the pixel level for bridge defects using an image calibration tool to obtain a bridge defect dataset, and divide the bridge defect dataset into a training set, a validation set, and a test set according to a preset ratio; The bridge defect recognition model is trained using the training set, the training result of the bridge defect recognition model is adjusted using the validation set, and the training effect of the bridge defect recognition model is verified using the test set until the recognition accuracy of the bridge defect recognition model reaches the preset requirement.

[0009] Furthermore, the preprocessing includes: rotating, flipping and / or adding noise to the bridge data in the bridge inspection sample set.

[0010] In a second aspect, the present invention provides a bridge defect identification device, comprising: The bridge image acquisition module is configured to: acquire a bridge image; The bridge defect recognition module is configured to: input the bridge image into the trained bridge defect recognition model to obtain a bridge defect recognition image, wherein the bridge defect recognition image is labeled with the defects contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the basic model of bridge disease semantic segmentation based on encoder-decoder, and the improvement includes: adding a multi-scale feature comparison mechanism to the basic model of bridge disease semantic segmentation, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

[0011] Furthermore, in the bridge defect identification module, the multi-scale feature comparison mechanism performs the following operations on the feature matrix in the bridge defect identification model: On the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain an optimal feature matrix that best represents the convolution layer, wherein the feature matrix is ​​obtained by extracting features from the bridge image by the encoder; The feature matrix fusion operation is performed on convolutional layers of all scales to obtain the optimal feature matrices corresponding to convolutional layers of all scales, and all the optimal feature matrices are merged to form a multi-scale feature pyramid; Performing a cross-scale reduction operation on the multi-scale feature pyramid to obtain contrast information of the bridge disease target and surrounding background information, thereby generating a multi-scale feature contrast matrix; The multi-scale feature contrast matrix is ​​combined with the semantic segmentation features obtained from the original bridge defect semantic segmentation basic model; The fused semantic segmentation features are used to classify whether they belong to a defect target through the classifier in the original bridge defect semantic segmentation basic model.

[0012] Furthermore, in the bridge defect recognition module, the operations performed by the mapping convolution mechanism on the input of the encoder / decoder include: Use half the number of convolution kernels in the original bridge defect semantic segmentation basic model to perform traditional convolution on the encoder / decoder input to obtain a traditional convolution feature map; The traditional convolution feature map is extracted through ordinary convolution to generate an ordinary convolution feature map; The traditional convolution feature map and the ordinary convolution feature map are spliced ​​together to obtain the final feature map; After the input of the encoder / decoder is processed by the mapping convolution mechanism to obtain the final feature map, the encoder / decoder performs subsequent processing on the obtained final feature map.

[0013] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the bridge defect identification method described in any one of the first aspects.

[0014] In a fourth aspect, the present invention provides a computer device, comprising: Memory, for storing computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of the bridge defect identification method described in any one of the first aspects.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are: The present invention provides a bridge defect recognition method, device, storage medium and equipment. A multi-scale feature comparison mechanism is constructed in a basic model to perform cross-scale reduction operations on the multi-scale features of bridge defect targets, highlighting the comparison between different scale features of different convolution layers to enhance the comparison between image bridge defect targets and surrounding background information, thereby improving the accuracy of bridge defect target recognition. At the same time, a mapping convolution mechanism is constructed in the basic model to directly map similarity features between different convolutions to avoid the repeatability of convolution, reduce the calculation amount of the model, and effectively improve the real-time performance of the model in bridge defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a bridge defect identification method provided by an embodiment of the present invention; Figure 2 is a schematic diagram of the architecture of a bridge defect identification model provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a multi-scale feature pyramid provided by an embodiment of the present invention; Figure 4 is a schematic diagram of feature similarity between different channels of the second convolutional layer in the bridge defect recognition model provided by an embodiment of the present invention; Figure 5is a schematic diagram of a full traditional convolution provided by an embodiment of the present invention; Figure 6 It is a schematic diagram of the mapping convolution provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0018] Example 1.

[0019] like Figure 1 As shown, the present invention provides a bridge disease identification method, comprising the following steps: Get bridge images; The bridge image is input into the trained bridge defect recognition model to obtain a bridge defect recognition image, wherein the bridge defect recognition model is improved on the basis of the bridge defect semantic segmentation basic model based on encoder-decoder. The improvement includes: adding a multi-scale feature comparison mechanism to the bridge defect semantic segmentation basic model, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism; The bridge defect recognition image annotates the defects contained in the bridge image and their categories, thereby obtaining the bridge defect recognition result.

[0020] In order to effectively solve the shortcomings of the current bridge defect recognition technology, the present invention proposes a bridge defect recognition method. By constructing a multi-scale feature comparison mechanism in the basic model, a cross-scale reduction operation is performed on the multi-scale features of the bridge defect target, and the contrast between different scale features of different convolution layers is highlighted to enhance the contrast between the image bridge defect target and the surrounding background information, thereby improving the accuracy of bridge defect target recognition. At the same time, a mapping convolution mechanism is constructed in the basic model to directly map the similarity features between different convolutions to avoid the repeatability of convolution, reduce the calculation amount of the model, and effectively improve the real-time performance of the model in bridge defect recognition.

[0021] Example 2.

[0022] The embodiment of the present invention provides a bridge disease identification method, comprising the following steps: Step S1: Acquire bridge images collected by drones to form a bridge image sample set, and pre-process the bridge image sample set.

[0023] Step S11, completing the data collection of the bridge by using the drone, and obtaining the bridge image data by using the real-time transmission interface; Step S12, by means of rotation, flipping, adding noise, etc., the diversity of the data is enhanced to avoid the problem of data skew, so as to improve the robustness of subsequent training.

[0024] Step S2: Use an image calibration tool to annotate bridge images at the pixel level for bridge defects to form a bridge defect dataset, and divide the bridge defect dataset into a training set, a validation set, and a test set according to a set ratio.

[0025] Step S21, using an image calibration tool to manually mark bridge disease targets on the captured image to obtain bridge disease labels; Step S22, divide the data set into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0026] Step S3: Build a basic model for bridge damage semantic segmentation based on encoder-decoder.

[0027] Step S31, building a basic model for semantic segmentation of bridge defects through a convolution-based encoder and decoder, the encoder classifies and analyzes low-level local pixel values ​​of the image based on convolution to obtain high-level semantic information; In step S32, the decoder upsamples the reduced feature image according to the pooling layer index through deconvolution, and assigns different colors to the pixels of the bridge defects on the image to classify the bridge defects.

[0028] Step S4: construct a multi-scale feature comparison mechanism in the basic model of bridge defect semantic segmentation, perform cross-scale subtraction operations on the multi-scale features of the image, and achieve contrast enhancement between the bridge defect target and the surrounding background information, so as to effectively improve the recognition accuracy of the model for bridge defects.

[0029] The operations performed by the multi-scale feature comparison mechanism on the feature matrix in the bridge defect identification model include: On the convolutional layer of the encoder, feature matrices of multiple channels in the convolutional layer are fused through correlation analysis to obtain the optimal feature matrix that best represents the convolutional layer. The feature matrix is ​​obtained by extracting features from the bridge image by the encoder. The feature matrix fusion operation is performed on all scales of convolutional layers to obtain the optimal feature matrices corresponding to all scales of convolutional layers. All optimal feature matrices are merged to form a multi-scale feature pyramid. The structure of the multi-scale feature pyramid is as follows: Figure 3 As shown; Perform cross-scale reduction operations on the multi-scale feature pyramid to obtain the contrast information of the bridge defect target and the surrounding background information, thereby generating a multi-scale feature contrast matrix to better highlight the bridge defect target; The multi-scale feature contrast matrix is ​​fused with the semantic segmentation features obtained in the original bridge defect semantic segmentation basic model, and the multi-scale feature contrast enhancement effect is achieved in the semantic segmentation through fusion with the high-order semantic segmentation features, so as to better highlight the contrast between the bridge defect and the surrounding environment and improve the semantic segmentation recognition accuracy of the bridge defect; The fused semantic segmentation features are used to classify whether they belong to a defect target through the classifier in the original bridge defect semantic segmentation basic model.

[0030] Step S5: Replace the convolution mechanism in the basic model of bridge defect semantic segmentation with a mapping convolution mechanism, perform mapping operations on the convolution features, achieve direct replacement between similar features, avoid repeated convolution operations, reduce the amount of model calculation, and improve the real-time performance of the model in bridge defect recognition.

[0031] The operations performed by the mapped convolution mechanism on the encoder / decoder input include: Use half the number of convolution kernels in the original bridge defect semantic segmentation basic model to perform traditional convolution on the encoder / decoder input to obtain a traditional convolution feature map; The traditional convolution feature map is extracted through ordinary convolution to generate an ordinary convolution feature map; The traditional convolution feature map and the ordinary convolution feature map are spliced ​​together to obtain the final feature map; After the input of the encoder / decoder is processed by the mapping convolution mechanism to obtain the final feature map, the encoder / decoder performs subsequent processing on the obtained final feature map.

[0032] By using half the number of convolution kernels for traditional convolution, the amount of calculation can be reduced by half, and a traditional convolution feature map with half the amount of calculation can be obtained, wherein the traditional convolution is a full combination of convolution-normalization-nonlinear activation, and the amount of calculation is large. Therefore, the present invention uses half the number of traditional convolution kernels; wherein ordinary convolution only performs convolution operation, does not include normalization and nonlinear activation, and the amount of calculation is small; the two sets of feature maps are spliced ​​together, that is, the features of ordinary convolution replace the traditional convolution features, and the same number of feature maps as the full traditional convolution are obtained. Therefore, through this mapping convolution (half traditional convolution + half ordinary convolution) method, while achieving the same number of feature maps as the full traditional convolution, the amount of calculation can be greatly reduced, and the real-time semantic segmentation and recognition of bridge defects can be improved.

[0033] After step S5 is completed, the structure of the bridge damage model is as follows: Figure 2 shown.

[0034] Step S6, load the data set into the improved model for training, use the training set for training, use the validation set to adjust the training results, use the test set to verify the model effect, configure the training parameters, and after the model training is completed, finally output the trained bridge disease recognition model for bridge diseases.

[0035] The specific training parameters are: training iteration cycle is 100, batch number of images is 16, learning rate is 0.01, and optimizer weight decay is 0.0005.

[0036] Step S7: input the bridge image acquired in real time into the trained bridge defect recognition system to complete the recognition of bridge defects.

[0037] The output of the bridge defect target recognition result is represented by the probability that each pixel belongs to the bridge defect category.

[0038] In step S4, multi-scale comparison fuses the features on different channels in the convolution layer through correlation analysis to obtain the most representative feature matrix in this convolution layer: The characteristics of different channels in the convolutional layer are , the specific characteristics of each channel are ,in j Represents different convolutional layers, i represents the different channels in each convolutional layer, For the convolutional layer j Middle Channel n The feature matrix of , so the feature set of all different channels in the convolutional layer is , n is the number of channels, k is the label of the feature set, is the feature set of all different channels in the convolutional layer, then for any two feature matrices: and , hoping to find two vector matrices and Make and The correlation between them is the largest, that is, the fusion problem is transformed into an optimization problem: (1); in, T represents transpose, argmax represents the argmax function, for and The correlation coefficient between Represents the convolutional layer j Middle Channel i The characteristic matrix of Represents the convolutional layer j Middle Channeli+ The feature matrix of 1, express The corresponding vector matrix is, express The corresponding vector matrix; By formula (2), and pass and Two vector matrices are fused: (2); in Represents the feature matrix after a single fusion; The fused feature matrix Replace the original convolutional layer with all feature matrices in the collection and , and form a new feature set with the remaining feature matrices in the set : (3); exist Re-acquire and As the characteristic matrix in the above formula (1) and , perform feature fusion through formula (1), formula (2), and formula (3), and generate the next new feature set , and the calculation is iterated in this loop until all channels are looped through (i.e. i = n ), to get the fusion that best represents the convolutional layer n The optimal feature matrix after , n Indicates the number of channels of the convolutional layer.

[0039] Due to the inconsistency of the feature scales of the convolutional layers, the different scale feature sets obtained by fusion in each convolutional layer are Constructing a multi-scale feature pyramid ,like Figure 3 As shown in the figure, in the multi-scale feature pyramid, the features with larger scales contain more image target details, while the features with smaller scales can better reflect the local image background information due to the downsampling operation of the pooling layer. Therefore, the cross-scale subtraction operation on the multi-scale pyramid can obtain the contrast information of the image target and the surrounding background information, so as to better highlight the bridge disease target. The specific algorithm is as follows: (4); in and Represents different scale features, represents linear interpolation, making features of different scales have the same size, represents the feature matrix point-to-point subtraction operation, that is, the cross-scale subtraction operation, That is, the multi-scale contrast information of the image target and the surrounding background information.

[0040] The multi-scale contrast information of the image target and the surrounding background information is normalized and fused with the semantic information extracted by the basic model of the encoder-decoder. The normalized features are classified through the softmax function to obtain the probability distribution of each pixel belonging to the bridge disease category, thereby realizing the recognition of bridge disease targets: (5); in u Represents the bridge damage category, P represents the probability distribution of bridge damage categories (i.e., identification results), BN represents the feature normalization operation, s Represents the semantic information extracted by the encoder-decoder base model.

[0041] Because traditional convolution may have similar features in multiple channels when extracting features, such as Figure 4 The feature maps of the red, yellow, and blue boxes have strong similarities, which results in a waste of computing resources. Therefore, the present invention reduces the amount of computation of the model while achieving the same number of features as the traditional convolution through the mapping convolution method to improve the real-time performance of the model. The mapping convolution operation is as follows: Figure 6 As shown, we first use traditional convolution (The number of convolution kernels is half of the normal number of convolution kernels), Q To input feature data, obtain the intrinsic feature map : (6); (7); in represents a normal convolution operation, is a lightweight feature map, m is the number of feature maps, G The feature map of the convolution is the output map.

[0042] Compared with full traditional convolution (such as Figure 5 Compared with the traditional convolution, the mapped convolution (half traditional convolution + half ordinary convolution) can achieve the same number of feature maps as the full traditional convolution, as shown in Figure 6 As shown, the amount of calculation can be greatly reduced. In the same convolution layer, the amount of calculation using full traditional convolution and mapped convolution is as follows: (8); (9); in, o is the number of convolution kernels, and the input format of the image is , c is the input channel, d and v are the height and width of the image, respectively. l is the convolution kernel size of the traditional convolution, b is the convolution kernel size of ordinary convolution, z is the number of convolution transformations, is the computational amount of traditional convolution in mapping convolution, is the computational complexity of the ordinary convolution in the mapping convolution. In addition, the convolution kernel sizes are similar, that is, , is the amount of computation of all traditional convolutions, is the amount of computation of the mapped convolution, so the comparison of the computation of the full traditional convolution and the mapped convolution is as follows: (10).

[0043] Therefore, the computational complexity of full traditional convolution is z times (z>1) that of mapped convolution. While achieving the same number of feature maps as full traditional convolution, mapped convolution reduces the computational complexity to 1 / z of the original, thus improving the real-time performance of semantic segmentation and recognition of bridge defects.

[0044] Compared with the traditional bridge defect target recognition method based on semantic segmentation, the method provided by the present invention extracts the contrast features of the image on the multi-scale convolution layer through a multi-scale feature contrast mechanism, and integrates them with high-order semantic information to achieve the enhancement of the contrast features in the semantic information. This strong contrast is conducive to highlighting the bridge defect target from the surrounding environment, so as to improve the recognition accuracy of the bridge defect target. In addition, since the traditional convolution is a full combination of convolution-normalization-nonlinear activation, the calculation amount is large, and the ordinary convolution only performs convolution operations, does not include normalization and nonlinear activation, and the calculation amount is small, the mapping convolution is in the form of half traditional convolution + half ordinary convolution. While obtaining the same number of feature maps as the traditional convolution, the calculation amount of the model can be greatly reduced to improve the real-time performance of bridge defect target recognition.

[0045] Example 3.

[0046] The present invention provides a bridge disease identification device, comprising: The bridge image acquisition module is configured to: acquire a bridge image; The bridge defect recognition module is configured to: input the bridge image into the trained bridge defect recognition model to obtain a bridge defect recognition image, wherein the bridge defect recognition image is labeled with the defects contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the basic model of bridge disease semantic segmentation based on encoder-decoder, and the improvement includes: adding a multi-scale feature comparison mechanism to the basic model of bridge disease semantic segmentation, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

[0047] Example 4.

[0048] The present invention provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the bridge defect identification method provided in Example 1 are implemented: Get bridge images; Inputting the bridge image into the trained bridge disease recognition model to obtain a bridge disease recognition image, wherein the bridge disease recognition image is labeled with the diseases contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the encoder-decoder based bridge disease semantic segmentation basic model. The improvements include: adding a multi-scale feature comparison mechanism to the bridge disease semantic segmentation basic model, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

[0049] Example 5.

[0050] The present invention provides a computer device, comprising: Memory, for storing computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of the bridge disease identification method provided in Example 1: Get bridge images; Inputting the bridge image into the trained bridge disease recognition model to obtain a bridge disease recognition image, wherein the bridge disease recognition image is labeled with the diseases contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the encoder-decoder based bridge disease semantic segmentation basic model. The improvements include: adding a multi-scale feature comparison mechanism to the bridge disease semantic segmentation basic model, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

[0051] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0054] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A bridge disease identification method, characterized in that: include: Get bridge images; Inputting the bridge image into the trained bridge disease recognition model to obtain a bridge disease recognition image, wherein the bridge disease recognition image is labeled with the diseases contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the basic model of bridge disease semantic segmentation based on encoder-decoder, and the improvement includes: adding a multi-scale feature comparison mechanism to the basic model of bridge disease semantic segmentation, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

2. The bridge disease identification method according to claim 1, characterized in that: The operations performed by the multi-scale feature comparison mechanism on the feature matrix in the bridge defect recognition model include: On the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain an optimal feature matrix that best represents the convolution layer, wherein the feature matrix is ​​obtained by extracting features of the bridge image through each channel in the convolution layer of the encoder; The feature matrix fusion operation is performed on convolutional layers of all scales to obtain the optimal feature matrices corresponding to convolutional layers of all scales, and all the optimal feature matrices are merged to form a multi-scale feature pyramid; Performing a cross-scale reduction operation on the multi-scale feature pyramid to obtain contrast information of the bridge disease target and surrounding background information, thereby generating a multi-scale feature contrast matrix; The multi-scale feature contrast matrix is ​​combined with the semantic segmentation features obtained from the original bridge defect semantic segmentation basic model; The fused semantic segmentation features are used to classify whether they belong to a defect target through the classifier in the original bridge defect semantic segmentation basic model.

3. The bridge disease identification method according to claim 2, characterized in that: In the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain the optimal feature matrix that best represents the convolution layer, including: Get the feature matrices of any two different channels in the convolutional layer from the set of all feature matrices of the convolutional layer and , we can obtain the value of and The two vector matrices with the largest correlation and : （1）； in, T represents transpose, argmax represents the argmax function, for and The correlation coefficient between Represents the convolutional layer j Middle Channel i The characteristic matrix of Represents the convolutional layer j Middle Channel i+ The feature matrix of 1, express The corresponding vector matrix is, express The corresponding vector matrix; By formula (2), and pass and Two vector matrices are fused: （2）； in Represents the feature matrix after a single fusion; The fused feature matrix Replace the original convolutional layer with all feature matrices in the collection and , and form a new feature set with the remaining feature matrices in the set : （3）； exist Re-acquire and As the characteristic matrix in the above formula (1) and , perform feature fusion through formula (1), formula (2), and formula (3), and generate the next new feature set , and the calculation is repeated until all channels are cycled through to obtain the fusion that best represents the convolution layer. n The optimal feature matrix after , n Indicates the number of channels of the convolutional layer.

4. The bridge disease identification method according to claim 1, characterized in that: The operations performed by the mapping convolution mechanism on the encoder / decoder input include: Use half the number of convolution kernels in the original bridge defect semantic segmentation basic model to perform traditional convolution on the encoder / decoder input to obtain a traditional convolution feature map; The traditional convolution feature map is extracted through ordinary convolution to generate an ordinary convolution feature map; The traditional convolution feature map and the ordinary convolution feature map are spliced ​​together to obtain the final feature map; After the input of the encoder / decoder is processed by the mapping convolution mechanism to obtain the final feature map, the encoder / decoder performs subsequent processing on the obtained final feature map.

5. The bridge disease identification method according to claim 1, characterized in that: The training of the bridge disease identification model includes: Acquire multiple bridge images to construct a bridge inspection sample set, and pre-process the bridge inspection sample set; Annotate the bridge images in the bridge inspection sample set at the pixel level for bridge defects using an image calibration tool to obtain a bridge defect dataset, and divide the bridge defect dataset into a training set, a validation set, and a test set according to a preset ratio; The bridge defect recognition model is trained using the training set, the training result of the bridge defect recognition model is adjusted using the validation set, and the training effect of the bridge defect recognition model is verified using the test set until the recognition accuracy of the bridge defect recognition model reaches the preset requirement.

6. The bridge defect identification method according to claim 5, characterized in that: The preprocessing includes: rotating, flipping and / or adding noise to the bridge data in the bridge inspection sample set.

7. A bridge disease identification device, characterized in that: include: The bridge image acquisition module is configured to: acquire a bridge image; The bridge defect recognition module is configured to: input the bridge image into the trained bridge defect recognition model to obtain a bridge defect recognition image, wherein the bridge defect recognition image is annotated with the defects contained in the bridge image and their categories; Among them, the bridge disease recognition model is improved on the basis of the basic model of bridge disease semantic segmentation based on encoder-decoder, and the improvement includes: adding a multi-scale feature comparison mechanism to the basic model of bridge disease semantic segmentation, and changing the convolution mechanism in the encoder and decoder to a mapping convolution mechanism.

8. The bridge defect identification device according to claim 7, characterized in that: In the bridge defect identification module, the multi-scale feature comparison mechanism performs the following operations on the feature matrix in the bridge defect identification model: On the convolution layer of the encoder, feature matrices of multiple channels in the convolution layer are fused by correlation analysis to obtain an optimal feature matrix that best represents the convolution layer, wherein the feature matrix is ​​obtained by extracting features from the bridge image by the encoder; The feature matrix fusion operation is performed on convolutional layers of all scales to obtain the optimal feature matrices corresponding to convolutional layers of all scales, and all the optimal feature matrices are merged to form a multi-scale feature pyramid; Performing a cross-scale reduction operation on the multi-scale feature pyramid to obtain contrast information of the bridge disease target and surrounding background information, thereby generating a multi-scale feature contrast matrix; The multi-scale feature contrast matrix is ​​combined with the semantic segmentation features obtained from the original bridge defect semantic segmentation basic model; The fused semantic segmentation features are classified into whether they belong to the defect target through the classifier in the original bridge defect semantic segmentation basic model; In the bridge defect identification module, the operations performed by the mapping convolution mechanism on the input of the encoder / decoder include: Use half the number of convolution kernels in the original bridge defect semantic segmentation basic model to perform traditional convolution on the encoder / decoder input to obtain a traditional convolution feature map; The traditional convolution feature map is extracted through ordinary convolution to generate an ordinary convolution feature map; The traditional convolution feature map and the ordinary convolution feature map are spliced ​​together to obtain the final feature map; After the input of the encoder / decoder is processed by the mapping convolution mechanism to obtain the final feature map, the encoder / decoder performs subsequent processing on the obtained final feature map.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the bridge defect identification method according to any one of claims 1 to 6 are implemented.

10. A computer device, characterized in that: include: Memory, for storing computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of the bridge defect identification method according to any one of claims 1 to 6.

Images