Bridge corrosion detection method and device

Abstract

The present invention provides a bridge corrosion detection method and device, the method comprising: using a target detection model to identify each corrosion area and the first corrosion type of each corrosion area from a bridge image to be detected; using an image encoder to obtain image features of each corrosion area, using a text encoder to obtain text features of each first corrosion type; matching the image features of each corrosion area with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion area belong as the second corrosion type of each corrosion area. The present invention uses a multimodal fusion method to quickly and accurately detect bridge corrosion.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and in particular to a bridge corrosion detection method and device. Background Art

[0002] Bridge corrosion is a direct manifestation of the stress state, durability and safety of bridge structures, and is one of the important indicators for bridge safety inspection and structural research. Regular inspection of bridge corrosion is a basic means of bridge management and maintenance.

[0003] Bridge inspection methods are mainly divided into traditional inspection methods and image inspection methods. Traditional inspection methods include radar technology (GPR), acoustic emission, infrared thermal imaging and other technologies, but these methods have high requirements for the inspection environment and equipment. Image inspection methods based on deep learning are highly dependent on the acquisition quality of bridge images. At present, with the changes in external lighting and weather, it is difficult to obtain high-quality bridge images, which leads to inaccurate bridge corrosion detection. Summary of the invention

[0004] The present invention provides a bridge corrosion detection method and device, which are used to solve the defect of inaccurate bridge corrosion detection in the prior art and improve the accuracy of bridge corrosion detection.

[0005] The present invention provides a bridge corrosion detection method, comprising:

[0006] Using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected;

[0007] Using an image encoder to obtain image features of each erosion area, and using a text encoder to obtain text features of each first erosion type;

[0008] The image features of each corrosion region are matched with the text features of each first corrosion type, and the first corrosion type to which the text features matched with the image features of each corrosion region belong is used as the second corrosion type of each corrosion region.

[0009] According to a bridge corrosion detection method provided by the present invention, the image features of each corrosion area are matched with the text features of each first corrosion type, and the first corrosion type to which the text features matched with the image features of each corrosion area belong is used as the second corrosion type of each corrosion area, including:

[0010] Determining the similarity between the image features of each corrosion area and the text features of each first corrosion type;

[0011] Determine the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types;

[0012] When the maximum value is greater than a preset threshold, the first corrosion type corresponding to the maximum value is used as the second corrosion type of each corrosion area.

[0013] According to a bridge corrosion detection method provided by the present invention, after determining the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types, the method further includes:

[0014] When the maximum value is less than or equal to the preset threshold, determining the previous level corrosion type of each first corrosion type;

[0015] Using the text encoder to obtain text features of each previous level of corrosion type;

[0016] The image features of each corrosion region are matched with the text features of each previous corrosion type, and the previous corrosion type to which the text features matched with the image features of each corrosion region belong is used as the third corrosion type of each corrosion region.

[0017] According to a bridge corrosion detection method provided by the present invention, the target detection model is a YOLOv7 model.

[0018] According to a bridge corrosion detection method provided by the present invention, the backbone network of the YOLOv7 model is a ConvNext-T model.

[0019] According to a bridge corrosion detection method provided by the present invention, the YOLOv7 model includes multiple detection heads, each detection head includes a recursive gated convolution layer, a RepConv layer and a 1x1 convolution layer;

[0020] The output end of the backbone network of the YOLOv7 model corresponds one-to-one to the input end of the detection head.

[0021] According to a bridge corrosion detection method provided by the present invention, before using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the image of the bridge to be detected, the method further includes:

[0022] Inputting the original bridge image samples collected in the real system and the actual corrosion types of the original bridge image samples into the artificial system;

[0023] In the artificial system, new bridge image samples are generated based on the original bridge image samples, the original bridge image samples and the new bridge image samples are used as samples, and the actual corrosion types of the original bridge image samples are used as labels to train the target detection model, the image encoder and the text encoder.

[0024] The present invention also provides a bridge corrosion detection device, comprising:

[0025] An identification module, used for identifying each corrosion area and a first corrosion type of each corrosion area from the image of the bridge to be detected by using a target detection model;

[0026] An encoding module, used for obtaining the image features of each erosion area by using an image encoder, and obtaining the text features of each first erosion type by using a text encoder;

[0027] The matching module is used to match the image features of each corrosion area with the text features of each first corrosion type, and use the first corrosion type to which the text features matching the image features of each corrosion area belong as the second corrosion type of each corrosion area.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned bridge corrosion detection methods is implemented.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the bridge corrosion detection method as described in any one of the above is implemented.

[0030] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the bridge corrosion detection method as described above is implemented.

[0031] The bridge corrosion detection method and device provided by the present invention use a target detection model to detect the target of the bridge image to be detected in the first stage, and identify each corrosion area and the corrosion type of each corrosion area; in the second stage, an image encoder and a text encoder are used to respectively obtain the image features of each corrosion area and the text features of the corrosion type, and the corrosion type of each corrosion area is identified again by matching the image features and the text features, and a multi-modal fusion method is used to quickly and accurately detect bridge corrosion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is one of the flow diagrams of the bridge corrosion detection method provided by the present invention;

[0034] Figure 2 It is a schematic diagram of the overall structure of the bridge corrosion detection model in the bridge corrosion detection method provided by the present invention;

[0035] Figure 3 It is a schematic diagram of matching image features and text features of the corrosion area in the bridge corrosion detection method provided by the present invention;

[0036] Figure 4 It is a schematic diagram of corrosion classification divided according to a tree structure in the bridge corrosion detection method provided by the present invention;

[0037] Figure 5 It is a schematic diagram of the structure of the recursive gated convolution in the bridge corrosion detection method provided by the present invention;

[0038] Figure 6 It is a structural schematic diagram of an improved YOLOv7 model in the bridge corrosion detection method provided by the present invention;

[0039] Figure 7 This is the second flow chart of the bridge corrosion detection method provided by the present invention;

[0040] Figure 8 It is a schematic diagram of the architecture of the parallel execution system in the bridge corrosion detection method provided by the present invention;

[0041] Fig. 9 It is a schematic diagram of the parallel execution process in the bridge corrosion detection method provided by the present invention;

[0042] Fig.10 It is a structural schematic diagram of the bridge corrosion detection device provided by the present invention;

[0043] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Combine the following Figure 1 A bridge corrosion detection method of the present invention is described, comprising:

[0046] Step 101, using a target detection model to identify each corrosion area and a first corrosion type of each corrosion area from a bridge image to be detected;

[0047] In this embodiment, bridge corrosion detection is divided into two stages. In the first stage, the target detection model is used to perform target detection on the bridge image to be detected, accurately identify the damage on the surface of the bridge structure, and extract the detected bounding box image (Bounding Boxes), that is, the corrosion area and the class label (Class Labels), that is, the first corrosion type of the corrosion area. The results are output to the image encoder and the text encoder. This embodiment does not limit the target detection model. The first corrosion type is shown in Table 1.

[0048] Table 1 Examples of corrosion types

[0049]

[0050]

[0051] Step 102, using an image encoder to obtain image features of each erosion area, and using a text encoder to obtain text features of each first erosion type;

[0052] In the second stage, the text encoder and image encoder are fused into the feature extractor of the single-stage object detection model. The text encoder is responsible for processing natural language description information, while the image encoder focuses on processing image information. After the two are combined, they are input into the model for joint encoding, which converts the image and natural language description into vector representations. The model uses these vectors for feature extraction, which enhances the model's semantic understanding and object detection capabilities.

[0053] Based on the object detection model, a text encoder and an image encoder are used. Through this hierarchical design, the network can learn the commonalities and differences between different types of corrosion with similar appearance, so that the network has the ability to identify unknown corrosion and achieve accurate detection of unknown corrosion. This method of comprehensive use of text and image information not only improves the performance of the model, but also makes the model more adaptable and better suited for complex bridge damage detection tasks.

[0054] Step 103 , matching the image features of each corrosion region with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion region belong as the second corrosion type of each corrosion region.

[0055] The image features of each corrosion area are matched with the text features of each first corrosion type to understand the similarity between the detected corrosion area and the natural language description. Combined with GPS (Global Positioning System) data, the extracted bridge corrosion area is semantically understood to identify the type, status and other information of the bridge. Finally, the detection and identification results are output, including the coordinate information of the bridge area and the semantic understanding results, so as to realize the automated detection and analysis of bridge damage.

[0056] In this embodiment, in the first stage, the target detection model is used to detect the target of the bridge image to be detected, and each corrosion area and the corrosion type of each corrosion area are identified; in the second stage, the image encoder and the text encoder are used to obtain the image features of each corrosion area and the text features of the corrosion type respectively, and the corrosion type of each corrosion area is identified again by matching the image features and the text features, and the bridge corrosion is detected quickly and accurately by using a multimodal fusion method.

[0057] On the basis of the above embodiment, the method of matching the image features of each corrosion region with the text features of each first corrosion type in this embodiment, and using the first corrosion type to which the text features matched with the image features of each corrosion region belong as the second corrosion type of each corrosion region, includes:

[0058] Determining the similarity between the image features of each corrosion area and the text features of each first corrosion type;

[0059] Determine the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types;

[0060] When the maximum value is greater than a preset threshold, the first corrosion type corresponding to the maximum value is used as the second corrosion type of each corrosion area.

[0061] This embodiment adds a text encoder and an image encoder. These two components are responsible for converting text and image data into feature representations that the model can understand when processing them. By calculating the similarity between image features and text features, the association between different modal information can be deeply explored, so that the model can better understand the semantic connection between images and text. This multimodal fusion method not only improves the model's ability to process complex information, but also enables the model to more comprehensively understand the context and details in the bridge corrosion detection task, thereby enhancing the model's semantic understanding and target detection capabilities.

[0062] Figure 2 Schematic diagram of the overall structure of the bridge corrosion detection model in the bridge corrosion detection method provided by the present invention. Figure 2 As shown, bridge corrosion detection methods include:

[0063] The image data of the bridge surface is input into the backbone network based on the single-stage target detection model YOLOv7 to obtain the corrosion areas of the bridge image to be detected, the detection frames of the corrosion areas and the first corrosion types of the corrosion areas to form a detection report.

[0064] The text data in the inspection report is input into the text encoder for feature extraction to form a text vector space, and the text data of each first corrosion type is converted into a set of feature vectors, thereby obtaining a feature matrix of the form (N, D). Among them, N represents the number of first corrosion types. D represents the dimension of the feature vector. At the same time, the complete image of the bridge to be inspected and each corrosion area are input into the image encoder to generate an image vector space.

[0065] The image features and text features are matched cross-modally to obtain the similarity score between each image feature and each text feature. Specifically, the image feature vector of each corrosion area is matched with the text feature vectors in all detection reports to obtain a similarity matrix of the form (N, M), where N is the number of the first corrosion type and M is the number of corrosion areas, such as Figure 3 shown.

[0066] The image features and text features are fused by element-by-element multiplication, and then each eroded area is classified and predicted through the fully connected layer and the softmax layer.

[0067] According to the similarity matrix, damage detection is performed on the surface of the bridge structure. A threshold of similarity score is selected, and the corrosion areas and detection reports with similarity scores greater than the threshold are taken as a set of matching results. At the same time, the matching results are screened and analyzed according to attributes such as damage type, damage location and damage severity, so as to obtain more information about the detection target, correct or reallocate the corrosion category in the target detection results, and correct the false positives or negatives of the single target detection model in the actual detection work.

[0068] Based on the above embodiment, after determining the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types, this embodiment further includes:

[0069] When the maximum value is less than or equal to the preset threshold, determining the previous level corrosion type of each first corrosion type;

[0070] Using the text encoder to obtain text features of each previous level of corrosion type;

[0071] The image features of each corrosion region are matched with the text features of each previous corrosion type, and the previous corrosion type to which the text features matched with the image features of each corrosion region belong is used as the third corrosion type of each corrosion region.

[0072] Figure 4 FIG. 1 is a schematic diagram of corrosion classification according to the tree structure in the bridge corrosion detection method provided by the present invention. Figure 4 As shown in the figure, a hierarchical corrosion classification representation method is proposed in combination with a word tree. By orderly dividing different types of corrosion, the relationship between different types of corrosion is constructed. Through the hierarchical design, the network can learn the commonalities and differences between corrosions of different types but similar appearances. During the training process, the mutual connection between various types of corrosion is strengthened, so that the network has the ability to identify unknown corrosion. According to the three primary corrosion categories divided into corrosion morphology, the specific corrosion categories under the primary corrosion category are defined as secondary corrosion categories.

[0073] In the case where the first corrosion type is a secondary corrosion type, a similarity matrix between the text features of the first corrosion type and the image features of the corrosion area is determined. If the maximum similarity value corresponding to any corrosion area in the similarity matrix is ​​less than or equal to a preset threshold, the similarity between the text features of each primary corrosion type and the image features of the corrosion area is determined. If the maximum similarity value is greater than the preset threshold, the primary corrosion type corresponding to the maximum similarity value is used as the corrosion type of the corrosion area.

[0074] On the basis of the above embodiments, the target detection model described in this embodiment is a YOLOv7 model.

[0075] The target detection model YOLOv7 is used to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected. The YOLOv7 model can be improved for bridge corrosion detection scenarios by combining the self-attention mechanism to achieve fast and accurate detection of bridge corrosion.

[0076] Based on the above embodiment, the backbone network of the YOLOv7 model described in this embodiment is a ConvNext-T model.

[0077] This embodiment uses the ConvNext-T model to replace the backbone network of the YOLOv7 model, thereby achieving the purpose of reducing model parameters and improving detection speed.

[0078] On the basis of the above embodiment, the YOLOv7 model described in this embodiment includes multiple detection heads, each detection head includes a recursive gated convolution layer, a RepConv layer and a 1x1 convolution layer;

[0079] The output end of the backbone network of the YOLOv7 model corresponds one-to-one to the input end of the detection head.

[0080] In the head of YOLOv7, recursive gated convolution (g n Conv), through gated convolution and recursive design, continuously performs element-by-element multiplication operations, performs efficient, scalable and translation-equivariant high-order spatial interactions, retains more high-order feature information; at the same time, it suppresses invalid feature weights and irrelevant information, thereby improving the overall accuracy of target detection.

[0081] The calculation method of the first-order recursive gated convolution (gConv) is:

[0082] Let x∈R HW×C For the input features, the output of the gated convolution y=gConv(x) can be written as:

[0083]

[0084] p 1 =f(q 0 )⊙p 0 ∈R HW×C ;

[0085] y=ф out (p 1 )∈R HW×C ;

[0086] Among them, in and out is a projection layer that performs channel mixing, and f is a depthwise convolution.

[0087]

[0088] Among them, Ω i is the center point of the i-th window, and w is the weight of the depthwise convolution f.

[0089] High-order recursive gated convolution (g n Conv) is calculated as:

[0090] First, use ф in Get a set of projection features p 0 and

[0091]

[0092] Next, we perform a recursive gated convolution (g n Conv) calculation:

[0093]

[0094] The output is scaled to 1 / α to stabilize the training, {f k} is a set of deep convolutional layers, {g k} is used to match dimensions in a different order:

[0095]

[0096] The last recursive step q n The output of out , we get g n The result of Conv. Recursive gated convolution (g n Conv) is shown in the following figure. Figure 5 shown.

[0097] Figure 6 Schematic diagram of the structure of the improved YOLOv7 model in the bridge corrosion detection method provided by the present invention. The YOLOv7 model includes multiple detection heads. Due to the uneven distribution of corrosion target sizes, in order to improve the recognition effect of small-sized corrosion targets, a prediction head for micro-corrosion detection is added. Figure 6 As shown in the figure, combined with the other three detection heads, the structure of the four detection heads can effectively alleviate the negative impact brought by the change of target scale and solve the problem of uneven size distribution of corrosion targets.

[0098] The bridge image to be detected is input into the backbone network of the YOLOv7 model to obtain feature maps C1, C2, C3, C4 and C5. The backbone network finally outputs a 32-fold downsampled feature map C5, which is then processed by SPPCSP, and the number of channels is changed from 1024 to 512.

[0099] The feature maps C4, C3, and C2 are fused with the feature maps P2, P3, P4, and P5 in a top-down manner, and then fused again with the feature maps P3, P4, and P5 in a bottom-up manner to obtain the final feature maps M2, M3, M4, and M5.

[0100] The dimensions of the feature maps C1, C2, C3, C4 and C5 may be 320×320×64, 160×160×256, 80×80×512, 40×40×1024 and 20×20×1024, respectively. The dimensions of the feature maps P2, P3, P4 and P5 may be 160×160×64, 80×80×128, 40×40×256 and 20×20×512, respectively.

[0101] The feature maps M2, M3, M4 and M5 are input into a detection head for detection. The number of channels of the detection result is num_anchors×(5+num_classes), where num_anchors is the number of prior boxes of each grid, and num_classes is the number of the first erosion types.

[0102] The feature maps M2, M3, M4 and M5 are respectively subjected to recursive gated convolution (g n Conv), performs 5 high-order spatial interactions to retain more high-order feature information, adjusts the number of channels through RepConv, and finally uses 1x1 convolution to get the prediction result. The prediction result can include three parts: objectness of the eroded area, the first erosion type class of the eroded area, and the detection box bbox of the eroded area.

[0103] The target detection model adds recursive gated convolution (g n Conv), can continuously perform element-by-element multiplication operations through gated convolution and recursive design, perform efficient, scalable and translation-equivariant high-order spatial interactions, retain more high-order feature information; at the same time, suppress invalid feature weights and irrelevant information, thereby improving the overall accuracy of target detection.

[0104] On the basis of the above embodiments, before using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the image of the bridge to be detected, this embodiment further includes:

[0105] Before using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected, the method further includes:

[0106] Inputting the original bridge image samples collected in the real system and the actual corrosion types of the original bridge image samples into the artificial system;

[0107] In the artificial system, new bridge image samples are generated based on the original bridge image samples, the original bridge image samples and the new bridge image samples are used as samples, and the actual corrosion types of the original bridge image samples are used as labels to train the target detection model, the image encoder and the text encoder.

[0108] like Figure 7 As shown in the figure, a bridge corrosion dataset is constructed. UAVs can be used to collect bridge corrosion images, and data cleaning and data enhancement are performed to construct a bridge corrosion dataset.

[0109] The collected bridge corrosion images can be normalized and converted into bridge corrosion images with a resolution of 3200×3200 to ensure the consistency of image size. Then, the erroneous data in the bridge corrosion images are deleted, the duplicate data is cleared, and the unchanged feature dimensions are removed to complete the data cleaning.

[0110] After data cleaning, the bridge corrosion images are amplified using geometric transformation, linear transformation, and bounding box enhancement. The enhanced bridge corrosion image dataset can be divided into training set, validation set, and test set according to 7:2:1. The target detection model is improved for the bridge corrosion detection scenario. The processed images are input into the computational experiment to obtain the detection results and load the results into the artificial system.

[0111] By setting up computational experiments, continuously training various models in the experimental scheme, and verifying the scheme at the same time, and when necessary, repeating a certain experimental step many times and in large quantities, we can comprehensively, accurately, timely and quantitatively obtain bridge inspection solutions in various complex environments, and analyze and evaluate them according to different indicator systems.

[0112] like Figure 8 As shown, a parallel theory bridge corrosion detection system is constructed, including a real corrosion detection system and an artificial corrosion detection system. The two exist independently and interact with each other to form a closed-loop control, realizing self-optimization and updating of the system.

[0113] The real corrosion detection system includes a data acquisition module, a corrosion detection module and a real data set. The data acquisition module collects bridge corrosion information in the real environment, including bridge environmental information, bridge pictures and specific parameters of bridge corrosion, to form a data set; the corrosion detection module detects the bridge pictures in the data set and obtains the detection results; the detection results and the data set together constitute the real data set and are passed to the artificial system.

[0114] The artificial system consists of an artificial corrosion generation module, a corrosion detection module, an artificial data set, an artificial scene generation module, and an artificial scene. By introducing parallel data-related theories and frameworks, an artificial corrosion generation module was creatively designed. This system uses generative adversarial network (GAN) technology, and through the artificial corrosion generation module, it can generate extremely realistic artificial corrosion images based on corrosion images in real data sets. These artificial corrosion images are not only highly realistic, but also combined with real data sets to form a large and diverse big data set.

[0115] The real system runs in the actual environment, capturing the corrosion of the bridge in real time and passing this information to the artificial system. The artificial system continues to simulate various corrosion situations in the virtual scene, receives feedback from the real system, and continuously improves itself based on this information, forming a dynamic and interconnected virtual-real system interaction.

[0116] This tightly integrated virtual-real system interaction builds a closed-loop control system that enables the bridge corrosion detection system to have the ability to self-optimize and update. In this way, the system can continuously improve its performance over time, adapt to changing environmental conditions, and keep working in the best state, ensuring high accuracy and reliability of bridge corrosion detection.

[0117] In order to provide effective training data, the parallel image basic framework is introduced, and the software-defined artificial image system is used to obtain specific image data from the real physical world, and a large amount of corrosion data is generated with the help of the Generative Adversarial Network (GAN). These diverse data provide sufficient training support for the model, and ultimately realize the automatic detection and analysis of bridge corrosion and other damage.

[0118] The Generative Adversarial Network (GAN) is used to take the bridge corrosion image dataset as input, and a large number of bridge corrosion images are generated through the continuous game between the generator and the discriminator. The generated bridge corrosion images and the collected real bridge corrosion images are combined into a richer corrosion image dataset.

[0119] In order to improve the detection efficiency, the original bridge corrosion image is segmented into small images with a resolution of 640×640, and then the generated small images are classified using the classification model. The improved target detection model is used to perform target detection on the segmented 640×640 small images, and the images containing corrosion are spliced ​​to obtain a complete corrosion image.

[0120] The generative adversarial network, classification model, and target detection model used in the computational experiment all adopt the transfer learning method, loading the training weights into the training model and training them based on the existing data set.

[0121] like Fig. 9 As shown in the figure, the artificial system is a virtual mapping of the real system in computer space, and the two are executed in parallel. The real data generated by the real system is continuously input into the artificial system, so that the virtual space continuously adjusts and optimizes the detection model and detection plan, and guides the real system to perform bridge detection; the real system continuously feeds back the detection results, so that the artificial system is further optimized; the results of its computational experiments will be used to merge with the real data to form feedback and realize true data-driven.

[0122] This embodiment introduces a closed-loop optimization method based on parallel learning theory, which can automatically extract valuable corrosion knowledge from complex data. At the same time, it can also optimize the knowledge in the actual bridge inspection report and convert it into data, forming a two-way closed loop. It not only realizes the deep integration of the corrosion image data on the surface of the bridge structure and the actual inspection report data, but also brings efficient and intelligent solutions to the field of corrosion detection, thereby improving the accuracy and reliability of bridge corrosion detection.

[0123] The bridge corrosion detection device provided by the present invention is described below. The bridge corrosion detection device described below and the bridge corrosion detection method described above can be referenced to each other.

[0124] like Fig.10 As shown, the device includes an identification module 1001, an encoding module 1002 and a matching module 1003, wherein:

[0125] The identification module 1001 is used to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected by using the target detection model;

[0126] The encoding module 1002 is used to obtain the image features of each corrosion area using an image encoder, and obtain the text features of each first corrosion type using a text encoder;

[0127] The matching module 1003 is used to match the image features of each corrosion region with the text features of each first corrosion type, and use the first corrosion type to which the text features matching the image features of each corrosion region belong as the second corrosion type of each corrosion region.

[0128] In this embodiment, in the first stage, the target detection model is used to detect the target of the bridge image to be detected, and each corrosion area and the corrosion type of each corrosion area are identified; in the second stage, the image encoder and the text encoder are used to obtain the image features of each corrosion area and the text features of the corrosion type respectively, and the corrosion type of each corrosion area is identified again by matching the image features and the text features, and the bridge corrosion is detected quickly and accurately by using a multimodal fusion method.

[0129] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the bridge corrosion detection method, which includes: using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected; using the image encoder to obtain the image features of each corrosion area, and using the text encoder to obtain the text features of each first corrosion type; matching the image features of each corrosion area with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion area belong as the second corrosion type of each corrosion area.

[0130] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the bridge corrosion detection method provided by the above-mentioned methods, which includes: using a target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the image of the bridge to be detected; using an image encoder to obtain image features of each corrosion area, and using a text encoder to obtain text features of each first corrosion type; matching the image features of each corrosion area with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion area belong as the second corrosion type of each corrosion area.

[0132] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the bridge corrosion detection method provided by the above-mentioned methods. The method includes: using a target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the image of the bridge to be detected; using an image encoder to obtain image features of each corrosion area, and using a text encoder to obtain text features of each first corrosion type; matching the image features of each corrosion area with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion area belong as the second corrosion type of each corrosion area.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bridge corrosion detection method, characterized in that: include: Using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected; Using an image encoder to encode the bridge image to be detected and the corrosion regions to obtain image features of the corrosion regions, and using a text encoder to encode text data of the first corrosion type of the corrosion regions to obtain text features of the first corrosion types; Matching the image features of each corrosion region with the text features of each first corrosion type, and using the first corrosion type to which the text features matched with the image features of each corrosion region belong as the second corrosion type of each corrosion region; The step of matching the image features of each corrosion region with the text features of each first corrosion type, and using the first corrosion type to which the text features matched with the image features of each corrosion region belong as the second corrosion type of each corrosion region, includes: Cross-modally matching the image features of each corrosion region and the text features of each first corrosion type to determine the similarity between the image features of each corrosion region and the text features of each first corrosion type; the similarity is cosine similarity; Determine the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types; When the maximum value is greater than a preset threshold, the first corrosion type corresponding to the maximum value is used as the second corrosion type of each corrosion area.

2. The bridge corrosion detection method according to claim 1, characterized in that: After determining the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types, the method further includes: When the maximum value is less than or equal to the preset threshold, determining the previous level corrosion type of each first corrosion type; Using the text encoder to obtain text features of each previous level of corrosion type; The image features of each corrosion region are matched with the text features of each previous corrosion type, and the previous corrosion type to which the text features matching the image features of each corrosion region belong is used as the third corrosion type of each corrosion region.

3. The bridge corrosion detection method according to any one of claims 1-2, characterized in that: The target detection model is the YOLOv7 model.

4. The bridge corrosion detection method according to claim 3, characterized in that: The backbone network of the YOLOv7 model is the ConvNext-T model.

5. The bridge corrosion detection method according to claim 3, characterized in that: The YOLOv7 model includes multiple detection heads, each of which includes a recursive gated convolution layer, a RepConv layer, and a 1x1 convolution layer; The output end of the backbone network of the YOLOv7 model corresponds one-to-one to the input end of the detection head.

6. The bridge corrosion detection method according to any one of claims 1-2, characterized in that: Before using the target detection model to identify each corrosion area and the first corrosion type of each corrosion area from the bridge image to be detected, the method further includes: Inputting the original bridge image samples collected in the real system and the actual corrosion types of the original bridge image samples into the artificial system; In the artificial system, new bridge image samples are generated based on the original bridge image samples, the original bridge image samples and the new bridge image samples are used as samples, and the actual corrosion types of the original bridge image samples are used as labels to train the target detection model, the image encoder and the text encoder.

7. A bridge corrosion detection device, characterized in that: include: An identification module, used for identifying each corrosion area and a first corrosion type of each corrosion area from the image of the bridge to be detected by using a target detection model; An encoding module, used to encode the bridge image to be detected and the corrosion regions using an image encoder to obtain image features of the corrosion regions, and to encode text data of the first corrosion type of the corrosion regions using a text encoder to obtain text features of the first corrosion types; A matching module, used for matching the image features of each corrosion region with the text features of each first corrosion type, and using the first corrosion type to which the text features matching the image features of each corrosion region belong as the second corrosion type of each corrosion region; The step of matching the image features of each corrosion region with the text features of each first corrosion type, and using the first corrosion type to which the text features matched with the image features of each corrosion region belong as the second corrosion type of each corrosion region, includes: Cross-modally matching the image features of each corrosion region and the text features of each first corrosion type to determine the similarity between the image features of each corrosion region and the text features of each first corrosion type; the similarity is cosine similarity; Determine the maximum value of the similarities between the image features of each corrosion area and the text features of all the first corrosion types; When the maximum value is greater than a preset threshold, the first corrosion type corresponding to the maximum value is used as the second corrosion type of each corrosion area.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the bridge corrosion detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the bridge corrosion detection method according to any one of claims 1 to 6 is implemented.

Images