A bridge defect identification method and device based on multimodal fusion

Abstract

This application proposes a bridge defect identification method and device based on multimodal fusion, which belongs to the field of multimodal deep learning technology. The method includes: obtaining an infrared image, a visible light image, and a position code of the bridge to be detected; inputting the infrared image into a first convolutional network to obtain a first image feature; inputting the visible light image into a second convolutional network to obtain a second image feature; superimposing the second image feature with the position code and inputting it into an encoder to obtain an encoding result; inputting the encoding result into a decoder to obtain a third image feature; inputting the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature; inputting the multimodal fusion feature into a pre-trained neural network to predict bridge defects and obtain a bridge defect identification result. This application comprehensively utilizes the feature information of infrared and visible light images to improve the accuracy of bridge defect detection tasks and enhance the robustness of the detection model.

Description

Technical Field

[0001] The present application belongs to the field of multimodal deep learning technology, and specifically relates to a bridge defect identification method and device based on multimodal fusion. Background Art

[0002] Bridges are hubs of transportation and play a vital role in the transportation system. In recent years, highways, high-speed railways, and public transportation systems have developed rapidly across the country. As a key component of these highway, railway, and public transportation systems, the quality and safety of bridges are particularly important. Currently, in-service bridges are experiencing serious damage, poor performance, low durability, and short service life. Therefore, in-depth research on bridge defect identification technology is the cornerstone of preventing bridge safety accidents. There are two main types of existing identification technologies: traditional bridge defect identification methods and deep learning-based identification and detection methods. Among them, traditional bridge defect identification methods usually use manual detection or manual feature input methods, which are time-consuming, labor-intensive, and inefficient; while deep learning-based identification and detection methods have the problem of low detection rate in bridge defect identification. Summary of the Invention

[0003] Based on the above technical problems, this application proposes a bridge defect identification method and device based on multimodal fusion, which improves the accuracy of bridge defect identification by effectively fusing two input information, infrared images and visible light images.

[0004] In the first aspect, this application proposes a bridge defect identification method based on multimodal fusion, comprising:

[0005] Obtain infrared light images, visible light images, and position codes of the bridge to be inspected;

[0006] Inputting the infrared light image into a pre-trained first convolutional network to obtain a first image feature;

[0007] Inputting the visible light image into a pre-trained second convolutional network to obtain a second image feature;

[0008] Superimposing the second image feature and the position code and inputting the superimposed image feature and the position code into an encoder to obtain an encoding result;

[0009] Inputting the encoding result into the decoder to obtain the third image feature;

[0010] Inputting the first image feature and the third image feature into a pre-trained multimodal fusion network to obtain a multimodal fusion feature;

[0011] The multimodal fusion features are input into a pre-trained neural network to predict bridge defects and obtain a bridge defect identification result, which includes: whether there is a bridge defect and / or the type of bridge defect.

[0012] Inputting the infrared light image into a first convolutional network to obtain a first image feature includes:

[0013] Preprocessing the infrared light image to obtain a preprocessing result;

[0014] Performing image enhancement on the preprocessing result to obtain an image enhancement result;

[0015] The image enhancement result is input into a first convolutional network to obtain a first image feature.

[0016] The first convolutional network includes: a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first fully connected layer, and a second fully connected layer; inputting the image enhancement result into the first convolutional network to obtain the first image feature includes the following steps:

[0017] The stride is 2 and the convolution kernel is The first convolution layer with a stride of 2 and a convolution kernel of The first pooling layer performs feature extraction on the image enhancement result to obtain hidden layer features;

[0018] The hidden layer features are input into the convolution kernel with a stride of 2. The second convolution layer has a stride of 2 and a convolution kernel of The second pooling layer obtains the feature map;

[0019] The feature map is input into the first fully connected layer, and the output result of the first fully connected layer is input into the second fully connected layer to obtain the first image feature.

[0020] The second convolutional network includes a residual module and a plurality of convolution kernels connected to the residual module.

[0021] The step of superimposing the second image feature and the position code and inputting the superimposed image feature and the position code into an encoder to obtain an encoding result includes:

[0022] The dimension of the second image feature Dimensionality reduction to , so that the dimension of the second image feature dimensionality reduction result is consistent with the dimension of the position code, and the second image feature dimensionality reduction result and the position code are superimposed to obtain a superposition result, wherein, is the number of second image features, is the width of the second image feature, is the height of the second image feature;

[0023] The superposition result is input to the encoder to obtain the encoding result.

[0024] Inputting the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature includes:

[0025] concatenating the first image feature and the third image feature, wherein the first dimension of the first image feature is equal to the first dimension of the third image feature;

[0026] The concatenated features are input into the third fully connected layer of the multimodal fusion network to obtain the multimodal fusion features.

[0027] The neural network includes: a first feedforward neural network for bounding box prediction and a second feedforward neural network for category prediction; the multimodal fusion features are input into the pre-trained neural network to predict bridge defects and obtain bridge defect recognition results, including:

[0028] Inputting the multimodal fusion features into a first feedforward neural network to obtain a multimodal fusion feature with a bridge disease bounding box;

[0029] The multimodal fusion features with bridge defect bounding boxes are input into the second feedforward neural network to obtain the bridge defect recognition results.

[0030] In a second aspect, the present application proposes a bridge defect identification device based on multimodal fusion, comprising:

[0031] A data acquisition module is used to obtain infrared images, visible light images and position codes of the bridge defects to be detected;

[0032] A first image feature acquisition module, configured to input the infrared light image into a first convolutional network to obtain a first image feature;

[0033] A second image feature acquisition module, configured to input the visible light image into a second convolutional network to obtain a second image feature;

[0034] an encoding module, configured to superimpose the second image feature and the position code and input the superimposed ...

[0035] A third image feature acquisition module, configured to input the encoding result into a decoder to obtain a third image feature;

[0036] a feature fusion module, configured to input the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature;

[0037] The result prediction module is used to input the multimodal fusion features into the neural network for prediction to obtain the bridge disease identification result.

[0038] In a third aspect, the present application proposes a computer program product, comprising: a computer program / instruction, which, when executed by a processor, implements the steps of the bridge defect identification method based on multimodal fusion.

[0039] In a fourth aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory stores instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the bridge defect identification method based on multimodal fusion.

[0040] In a fifth aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the bridge defect identification method based on multimodal fusion.

[0041] Beneficial effects:

[0042] This application proposes a bridge defect identification method and device based on multimodal fusion, which identifies defects through infrared and visible light images of the bridge. Based on the multimodal fusion framework, the feature information of infrared and visible light images is comprehensively utilized, which not only takes advantage of the infrared image's lack of external ambient light and less environmental influence, but also takes advantage of the visible light image's intuitiveness and clarity, making the two data features complementary and mutually usable, thereby improving the accuracy of the bridge defect detection task and enhancing the robustness of the detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a bridge defect identification method based on multimodal fusion according to an embodiment of the present application;

[0044] Figure 2 This is a flowchart of obtaining the first image feature according to an embodiment of the present application;

[0045] Figure 3 This is a flow chart of a bridge defect identification method based on multimodal fusion according to an embodiment of the present application;

[0046] Figure 4 This is a schematic diagram of the internal structure of the first convolutional network according to an embodiment of the present application;

[0047] Figure 5 Schematic diagram of dimensionality reduction of the second image feature in an embodiment of the present application;

[0048] Figure 6 This is a schematic diagram of a Transformer encoder according to an embodiment of the present application;

[0049] Figure 7 This is a schematic diagram of a Transformer decoder according to an embodiment of the present application;

[0050] Figure 8 This is a schematic diagram of the internal structure of the multimodal fusion embodiment of the present application;

[0051] Figure 9 This is a schematic diagram of multimodal fusion according to an embodiment of the present application;

[0052] Figure 10 This is a schematic diagram of bridge disease identification results in an embodiment of the present application;

[0053] Figure 11 This is a principle block diagram of a bridge defect identification device based on multimodal fusion in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present disclosure will be further described below with reference to the embodiments shown in the accompanying drawings.

[0055] Bridges are transportation hubs and play a vital role in the transportation system. In recent years, highways, high-speed railways, and public transportation systems have developed rapidly across China. However, bridges, as key components of these systems, still face maintenance challenges. For example, a large number of existing bridges suffer from severe damage, poor performance, low durability, and short service life. This increases the risk of bridge accidents and can even result in significant loss of life and property. Therefore, in-depth research on bridge defect identification technology is a cornerstone of preventing bridge accidents. Existing identification technologies primarily include two types: traditional bridge defect identification methods and traditional identification and detection methods based on deep learning.

[0056] (1) Traditional bridge defect identification methods

[0057] (1) Manual detection method

[0058] Currently, routine bridge maintenance both domestically and internationally primarily relies on manual inspection to detect bridge defects. This method involves regular inspections by bridge maintenance personnel, using visual inspections, supplemented by equipment such as ultrasonic detectors, crack detectors, and cable tension meters, to detect bridge defects. This manual inspection method is time-consuming, labor-intensive, and inefficient, and often poses safety risks to bridge maintenance personnel.

[0059] (2) Methods based on manual features

[0060] To achieve automated detection of bridge defects, methods based on handcrafted features have emerged. These methods primarily rely on manually constructing image features (such as the morphology of different defects, gradient histograms, and other features) and feeding these features into basic machine learning classifiers (such as support vector machines (SVMs)) for classification and identification. While these manually selected features improve the performance of automated bridge defect detection, they typically require extensive preprocessing and feature engineering, which not only increases the workload but also makes it difficult for humans to design sufficiently robust features when faced with complex and unfamiliar problems. Furthermore, these manually designed features are often superficial, targeting shallow characteristics such as gradients, colors, and textures. This results in the limited application of handcrafted feature-based methods for bridge defect detection in complex environments.

[0061] (2) Traditional recognition and detection methods based on deep learning

[0062] Traditional deep learning-based recognition and detection methods primarily include the two-step approach, exemplified by Faster-RCNN, and the one-step approach, exemplified by SSD / YOLO. The two-step approach first generates candidate rectangular regions for an object using a shallow network, then fine-tunes these regions using a deep network. The one-step approach, on the other hand, uses a convolutional neural network that can simultaneously predict the positions and categories of multiple bounding boxes, enabling end-to-end object detection and recognition. Its greatest advantage is its speed.

[0063] Although the traditional recognition and detection method based on deep learning can realize automatic feature selection of the network and automatic detection of bridge defects, which improves the traditional bridge defect recognition method, it still has the problem of low detection rate in bridge defect recognition.

[0064] In summary, the main limitations of current bridge defect identification methods are as follows:

[0065] (1) The target detection models all act directly on the original images taken by the camera, but ignore the information of other modalities in the bridge damage image dataset, such as infrared images. The ignored information is complementary to the original image information and is very useful in damage identification.

[0066] (2) Due to various environmental factors, such as lighting, the original image information is invisible to humans, and traditional recognition methods cannot recognize it, resulting in a sharp drop in recognition errors.

[0067] (3) Traditional target detection requires anchor frames and is divided into many steps to identify a single modality. It does not integrate multimodal data well for effective bridge disease identification.

[0068] Based on the above difficulties, this application proposes a bridge defect identification method and device based on multimodal fusion. Through the infrared image and visible light image of the bridge to be inspected, based on the multimodal fusion framework, the data information of different modalities is comprehensively utilized to detect the defective area of ​​the bridge to be inspected.

[0069] Example 1:

[0070] This embodiment proposes a bridge disease identification method based on multimodal fusion, such as Figure 1 、 Figure 3 Shown, including:

[0071] Step S1: Acquire an infrared image, a visible light image, and a position code of the bridge to be inspected;

[0072] Step S2: inputting the infrared light image into a pre-trained first convolutional network to obtain a first image feature;

[0073] In this embodiment, the infrared light image is input into the first convolutional network to obtain the first image feature, such as Figure 2 、 Figure 3 As shown, specifically including:

[0074] Step S2.1: preprocessing the infrared image to obtain a preprocessing result;

[0075] Step S2.2: performing image enhancement on the preprocessing result to obtain an image enhancement result;

[0076] Step S2.3: Input the image enhancement result into a first convolutional network to obtain a first image feature.

[0077] In this embodiment, the first convolutional network is a convolutional network pre-trained on historical infrared images, specifically including: a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first fully connected layer, and a second fully connected layer;

[0078] A convolutional neural network is a deep feedforward neural network characterized by local connections and weight sharing. It is a representative algorithm for deep learning and excels at image processing, particularly related machine learning problems such as image recognition. It significantly improves various visual tasks, such as image classification, object detection, and image segmentation, and is currently one of the most widely used models. It primarily consists of convolutional layers, pooling layers, and fully connected layers. Convolutional layers primarily process features using specific convolution kernels; pooling layers downsample features to reduce parameters and extract significant features; and fully connected layers output the final feature sequence.

[0079] In this embodiment, the enhanced result of the 8-bit infrared image is input to the first convolutional layer. The input data dimension is ; Using a step size of 2 and a convolution kernel of The first convolution layer with a stride of 2 and a convolution kernel of The first pooling layer extracts features from the image enhancement result to obtain hidden layer features; the hidden layer features are input into a convolution kernel with a step size of 2 and a The second convolution layer has a stride of 2 and a convolution kernel of The second pooling layer is used to obtain a feature map; the feature map is input to the first fully connected layer, and the output of the first fully connected layer is input to the second fully connected layer to obtain the first image feature, which is an image feature sequence, such as Figure 4 shown.

[0080] Step S3: inputting the visible light image into a pre-trained second convolutional network to obtain a second image feature;

[0081] Similarly, the second convolutional network is a historical visible light image (three-channel image ) pre-trained convolutional network, in this embodiment, is a residual network (ResNet), specifically including a residual module and multiple convolution kernels connected to the residual module.

[0082] Step S4: superimposing the second image feature and the position code and inputting the superimposed ...

[0083] The encoder encodes the visible light image features. The decoder decodes M third image features based on the feature representation output by the encoder, and performs multimodal fusion with the infrared image features to output the final result. In this embodiment, the encoder is a Transformer encoder. Other encoders can also be used in specific implementations. However, the Transformer encoder and Transformer decoder in this embodiment have very good results. Taking the residual network ResNet50 as an example, the three-channel image Input the residual network ResNet50 to generate C low-dimensional features with width W and height H (C=2048, W= / 32 and H= / 32), that is, C two-dimensional features. Since these features will be passed to a Transformer encoder, each feature must be reformatted in a way that allows the encoder to process each feature as a sequence. This is achieved by flattening the feature matrix into HW vectors and then concatenating each vector, such as Figure 5 As shown. The dimension of the second image feature is Dimensionality reduction to , so that the dimension of the second image feature dimensionality reduction result is consistent with the dimension of the position code, and the second image feature dimensionality reduction result and the position code are superimposed to obtain a superposition result, where C is the number of second image features, W is the width of the second image feature, is the height of the second image feature; the superposition result is input into the Transformer encoder, such as Figure 6 As shown, the final encoding result is obtained, where the position encoding can be obtained through learning and training or through predefinition. In addition, the three-dimensional features of the second image feature are converted into two dimensions by using the matrix transformation function in pytorch. This transformation only reduces the dimension and does not change the values ​​in the matrix, that is, H*W is converted from a two-dimensional matrix into a one-dimensional vector with HW elements. In this embodiment, it is necessary to ensure that the element values ​​remain unchanged after dimensionality reduction.

[0084] Step S5: inputting the encoding result into the decoder to obtain the third image feature;

[0085] In this embodiment, the encoding result is input to the decoder and then output as M third image features, which represent the number of output prediction results. That is, the encoding result is input to the Transformer decoding module and M decoding module outputs are obtained respectively, such as Figure 7 shown.

[0086] Step S6: inputting the first image feature and the third image feature into a pre-trained multimodal fusion network to obtain a multimodal fusion feature;

[0087] In this embodiment, a multimodal fusion network is used to effectively fuse the features between infrared light images and visible light images, where multimodality refers to various forms of combinations of two or more modal data. The visible light image features are feature data that have been processed and refined by multi-layer convolutional neural networks and Transformer encoders and decoders, while the infrared light image features are feature data obtained by processing multi-layer convolutional neural networks. Taking the two as input, multimodal fusion features are obtained through a multimodal fusion network. Multimodal features make up for the shortcomings of incomplete information of single-modal features, realize information supplementation, provide more comprehensive and effective information for subsequent prediction results, improve the accuracy of prediction results, and improve the robustness of the prediction model.

[0088] The multimodal fusion network concatenates the output of the Transformer decoder and the infrared image features, that is, concatenates the first image features with the third image features. The first dimension of the first image features is equal to the first dimension of the third image features. The concatenated feature vector is input into the fully connected layer for feature fusion calculation to obtain a feature representation that fuses the deep information of the two. The concatenated features are input into the third fully connected layer of the multimodal fusion network to obtain multimodal fusion features. The third fully connected layer of the modal fusion network consists of two linear layers to obtain a deeper feature representation. For example: the output of the Transformer decoder, that is, the third image feature is 3×8, and the infrared feature, that is, the first image feature is 3×X. The two are concatenated to form a matrix of 3*(8+X), such as Figure 8 、 Figure 9 shown.

[0089] Step S7: Inputting the multimodal fusion features into a pre-trained neural network to predict bridge defects and obtain a bridge defect recognition result, which includes: whether a bridge defect exists and / or the type of bridge defect.

[0090] The neural network includes: a first feedforward neural network for bounding box prediction and a second feedforward neural network for category prediction; the first feedforward neural network and the second feedforward neural network are both pre-trained by historical multimodal fusion feature data, and the multimodal fusion features are input into the pre-trained neural network to predict bridge defects, thereby obtaining a bridge defect recognition result, including:

[0091] The multimodal fusion feature is input into the first feedforward neural network to obtain the multimodal fusion feature with the bridge disease boundary box The first feedforward neural network includes a first ReLU-activated perceptron, a second ReLU-activated perceptron, and a third ReLU-activated perceptron. The first ReLU-activated perceptron is connected to the second ReLU-activated perceptron, and the second ReLU-activated perceptron is connected to the third ReLU-activated perceptron. The second feedforward neural network includes a linear layer and a softmax prediction layer connected to the linear layer. In this embodiment, the output dimension of the first ReLU-activated perceptron is 1024, the output dimension of the second ReLU-activated perceptron is 256, and the output dimension of the third ReLU-activated perceptron is 4. The final output dimension 4 represents the position (x, y), height, and width of the center of the bounding box.

[0092] The multimodal fusion features with the bridge defect bounding box are input into the second feedforward neural network to obtain the bridge defect recognition results. .

[0093] Constructing a suitable loss function can ensure accurate multimodal fusion results. In this embodiment, the multimodal Transformer fusion detection framework uses a binary matching loss, which includes: whether the category is correct (category loss), whether the size of the bounding box with the bridge defect is appropriate (bounding box height and width loss), and whether the position (x, y) of the bounding box is appropriate (bounding box position loss). The loss in the multimodal Transformer fusion detection framework can be expressed as:

[0094]

[0095] N predictions, where N is greater than the number of objects (i.e., bridge defects) in an image. is a set of N real targets, where the set without a target is represented by Ø. is a set of N prediction targets, To predict the category of the bounding box, is the bounding box of the annotation, For the best matching predicted bounding box, generate an optimal bipartite match between the prediction and the annotation, and then optimize the loss of the feature bounding box , W is the weight coefficient for balancing the imbalance of the categories. In this embodiment, W=1. The specific steps are as follows:

[0096] (a) Find the optimal bipartite matching:

[0097] A predicted bounding box corresponds to a true bounding box. is a set of N real targets, where the set without target is represented by Ø. There are N prediction target sets, and we need to find the optimal bipartite matching :

[0098]

[0099] in, is in GT and index is Predicted matching loss. The best matching is calculated using the Hungarian algorithm.

[0100] (b) Compute the bounding box loss:

[0101]

[0102] in, is the categorical cross entropy loss, is the optimal bipartite matching, Using L1 loss and generalized intersection-over-union loss ,but:

[0103]

[0104] in, and is a hyperparameter, Calculate the L1 loss between the predicted bounding box and the true bounding box, is the generalized intersection-over-union loss, which is calculated as follows:

[0105]

[0106] in, For intersection-union ratio calculation, is the area of ​​the minimum enclosed region, that is, the area of ​​the smallest box that contains both the predicted bounding box and the true bounding box. Therefore, the latter term calculates the proportion of the area in the enclosed region that does not belong to the two bounding boxes to the enclosed region. This not only focuses on the overlapping area, but also other non-overlapping areas, which can better reflect the degree of overlap between the two.

[0107] In order to evaluate the effectiveness of the bridge defect recognition method based on multimodal fusion, the bridge defect recognition method provided by this application was evaluated. Thousands of manually annotated bridge defect image test data sets were obtained from the Internet, mainly including bridge surface crack disease images. In the above test data sets, multimodal data (visible light and infrared light images) accounted for 70%, and single-modal data accounted for 30%, that is, images of the infrared image modality of bridge defects were missing. The bridge defect recognition method based on multimodal fusion provided by this application was used to identify bridge defects in the test data set. The results of the test data set are as follows: Figure 10 As shown, the gray bounding box represents the recognition result of the bridge disease. Compared with the traditional bridge disease recognition method, the corresponding bridge disease recognition result in this application is more accurate.

[0108] This embodiment proposes a bridge defect identification method based on multimodal fusion. Specifically, the infrared image and visible light image of the bridge defect are input into the bridge defect network based on multimodal fusion pre-trained in this application. Based on the multimodal Transformer fusion framework, the feature information of the infrared light image and the visible light image is comprehensively utilized. It not only utilizes the advantages of infrared light images that do not require the help of external ambient light and are less affected by the environment, but also takes advantage of the intuitive and clear advantages of visible light images. The data features of the two are complementary and mutually usable, thereby improving the accuracy of the bridge defect detection task and enhancing the robustness of the detection model.

[0109] Example 2:

[0110] This embodiment proposes a bridge disease identification device based on multimodal fusion, such as Figure 11 Shown, including:

[0111] A data acquisition module is used to obtain infrared images, visible light images and position codes of the bridge defects to be detected;

[0112] A first image feature acquisition module, configured to input the infrared light image into a first convolutional network to obtain a first image feature;

[0113] A second image feature acquisition module, configured to input the visible light image into a second convolutional network to obtain a second image feature;

[0114] an encoding module, configured to superimpose the second image feature and the position code and input the superimposed ...

[0115] A third image feature acquisition module, configured to input the encoding result into a decoder to obtain a third image feature;

[0116] a feature fusion module, configured to input the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature;

[0117] The result prediction module is used to input the multimodal fusion features into the neural network for prediction to obtain the bridge disease identification results

[0118] Among them, the data acquisition module is connected to the first image feature acquisition module, the second image feature acquisition module is connected to the encoding module, the encoding module is connected to the third image feature acquisition module, the first image feature acquisition module and the third image feature acquisition module are respectively connected to the feature fusion module, and the feature fusion module is connected to the result prediction module.

[0119] This embodiment proposes a bridge defect identification device based on multimodal fusion. Infrared image features are extracted through a first image feature acquisition module, and visible light image features are extracted through a second image feature acquisition module, an encoding module, and a third image feature acquisition module. The infrared image features and visible light image features are fused with each other through a feature fusion module to output multimodal fusion features. The multimodal fusion features are input into a neural network for prediction to obtain bridge defect identification results.

[0120] Example 3:

[0121] This embodiment provides a computer program product, including: a computer program / instructions, which, when executed by a processor, implement the steps of the bridge defect identification method based on multimodal fusion.

[0122] Example 4:

[0123] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory stores instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the bridge defect identification method based on multimodal fusion.

[0124] The electronic device can be a mobile phone, computer, or tablet computer, and includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the bridge defect identification method based on multimodal fusion as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0125] The processor is configured to execute all or part of the steps of the bridge defect identification method based on multimodal fusion as described in the above-mentioned embodiment. The memory is configured to store various types of data, such as instructions for any application or method in the electronic device, as well as data related to the application.

[0126] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, and is used to execute the bridge defect identification method based on multimodal fusion described in the above embodiment.

[0127] Example 5:

[0128] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed, a processor executes the bridge defect identification method based on multimodal fusion.

[0129] The functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. If the functions are implemented as software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0130] Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the bridge defect identification method based on multimodal fusion described in each embodiment of the present application.

[0131] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which a computer program is stored. When the computer program is executed by the processor, the various steps of the bridge defect identification method based on multimodal fusion described above can be implemented.

[0132] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0133] The scope of protection of the present disclosure is not limited to the above-described embodiments. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the scope and spirit of the present disclosure. If such modifications and variations fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include such modifications and variations.

Claims

1. A bridge defect identification method based on multimodal fusion, characterized in that: include: Obtain infrared light images, visible light images, and position codes of the bridge to be inspected; Inputting the infrared light image into a pre-trained first convolutional network to obtain a first image feature; Inputting the visible light image into a pre-trained second convolutional network to obtain a second image feature; Superimposing the second image feature and the position code and inputting the superimposed image feature and the position code into an encoder to obtain an encoding result; Inputting the encoding result into the decoder to obtain the third image feature; Inputting the first image feature and the third image feature into a pre-trained multimodal fusion network to obtain a multimodal fusion feature; Inputting the multimodal fusion features into a pre-trained neural network to predict bridge defects and obtain a bridge defect recognition result, wherein the bridge defect recognition result includes: whether a bridge defect exists and / or the type of the bridge defect; The step of superimposing the second image feature and the position code and inputting the superimposed image feature and the position code into an encoder to obtain an encoding result includes: The dimension of the second image feature Dimensionality reduction to , so that the dimension of the second image feature dimensionality reduction result is consistent with the dimension of the position code, and the second image feature dimensionality reduction result and the position code are superimposed to obtain a superposition result, wherein, is the number of second image features, is the width of the second image feature, is the height of the second image feature; Inputting the superposition result into the encoder to obtain the encoding result; Inputting the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature includes: concatenating the first image feature and the third image feature, wherein the first dimension of the first image feature is equal to the first dimension of the third image feature; The concatenated features are input into the third fully connected layer of the multimodal fusion network to obtain the multimodal fusion features.

2. The bridge defect identification method based on multimodal fusion according to claim 1 is characterized in that: Inputting the infrared light image into a first convolutional network to obtain a first image feature includes: Preprocessing the infrared light image to obtain a preprocessing result; Performing image enhancement on the preprocessing result to obtain an image enhancement result; The image enhancement result is input into a first convolutional network to obtain a first image feature.

3. The bridge defect identification method based on multimodal fusion according to claim 2 is characterized in that: The first convolutional network includes: a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first fully connected layer, and a second fully connected layer; inputting the image enhancement result into the first convolutional network to obtain the first image feature includes the following steps: The convolution kernel is used with a stride of 2. The first convolution layer with a stride of 2 and a convolution kernel of The first pooling layer performs feature extraction on the image enhancement result to obtain hidden layer features; The hidden layer features are input into the convolution kernel with a stride of 2. The second convolution layer has a stride of 2 and a convolution kernel of The second pooling layer obtains the feature map; The feature map is input into the first fully connected layer, and the output result of the first fully connected layer is input into the second fully connected layer to obtain the first image feature.

4. The bridge defect identification method based on multimodal fusion according to claim 1 is characterized in that: The second convolutional network includes a residual module and a plurality of convolution kernels connected to the residual module.

5. The bridge defect identification method based on multimodal fusion according to claim 1 is characterized in that: The neural network includes: a first feedforward neural network for bounding box prediction and a second feedforward neural network for category prediction; the multimodal fusion features are input into the pre-trained neural network to predict bridge defects and obtain bridge defect recognition results, including: Inputting the multimodal fusion features into a first feedforward neural network to obtain a multimodal fusion feature with a bridge disease bounding box; The multimodal fusion features with bridge defect bounding boxes are input into the second feedforward neural network to obtain the bridge defect recognition results.

6. A bridge defect identification device based on multimodal fusion, characterized in that: include: A data acquisition module is used to obtain infrared images, visible light images and position codes of the bridge defects to be detected; A first image feature acquisition module, configured to input the infrared light image into a first convolutional network to obtain a first image feature; A second image feature acquisition module, configured to input the visible light image into a second convolutional network to obtain a second image feature; an encoding module, configured to superimpose the second image feature and the position code and input the superimposed ... A third image feature acquisition module, configured to input the encoding result into a decoder to obtain a third image feature; a feature fusion module, configured to input the first image feature and the third image feature into a multimodal fusion network to obtain a multimodal fusion feature; A result prediction module is used to input the multimodal fusion features into a neural network for prediction to obtain a bridge disease identification result; The encoding module is used for: The dimension of the second image feature Dimensionality reduction to , so that the dimension of the second image feature dimensionality reduction result is consistent with the dimension of the position code, and the second image feature dimensionality reduction result and the position code are superimposed to obtain a superposition result, wherein, is the number of second image features, is the width of the second image feature, is the height of the second image feature; Inputting the superposition result into the encoder to obtain the encoding result; The feature fusion module is used to: concatenating the first image feature and the third image feature, wherein the first dimension of the first image feature is equal to the first dimension of the third image feature; The concatenated features are input into the third fully connected layer of the multimodal fusion network to obtain the multimodal fusion features.

7. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the bridge defect identification method based on multimodal fusion according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory stores instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the bridge defect identification method based on multimodal fusion according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that It stores executable instructions, which, when executed, enable the processor to execute the bridge defect identification method based on multimodal fusion according to any one of claims 1 to 5.

Images