Bridge disease detection method and system based on improved multi-modal visual language model

Abstract

The invention discloses a bridge disease detection method and system based on an improved multi-mode visual language model, and the method comprises the steps: obtaining a historical bridge original image which comprises a bridge cantilever casting section, a cast-in-place section, a web, a diaphragm plate, a closure section and a concrete box girder; constructing an improved multi-modal visual language target detection model; based on the bridge original image, training the improved multi-modal visual language target detection model to obtain a trained improved multi-modal visual language target detection model as a bridge disease detection model; obtaining a to-be-detected bridge image, and dividing the to-be-detected bridge image into a plurality of overlapped image blocks based on an SAHI sliding window mechanism; detecting each image block in a local region based on the bridge disease detection model to obtain a region detection result; and on the basis of the detection result of each region, generating a global detection result in a splicing and fusion mode.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge defect detection, and in particular to a bridge defect detection method and system based on an improved multimodal visual language model. Background Art

[0002] As a world power in highway and bridge construction, my country has the world's longest highway network with a total mileage of more than 177,000 kilometers, more than 800,000 highway bridges, and many complex and large bridge structure systems. These bridges play a vital role in my country's transportation network.

[0003] However, as the aging of built bridges becomes increasingly serious and the bridge load caused by the increase in traffic flow is too large, once encountering extreme weather and geological movements, some bridges will suffer serious collapse accidents, seriously threatening the safety of people's lives and property. Therefore, it is particularly important to carry out daily inspection and maintenance of highway bridges. Traditional bridge inspection mainly relies on manual inspection, and inspectors evaluate the bridge structure through visual inspection or simple instruments. However, this method is not only time-consuming and labor-intensive, but also easily affected by human subjective factors, resulting in inaccurate detection results. Existing studies have proposed using drones or wall-climbing robots as data collection terminals for bridge disease information, combined with traditional deep learning and machine learning intelligent detection technologies to detect and monitor bridge disease information. Although these technologies can automatically identify and locate common diseases on the surface of bridges, such as cracks, corrosion, and exposed steel bars, in actual use, they are not capable of processing noise in complex scenarios, and their recognition and detection capabilities are heavily dependent on predefined category data sets. The quality of the pre-trained data set restricts the ability of the model in large-scale scene applications, and it is impossible to effectively monitor the disease feature information outside the data set. The understanding and generalization ability of the actual model is seriously insufficient. In addition, the existing models cannot accurately identify ultra-high-resolution images during field use. The violent scaling of images to the pre-trained size will cause serious distortion and deformation, resulting in the inability to perform specific calculations based on camera parameters for the subsequent analysis of the development of the disease. In addition, the drones or wall-climbing robots that the existing models rely on are at risk of crashing due to vortex cyclones during operation, resulting in high actual use costs and risks. These traditional deep learning machine detection methods and manual detection methods have detection blind spots in bridge detection, that is, areas that traditional methods cannot detect are often key areas for bridge detection.

[0004] Therefore, how to conduct refined detection in areas that cannot be effectively covered by traditional artificial intelligence detection methods in a low-cost, high-efficiency, safe and reliable manner is an important issue that needs to be solved urgently.

[0005] Deep learning-based artificial intelligence models are currently the focus of research in the field of bridge disease detection. Using AI intelligent models to replace manual detection can significantly improve the efficiency and accuracy of detection, reduce labor costs, and reduce safety risks during the detection process. Regular bridge inspections can effectively ensure the safety of transportation and the actual service life of bridges. Although the existing deep learning-based bridge inspection technology has achieved certain results, it still cannot be widely promoted and applied in actual use.

[0006] First, the existing deep learning defect detection algorithm has weak detection capabilities in complex scenes of the bridge belly, is seriously affected by strong light or shadows, has poor denoising capabilities for similar interference caused by structural connections, and the defect feature information at a certain moment cannot be detected by the algorithm. Then, the CNN network model used in the existing detection cannot effectively identify the bridge defect feature information outside the pre-training data set, has insufficient recognition capabilities for defect feature information that did not appear during the training process, cannot quickly adapt to the object detection needs of new scenes, and is heavily dependent on manually annotated data information. The dependence on predefined and trained object categories limits the applicability of the existing model in more scenes. Secondly, the model used in the existing detection method cannot calculate the same image size for defect images of different sizes and resolutions. The existing method loses the defect feature information of the ultra-high resolution image by compressing the image to the pre-training image size for detection. The image is seriously inconsistent with the equipment calibration information, resulting in no unified standard for quantitative defect change monitoring in the later defect feature processing process. Then, the existing bridge disease detection algorithm usually requires manual control of complex mechanical equipment such as drones for data collection and analysis during use. The operation of the equipment is heavily dependent on the experience of technicians. In addition, since the detection focus is mostly located in the area with severe signal interference on the belly of the bridge, the mechanical equipment will be affected by complex weather and there will be great safety risks. Finally, the terminal equipment developed based on the existing deep learning bridge disease detection algorithm is mainly drones and wall-climbing robots. The purchase, use and maintenance costs of the machinery and equipment are high. The precision machinery and equipment have strict requirements on the working environment. The cumbersome detection process leads to limited detection frequency, and the bridge health status is not updated in time.

[0007] In view of this, this application is hereby filed. Summary of the invention

[0008] The technical problem to be solved by the present invention is that the existing bridge defect detection methods have problems such as inaccurate defect identification due to the insufficient detection range. The purpose is to provide a bridge defect detection method and system based on an improved multimodal visual language model, design an improved multimodal visual language target detection model, and train to obtain a bridge defect detection model; the model improves the backbone network, feature fusion network and detection head of the original Yolov10s target detection model to introduce more semantic cues, with a wide detection range, so that bridge defect detection is more accurate and efficient, and real-time defect detection of key areas of the bridge is achieved.

[0009] The present invention is achieved through the following technical solutions:

[0010] In a first aspect, the present invention provides a bridge disease detection method based on an improved multimodal visual language model, the method comprising:

[0011] Acquire a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilever cast section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge;

[0012] Construct an improved multimodal visual language target detection model; based on the original bridge image, train the improved multimodal visual language target detection model to obtain the trained improved multimodal visual language target detection model as a bridge disease detection model;

[0013] Acquire a bridge image to be detected, and divide the bridge image to be detected into multiple overlapping blocks based on the SAHI sliding window mechanism;

[0014] Based on the bridge disease detection model, each block is detected in the local area to obtain the regional detection result; based on each regional detection result, the global detection result is generated by splicing and fusion.

[0015] Furthermore, the improved multimodal visual-language target detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual-language fusion module on the basis of the existing Yolov10s target detection model.

[0016] Furthermore, the improved multimodal visual language target detection model includes a feature extraction backbone network Backbone, a feature fusion network Neck and a detection head Head connected in sequence;

[0017] The feature extraction backbone network Backbone introduces an optimized multi-layer link self-attention mechanism based on the existing feature extraction backbone network, enabling the feature extraction network to capture global context information at different scales and levels and extract multi-scale visual feature information;

[0018] The feature fusion network Neck replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of disease features is enhanced by combining with pre-defined text feature description information. The feature fusion network outputs the fused image feature information and text feature information;

[0019] The head is detected by introducing an overall loss function to assist the model in realizing negative feedback network training of text feature information and image feature information; the overall loss function includes the first loss function and the second loss function. The first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

[0020] Furthermore, the feature extraction backbone network Backbone includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor and a fifth feature extractor connected in sequence, and the fifth feature extractor is a feature extraction module that integrates an optimized multi-layer link self-attention mechanism into a convolutional neural network;

[0021] The fifth feature extractor includes a first standard convolution layer, a depth-separable convolution layer, a first normalization layer, an optimized multi-layer link self-attention mechanism layer, a second normalization layer, and an MLP feature map processing layer;

[0022] The first standard convolution layer is connected to the depthwise separable convolution layer, the output of the first standard convolution layer is fused with the output of the depthwise separable convolution layer and then connected to the first normalization layer, the first normalization layer is connected to the optimized multi-layer link self-attention mechanism layer, the output of the first standard convolution layer is fused with the output of the depthwise separable convolution layer and then fused with the output of the optimized multi-layer link self-attention mechanism layer and connected to the second normalization layer, the second normalization layer is connected to the MLP feature map processing layer, and the MLP feature map processing layer is then connected to the subsequent standard convolution layer.

[0023] Furthermore, the execution process of the optimized multi-layer link self-attention mechanism layer is as follows:

[0024] Taking the feature map as input information, divide the feature map into S*S different regions;

[0025] Define the eigenvector X r , obtain the query value matrix Q, key value matrix K and value matrix V through linear mapping, and calculate the overall attention Attention(Q,K,V);

[0026] By building a connectivity graph to locate the relevance of a given area, the attention relationship between different areas is obtained;

[0027] Perform regional averaging on the query value matrix Q and value matrix V of each region, and derive and calculate the dot product corresponding to each region to obtain the adjacency matrix A r ;

[0028] By filtering out the adjacency matrix A at the coarse-grained level r The area with the smallest correlation is used to retain the top n areas with the highest correlation, and the routing index matrix I is obtained. r ;

[0029] The coarse-grained level of attention is calculated through the routing index matrix between different regions, and all relevant key-value matrices K and value matrices V in the attention region are combined to obtain the cluster K g With V g ;

[0030] K for clustering g With V g Perform attention calculation and local context enhancement LCE(V) derivation, and output the calculation result O=Attention(Q,K g ,V g )+LCE(V).

[0031] Furthermore, the image annotation is redefined as region-text pair in the feature fusion network. where t i is the bounding box B i The corresponding text information is a kind of object description text information about the detection target, instead of using a single category label as a pre-training method; in this way, the predicted bounding box {B k} and its corresponding object embedding {e k},{e k} represents the embedded representation of the target object description information output by the model.

[0032] Furthermore, the expression of the overall loss function is:

[0033] Loss(I)=Loss con +λ I ·Loss WIoU_v3

[0034]

[0035] Loss WIoU_v3 =γ·R WIoU ·Loss IoU

[0036]

[0037] Among them, λ I Represents the indicator factor, which is set to 1 when the input image sample I comes from detection or positioning data, and is set to 0 when the input image sample I comes from irrelevant annotation text;

[0038] Loss WIoU_v3 is the first loss function; Loss IoU represents the general anchor box loss function; W g With H g are the width and height of the smallest outer bounding box respectively; w and h represent the width and height of the prediction box in the target detection result respectively; w gt and h gt Respectively represent the width and height of the real box in the artificial label data; α and δ represent two adaptive hyperparameter constants; β represents the aggregate outlier degree;

[0039] Loss con is the second loss function; N represents the total number of samples of image-text pairs; exp() is the exponential function;

[0040] Similarity(e i ,w i ) is the region image representation e i And the corresponding text embedding representation w i The similarity between

[0041] Similarity(e i ,w j ) is the region image representation e i With the overall text embedding representation w j The similarity between j Refers to each text representation in the entire text collection.

[0042] In a second aspect, the present invention further provides a bridge disease detection system based on an improved multimodal visual language model, the system comprising:

[0043] An acquisition unit is used to acquire a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilevered section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge;

[0044] The model building and training unit is used to build an improved multimodal visual language target detection model; based on the original bridge image, the improved multimodal visual language target detection model is trained to obtain the trained improved multimodal visual language target detection model as a bridge disease detection model;

[0045] An image division unit, used for acquiring an image of a bridge to be detected, and dividing the image of the bridge to be detected into a plurality of overlapping blocks based on a SAHI sliding window mechanism;

[0046] The detection unit is used to detect each block in a local area based on the bridge disease detection model to obtain a regional detection result; based on each regional detection result, a global detection result is generated by splicing and fusion.

[0047] Furthermore, the improved multimodal visual language target detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual language fusion module based on the existing Yolov10s target detection model;

[0048] The improved multimodal visual language target detection model includes a feature extraction backbone network Backbone, a feature fusion network Neck and a detection head Head connected in sequence;

[0049] The feature extraction backbone network Backbone introduces an optimized multi-layer link self-attention mechanism based on the existing feature extraction backbone network, enabling the feature extraction network to capture global context information at different scales and levels and extract multi-scale visual feature information;

[0050] The feature fusion network Neck replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of disease features is enhanced by combining with pre-defined text feature description information. The feature fusion network outputs the fused image feature information and text feature information;

[0051] The head is detected by introducing an overall loss function to assist the model in realizing negative feedback network training of text feature information and image feature information; the overall loss function includes the first loss function and the second loss function. The first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

[0052] In a third aspect, the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned bridge defect detection method based on the improved multimodal visual language model is implemented.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] The present invention is based on a bridge defect detection method and system based on an improved multimodal visual language model, designs an improved multimodal visual language target detection model, and trains to obtain a bridge defect detection model; the model improves the backbone network, feature fusion network and detection head of the original Yolov10s target detection model to introduce more semantic cues, with a wide detection range, making bridge defect detection more accurate and efficient, and realizing real-time defect detection in key areas of the bridge. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0056] Figure 1 This is the overall research idea diagram of the present invention;

[0057] Figure 2 This is the overall design structure diagram of the improved multimodal visual language target detection model of the present invention;

[0058] Figure 3 This is a flow chart of a bridge defect detection method based on an improved multimodal visual language model of the present invention;

[0059] Figure 4 The structure diagram of the PSBRA module after optimization of the present invention;

[0060] Figure 5 This is a schematic diagram of the Bi-level routing attention framework mechanism based on bridge disease characteristics of the present invention;

[0061] Figure 6 The overall architecture diagram of the VLPAN module of the present invention;

[0062] Figure 7 This is a schematic diagram of the edge reasoning workflow combined with the SAHI sliding window mechanism of the present invention;

[0063] Figure 8 It is a structural block diagram of the bridge disease detection system based on the improved multimodal visual language model of the present invention;

[0064] Fig. 9 It is a schematic diagram of the structure of the detection device for key bridge disease areas of the present invention. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0066] The overall research idea of ​​the present invention is as follows Figure 1 As shown, it mainly includes a bridge disease detection method based on an improved multimodal visual language model and an automatic data acquisition and storage system based on an optimized SAHI sliding window mechanism and an enhanced information retrieval mechanism.

[0067] First, based on the original Yolov10s general object detection model, an optimized multi-layer link self-attention mechanism (PSBRA) is introduced into its BackBone network to enhance the model's feature extraction and analysis capabilities for images; secondly, a CLIP-based visual language multimodal aggregation network mechanism (VLPAN) is introduced to perform cross-modal feature matching and fusion of feature information of different scales with semantic text information, and to expand the model's understanding of disease feature information based on text prompt information, thereby generating text analysis information of bridge disease conditions; finally, the WIoU_v3 loss function is used to replace the original CIOU binary cross entropy loss to improve the bounding box regression performance of the model and enhance the detection capability of the model. The regional text contrast function is used to realize cross-modal image text training, and the text information feature description is used to assist in processing the disease feature information of unknown scenes, thereby enhancing the model's understanding of text information and image features.

[0068] On the other hand, based on the self-powered image data acquisition and transmission equipment, a control center with SAHI sliding window mechanism as the core is designed to perform image standardization detection on the collected data, retain the ultra-high resolution and size of the original ultra-high image, and avoid the loss of actual feature information caused by image compression and size reorganization. In view of the problems of blind spots in manual detection of traditional bridge defects, safety hazards of drones flying on the belly of bridges, limited detection areas of crawling robots, low detection capabilities of existing CNN algorithm models (such as the original Yolov series model, RNN model and Deeplab model, etc.) for untrained data information, and weak detection capabilities of existing detection systems when processing multimodal feature information, the text information of the defect feature description is used to enhance the model's understanding of image feature information, assist the model in processing defect recognition under untrained conditions, reduce the cost of algorithm model reconstruction and multiple training, and achieve real-time defect detection of key areas of bridges with a low-cost, high-efficiency and safer automated monitoring technology.

[0069] Design point 1: Bridge damage detection based on improved multimodal visual language model

[0070] The present invention is mainly based on the improvement of the original Yolov10s target detection model. Although the existing bridge disease detection algorithm has achieved some results, in actual use, it is not capable of detecting bridge diseases that did not appear in the training process, the model generalization ability is poor, and there are problems such as serious influence of light changes, noise interference of similar feature information, and poor detection accuracy of feature information. It cannot meet the daily detection needs under complex scenes under the belly of the bridge. Therefore, the present invention aims at the original feature extraction backbone network Backbone, feature fusion network Neck, and detection head of the model. According to the needs of actual business application scenarios, the present invention conducts innovative improvements in important aspects such as enhancing feature extraction and global context capture capabilities, integrating text semantic information for detection enhancement, and reducing sensitivity to abnormal samples, so as to meet the engineering technical requirements and practical needs of bridge disease detection. The overall design structure of the improved multimodal visual language target detection model is as follows: Figure 2 shown.

[0071] Design point 2: Based on SAHI sliding window mechanism

[0072] In order to solve the problem of excessive consumption of video memory if ultra-high resolution images are directly input into the original model in bridge defect detection, and the model will lose detailed information due to image scaling. The different scaling ratios of different image sizes make it extremely difficult to calculate the length and width information of the defects in the later stage. Therefore, in order to retain the detailed information and image size characteristics of the image as much as possible, we designed an edge reasoning structure combined with the SAHI (Slicing Aided Hyper Inference) sliding window mechanism. While keeping the length and width scale characteristics and image size of the original image unchanged, the model's detection ability for tiny defect targets in ultra-high pixel images is enhanced, and a unified window size is set for a standardized block detection process to reduce complex background noise interference. In the process of bridge defect detection, the original high-definition image usually contains a lot of detailed information, but due to the high resolution, direct full-image detection often leads to a decline in model performance, especially when detecting small defects, it is easy to miss detection. Most importantly, the existing models also force compression of non-pre-trained image sizes. Especially when the image sizes are inconsistent, the scaling of the existing models will cause serious image distortion. Therefore, through the SAHI sliding window mechanism, large-size images are dynamically divided into multiple overlapping tiles in proportion to form multiple small fragments suitable for the model input size for block-by-block detection. Each tile is independently detected in a local area, and a global detection result is generated by splicing and fusion, thereby improving the accuracy of the overall detection. This dynamic slicing mechanism ensures that each image can be processed effectively and avoids information loss due to image scaling. At the same time, SAHI can process images of multiple resolutions without manually resizing each image.

[0073] Example 1

[0074] like Figure 3 As shown, the present invention is based on an improved multimodal visual language model for bridge disease detection method, the method comprising:

[0075] Step 1, obtaining a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilevered section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge;

[0076] Step 2, constructing an improved multimodal visual language target detection model; based on the original bridge image, training the improved multimodal visual language target detection model to obtain a trained improved multimodal visual language target detection model as a bridge disease detection model;

[0077] In this embodiment, the improved multimodal visual language target detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual language fusion module on the basis of the existing Yolov10s target detection model.

[0078] Specifically, the improved multimodal visual language target detection model includes a feature extraction backbone network Backbone, a feature fusion network Neck and a detection head Head connected in sequence; the details are as follows:

[0079] First, the feature extraction network (Backbone) is the basic part of the entire model. After preprocessing the data information at the general input end, in order to improve the model's ability to extract disease feature information in complex bridge scenes, based on the existing feature extraction backbone network, the present invention introduces an optimized multi-layer link self-attention mechanism PSBRA module.

[0080] The PSBRA module is a feature extraction module that integrates the multi-layer link self-attention mechanism into the convolutional neural network. Compared with the original Yolov10s network model that can only extract features within the local receptive field, PSBRA is a feature extraction functional structure based on the self-attention mechanism. It can establish more effective contextual relationships between features at different levels, capture long-distance dependencies between features in a global scope, and enhance the ability to capture long-distance dependent features.

[0081] The PSBRA module introduces the link self-attention mechanism at multiple levels, so that the feature extraction network can capture global context information at different scales and levels. In the actual bridge defect detection process, the feature representation ability of the defect feature information at a single scale is weak, and the local detail features are not obvious. The original capture method cannot effectively extract its key feature information. Therefore, the improved model of the present invention can not only focus on local details, but also integrate global information at different scales, so as to understand the image content more comprehensively.

[0082] PSBRA establishes contextual relationships between features at different levels through the self-attention mechanism, so that the model can effectively distinguish the disease feature information from the global feature context information when processing complex scenes (such as bridge shadows, strong light interference, or similar features at structural connections), and has the ability to obtain disease feature information more accurately. By strengthening this global contextual relationship, PSBRA reduces the sensitivity to noise and interference information generated by irrelevant areas of bridge diseases during feature extraction, and improves the robustness of the model.

[0083] Compared with the original feature extraction method, PSBRA is not only more sensitive in the spatial dimension, but also can integrate information in a larger range, especially in detecting cracks in the shadow area of ​​the bridge belly, detecting corrosion under strong light, and identifying spalling areas under multi-noise interference background. All of these require the model to have more accurate feature extraction capabilities and assist the subsequent feature aggregation network to learn the detail features more efficiently. Therefore, in order to achieve the refined recognition of disease features in dim environments in key bridge detection areas, the PSBRA module structure introduced in the Backbone network is as follows: Figure 4 and Figure 5 shown.

[0084] In order to be compatible with the backbone network of the original Yolov10s network model, a functional module named PSBRA is designed, and its overall structure is as follows Figure 4 As shown in the figure, it consists of DWConv deep separable convolution, LayerNorm layer normalization processing, Bi-level routing attention mechanism layer, MLP multi-layer perceptron, and information fusion processing is performed through "+". DWConv, LN and MLP are all commonly used feature map processing techniques, mainly for further processing of input and output information, and are used to adapt the original Yolov10s network model. Figure 4 The structure that has changed is on the left side, while the structure on the right side has not changed.

[0085] Specifically, the feature extraction backbone network Backbone includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor and a fifth feature extractor connected in sequence, and the fifth feature extractor is a feature extraction module that integrates an optimized multi-layer link self-attention mechanism into a convolutional neural network;

[0086] The fifth feature extractor includes a first standard convolution layer, a depth-separable convolution layer, a first normalization layer, an optimized multi-layer link self-attention mechanism layer, a second normalization layer, and an MLP feature map processing layer;

[0087] The first standard convolution layer is connected to the depth-separable convolution layer. The output of the first standard convolution layer is fused with the output of the depth-separable convolution layer and then connected to the first normalization layer. The first normalization layer is connected to the optimized multi-layer link self-attention mechanism layer. The output of the first standard convolution layer is fused with the output of the depth-separable convolution layer and then fused with the output of the optimized multi-layer link self-attention mechanism layer and connected to the second normalization layer. The second normalization layer is connected to the MLP feature map processing layer, and the MLP feature map processing layer is then connected to the subsequent standard convolution layer, that is, Figure 4 The structure on the middle right side will not be described in detail here.

[0088] The core of the PSBRA module designed by the present invention is the Bi-level routing attention framework (i.e., the Bi-level routing attention attention mechanism layer, i.e., the optimized multi-layer link self-attention mechanism layer). The present invention designs the framework mechanism principle for the characteristics of bridge diseases as follows: Figure 5 shown.

[0089] The Bi-level routing attention framework takes the feature map as input information. Based on the original attention mechanism, the input feature data is first encoded to obtain the feature vector X, X∈R H×W×C , R, W, H, C are the batch size, width, height and number of channels of the feature map. The feature vector is defined by dividing the feature map into S*S different regions. Then, we can obtain Where Q represents the query value matrix, K represents the key value matrix, and V represents the value matrix. To prevent the gradient of subsequent results from disappearing, d K To represent the dimension of matrix K, the overall attention mechanism is derived and calculated as shown in Formula 1:

[0090]

[0091] Then, a connectivity graph is established to locate the correlation of a given region and obtain the attention relationship between different regions. The query value matrix Q and the value matrix V of each region are averaged, and the dot product corresponding to each region is derived to obtain the adjacency matrix A. r , and its derivation and calculation method is shown in Formula 2:

[0092] A r =Q r (K r ) T (2)

[0093] By filtering out A at the coarse-grained level r The area with the smallest correlation is used to retain the top n areas with the highest correlation, and the routing index matrix I is obtained. r , and its calculation method is shown in Formula 3:

[0094] I r =TopnIndex(A r ) (3)

[0095] Then the coarse-grained attention is calculated through the routing index matrix between different regions. For Q in a region i, the key-value pairs of n regions that it will notice are indexed as Then, all relevant K and V in the attention area are collected and K is calculated by the following formula 4: g With V g ,

[0096]

[0097] Finally, the K g With V g The attention operation and local context enhancement LCE(V) are derived, where LCE(·) represents the general local enhancement function, and finally the calculation result O is output. The calculation method is shown in Formula 5:

[0098] O=Attention(Q,K g ,V g )+LCE(V) (5)

[0099] Second, the feature fusion network Neck replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of disease features is enhanced by combining with pre-defined text feature description information. The feature fusion network outputs the fused image feature information and text feature information.

[0100] The neck network Neck is the feature fusion part of the entire algorithm model. In order to ensure that the trained neural network model can still effectively detect the newly emerged bridge disease feature information in other scenarios, improve the model's ability to recognize disease features in more unknown changing scenarios, and strengthen the model's ability to understand disease feature information by combining the description information prompted by natural language, and solve the problem that the recognition ability of the traditional YOLO series model is limited by the pre-training data set, this patent introduces the optimized VLPAN cross-modal visual language fusion module to replace the original feature fusion network.

[0101] The VLPAN module is a cross-modal feature fusion network based on image features and text semantic information. The feature extraction network sends the extracted multi-scale visual feature information to the VLPAN module, aggregates the multi-scale features from different feature layers, and enhances the recognition ability of disease features by combining them with pre-defined text feature description information (such as "a dark black horizontal crack is in the middle of the gray-white bridge pier."). Therefore, compared with traditional target detection methods, image annotation is redefined as region-text pairs. where t i is the bounding box B i The corresponding text information is a kind of object description text information about the detection target, instead of using a single category label as a pre-training method. In this way, the predicted bounding box {B k} and its corresponding object embedding {e k},{e k} represents the embedded representation of the target object description information output by the model. In order to solve the problem of mismatch between the vector features of text information and image information, a text encoder W = TextEncoder(T)∈R based on CLIP pre-training is designed. C×D , where C is the number of nouns and adjectives, D is the embedding dimension, and the input predefined text information t i Coded as w j Embedding representation, w j ∈W represents the jth text embedding, that is, the input text description information will be converted into the form of vocabulary embedding representation, and the image feature information obtained from the multi-scale feature extraction network can be fused and analyzed with the vocabulary embedding representation, and the similarity between the image detection object and the text embedding is calculated to match the image feature information and the text feature information. The calculation formula is shown in 6.

[0102] Similarity k,j =α·L2-Norm(e k )·L2-Norm(w j ) T +β (6)

[0103] Among them, L2-Norm represents L2 normalization calculation, α is a linear scaling factor, β is an offset factor, and the high similarity calculation result is considered to be a positive sample. The positive sample pairing result helps the model learn to obtain correct disease feature information. Through this method, while capturing the global context and long-distance information dependency of disease feature information, it can effectively solve the problem of pairing training of text information and image information. The overall architecture of the VLPAN module designed by the present invention is as follows Figure 6 shown.

[0104] Among them, C3, C4 and C5 represent the multi-scale feature maps output by the backbone network, P3, P4 and P5 represent the feature pyramid after information fusion, P3 corresponds to the high-resolution feature map information, P4 corresponds to the medium-resolution feature map information, and P5 corresponds to the low-resolution feature map. {P3, P4, P5} provides image feature information for subsequent detection of the head, and image-aware embedding provides text feature information for subsequent detection of the head.

[0105] By introducing the text-guided T-CSPlayer (text-part cross-stage connection layer) and image pooling attention mechanism components, the interaction between the bridge disease image features and text features is further enhanced, and the model's representation ability based on visual semantics is improved. On the basis of the original CSPlayer, text-guided information is introduced for multimodal information fusion. For a given text vocabulary embedding w j And image feature representation X l ∈R H×W×D (l∈{3,4,5}), in the last DarknetBottleneck part, max-sigmoid attention is used to aggregate text features into image features. The DarknetBottleneck part is the component structure of the original model, and its calculation method is shown in Formula 7.

[0106]

[0107] in represents the updated image feature representation, which is connected with the cross-stage feature information as the output result, and δ represents the sigmoid function. At the same time, the image pooling attention mechanism is used to promote the image perception mechanism to strengthen the text embedding representation ability, and the aggregated image feature information is used to further update the text embedding representation. The maximum pooling operation is performed on the multi-scale feature information output by T-CSPLayer to obtain a 3×3 area, thereby obtaining 27 blocks of vector feature representation. represents the image-aware embedding representation obtained by dividing the image into small blocks, where the update calculation of the image-aware embedding is shown in Formula 8,

[0108]

[0109] in Represents the updated text feature representation, and MultiHead-Attention represents the general attention calculation function based on the Transformer model.

[0110] Third, the head detection is achieved by introducing an overall loss function to assist the model in realizing negative feedback network training of text feature information and image feature information; the overall loss function includes the first loss function and the second loss function. The first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

[0111] Detecting the head is the predictive analysis part of the whole model. In order to improve the detection accuracy of the neural network model and help the model converge faster, the present invention introduces the improved regression branch WIoU_v3 loss function to replace the original CIoUloss function, and introduces the region-text contrast loss function for text vocabulary embedding representation and image features to help the model realize negative feedback network training for text feature information and image feature information.

[0112] Specifically, the WIoU_v3 loss function is a regression branch loss function based on a dynamic non-monotonic focusing mechanism (FM). By introducing a weighting mechanism, it can accurately balance the parameters of the bounding box, making the model more accurate in locating defects at different scales. The original CIOU loss function mainly considers the overlap of the bounding box and the distance between the center points, but ignores the shape difference and aspect ratio between the prediction box and the target box. Compared with the static focusing mechanism of the original CIoU loss function, which cannot effectively distinguish the quality of the annotation box, and has insufficient learning and generalization capabilities for low-quality data information, the WIoU_v3 loss function is a loss function formulated through a gradient gain allocation strategy, which can automatically evaluate the quality of the anchor box. While ensuring the competitiveness of the high-quality boundary anchor box, it effectively reduces the penalty for low-quality data information, reduces the impact of unstable gradients, and enhances the model's ability to parse various quality data information, thereby ensuring the model's robustness in processing data under different interference conditions (such as different weather conditions, illumination angles, and image feature blur caused by bridge structure reflection). At the same time, during the training process, the convergence speed is accelerated while maintaining the model's learning ability. In order to realize the accurate recognition of different quality data set information by the algorithm model, the present invention uses the WIoU_v3 loss function to replace the CIoU loss function in the original regression branch in the Head detection head. The calculation method is shown in Formula 9, Formula 10 and Formula 11:

[0113] Loss WIoU_v3 =γ·R WIoU ·Loss IoU (9)

[0114]

[0115]

[0116] Among them, Loss WIoU_v3 is the first loss function; Loss IoU represents the general anchor box loss function, W g With H g are the width and height of the smallest outer bounding box, w and h represent the width and height of the predicted box in the target detection result, respectively. gt and h gt They represent the width and height of the real box in the artificial label data, α and δ represent two adaptive hyperparameter constants, and β represents the aggregate outlier degree.

[0117] The WIoU_v3 loss function optimizes the bounding box regression process by weighting multiple factors such as the overlap between the prediction box and the target box, the center point distance, and the aspect ratio. Specifically, the WIoU_v3 loss function weightedly sums the bounding box overlap (IOU), center point distance (Dist), and aspect ratio (Aspect Ratio) errors to reduce the regression error caused by the imbalance of the aspect ratio, thereby improving the overall detection accuracy of the model. This improvement enables the Yolo-PVW algorithm to more accurately regress and locate the bounding box when facing bridge defects of different shapes and sizes, effectively improving the accuracy and robustness of the model's bridge defect detection results.

[0118] The Region-Text Contrastive Loss function is a contrastive learning method for aligning object regions in images with text descriptions. The purpose is to minimize the similarity between mismatched regions and text embeddings by maximizing the similarity between matching regions and text embeddings. By utilizing the mechanism of contrastive learning, relevant (matching) region-text pairs are brought closer (maximizing similarity) and irrelevant (mismatching) region-text pairs are separated (minimizing similarity). In this way, the model can learn which image regions correspond to which text descriptions. Based on the similarity calculation method mentioned above, image-text pairs with high similarity are obtained as positive sample pairs, and image-text pairs with low similarity are obtained as negative sample pairs. Specifically, image features with high similarity to text (noun or adjective description information) are considered to be positive sample pairs, and those with low similarity are considered to be negative sample pairs. The entire contrast loss is implemented through cross-entropy loss, which is used to maximize the similarity of positive samples and minimize the similarity of negative samples. The similarity calculation formula is Formula 6 mentioned above, and its calculation method is shown in Formula 12.

[0119]

[0120] Among them, Loss con is the second loss function; N represents the total number of samples of image-text pairs; exp() is the exponential function;

[0121] Similarity(e i ,w i ) is the region image representation e i And the corresponding text embedding representation w i The similarity between

[0122] Similarity(e i ,w j ) is the region image representation e i With the overall text embedding representation w j The similarity between them, where w j Refers to each text representation in the entire text collection.

[0123] So for a given image sample I and text T, the model outputs K object predictions And comments The predictions are matched with the annotations through the aligned image-text pair assignment strategy, and a classification label with a text index is assigned to each prediction result. With the help of the text encoder based on CLIP pre-training, the text encoding features of nouns and adjectives are given priority during the model training process, and the image-text pairs are directly used for training. The regional text contrast loss calculation head and the fusion WIoU decoupled detection head jointly participate in the feedforward neural network training. Therefore, the overall loss function calculation method designed is shown in Formula 13. The loss function evaluates the image-text pair rather than just a single category information.

[0124] Loss(I)=Loss con +λ I ·Loss WIoU_v3 (13)

[0125] Among them, λ I Represents the indicator factor, which is set to 1 when the input image sample I comes from detection or positioning data, and is set to 0 when the input image sample I comes from irrelevant annotation text.

[0126] Step 3, obtaining the bridge image to be detected, and dividing the bridge image to be detected into multiple overlapping blocks based on the SAHI sliding window mechanism;

[0127] SAHI (Slicing Aided Hyper Inference) is an inference optimization technology designed for the problem of small target detection in large-scale images. It dynamically cuts large images into smaller image blocks (i.e. "slices") and performs target detection on these slices to improve the detection capability and inference accuracy of small targets. In the actual production environment, the model is built and trained as follows:

[0128] Firstly, based on the improved Yolo-PVW bridge defect detection algorithm (i.e., bridge defect detection model), FFmpeg is used to extract I frames from video images collected in key areas. The Label Studio multimodal data annotation tool is used to manually annotate the dataset to obtain an annotated dataset. The MultiModalDataset data wrapper is set to preprocess and package the data to obtain an image-text pair dataset. The neural network weights are trained by inputting the annotated dataset of bridge apparent defects to obtain a neural network weight file in the .pth format, i.e., the Yolo-PVW network model. The structure type of this model is the Pytorch framework structure.

[0129] Then, because the original .pth format network model cannot be combined with the SAHI reasoning mechanism and directly deployed on the actual production hardware equipment, this patent uses model sequence standardization technology to convert the Yolo-PVW network model with the Pytorch framework as the core into a .onnx format file with the ONNX Runtime framework as the core.

[0130] Next, the network model in onnx format is embedded into the reasoning mechanism of SAHI, and the original input image is divided into multiple overlapping slices. Based on the aspect ratio of the original image, the network model in onnx format is used to perform recognition and detection of small target sliding slice reasoning. At the same time, a complete full-size reasoning is performed on the original image to detect large targets.

[0131] Finally, the two types of prediction results are combined into the overall recognition result through NMS technology to remove duplicate and low-quality recognition detection results. The overall workflow is as follows: Figure 7 As shown, the deployment structure developed by the present invention inputs high-resolution bridge monitoring image data, and outputs .jpg image data with anchor frames, confidence and disease category information, and .txt text data of the relative position and size of the disease.

[0132] Step 4: Detect each image block in a local area based on the bridge disease detection model to obtain a regional detection result; based on each regional detection result, generate a global detection result by splicing and fusion.

[0133] Example 2

[0134] like Figure 8 As shown, the difference between this embodiment and embodiment 1 is that this embodiment provides a bridge defect detection system based on an improved multimodal visual language model, and the system has a one-to-one correspondence with the bridge defect detection method based on an improved multimodal visual language model in embodiment 1; the system includes:

[0135] An acquisition unit is used to acquire a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilevered section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge;

[0136] The model building and training unit is used to build an improved multimodal visual language target detection model; based on the original bridge image, the improved multimodal visual language target detection model is trained to obtain the trained improved multimodal visual language target detection model as a bridge disease detection model;

[0137] An image division unit, used for acquiring an image of a bridge to be detected, and dividing the image of the bridge to be detected into a plurality of overlapping blocks based on a SAHI sliding window mechanism;

[0138] The detection unit is used to detect each block in a local area based on the bridge disease detection model to obtain a regional detection result; based on each regional detection result, a global detection result is generated by splicing and fusion.

[0139] As a further implementation, the improved multimodal visual language object detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual language fusion module based on the existing Yolov10s object detection model;

[0140] The improved multimodal visual language target detection model includes a feature extraction backbone network Backbone, a feature fusion network Neck and a detection head Head connected in sequence;

[0141] The feature extraction backbone network Backbone introduces an optimized multi-layer link self-attention mechanism based on the existing feature extraction backbone network, enabling the feature extraction network to capture global context information at different scales and levels and extract multi-scale visual feature information;

[0142] The feature fusion network Neck replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of disease features is enhanced by combining with pre-defined text feature description information. The feature fusion network outputs the fused image feature information and text feature information;

[0143] The head is detected by introducing an overall loss function to assist the model in realizing negative feedback network training of text feature information and image feature information; the overall loss function includes the first loss function and the second loss function. The first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

[0144] Among them, the execution process of each unit can be executed according to the process steps of the bridge disease detection method based on the improved multimodal visual language model in Example 1, and will not be repeated one by one in this embodiment.

[0145] In addition, the detection device designed to support the above method and system of the present invention is mainly composed of components such as solar panels, batteries, Raspberry Pi 3b, LoRa gateway, high-definition camera and waterproof housing, and is designed for real-time monitoring of bridge diseases.

[0146] (1) Solar panels and batteries: To ensure continuous power supply to the equipment, the device uses high-efficiency solar panels as the main power source. The solar panels are fixedly installed on the top of the bridge streetlights to maximize the reception of sunlight. The electricity generated by the solar panels is stored in the batteries through the controller to ensure that the monitoring device can continue to work at night or in cloudy conditions. The batteries use Li batteries, which can provide short-term stable power supply to the device in insufficient light conditions to support the device to regularly obtain disease feature image data of key areas of the bridge.

[0147] (2) Raspberry Pi 3b: Raspberry Pi 3b is the core control unit of the device, responsible for data processing, model reasoning and information transmission. Raspberry Pi is connected to the high-definition camera through the USB interface, starts up at a fixed time and controls the camera. When the hardware device is turned on, it automatically collects high-definition images of bridge defects. First, install the Ubuntu 20.04 operating system on the Raspberry Pi 3b and equip it with the onnx reasoning framework. Then copy the reasoning structure into the system, access the image data through the driver interface provided by opencv for reasoning analysis, and obtain the .jpg and .txt result data information of the bridge defect. Finally, the Raspberry Pi sends the identified bridge surface defect feature information in the form of .txt text to the remote terminal through the subsequent LoRa gateway, and regularly clears the cached .jpg recognition result image data. After completing the monitoring task, the device enters sleep mode and runs in low power mode. By collecting and processing the defect data on site in real time and only sending the result information to the terminal, the data transmission and operation and maintenance costs can be effectively reduced, so as to achieve large-scale popularization and installation in the field of bridge monitoring.

[0148] (3) LoRa Gateway: In order to achieve remote data transmission, a micro LoRa gateway is integrated in the device. Relying on the solar panel and battery function, it receives the Raspberry Pi 3b command to adjust the operating mode. LoRa (Long Range) is a low-power wide area network (LPWAN) technology with the characteristics of long-distance transmission and low power consumption. It is suitable for widely distributed application scenarios such as bridge monitoring. The LoRa gateway transmits the monitoring data processed by the Raspberry Pi (.txt text monitoring data) and the device number information to the terminal server wirelessly on a regular basis. The use of the LoRa gateway can significantly reduce the cost and maintenance of wired communications, while increasing the flexibility of system layout.

[0149] (4) High-definition camera: The RER-USB48MP01 high-definition camera is an important component of the device, responsible for capturing the surface details of the bridge in real time. The camera has an ultra-high resolution of 8000*6000 pixels, a 90-degree distortion-free viewing angle, low cost, and excellent environmental adaptability. In addition, this patent also provides it with an anti-glare light-transmitting filter lens to ensure that clear images can be obtained under different ambient light conditions. The RER-USB48MP01 high-definition camera is connected to the Raspberry Pi via USB, and the image data captured by it is used for subsequent bridge disease detection.

[0150] (5) Waterproof shell: In order to ensure that all devices of this patent can operate normally in various harsh environments, the device uses a high-level waterproof shell designed with PVC material, and the shell seams and interfaces are sealed with silicone sealing rings. The shell has IP67 waterproof and dustproof capabilities, which can effectively protect the internal electronic components from external factors such as rain and dust. In addition, the shell design also takes heat dissipation requirements into consideration, and a ventilation hole is provided at the bottom. When the device is in working state, the small hole will automatically open for heat dissipation to maintain the stability of the internal temperature of the device.

[0151] (6) Device fixing and installation: The entire device is designed as a modular structure, which is convenient for quick installation and maintenance. The device can be fixed on key parts of the bridge, such as piers, bridge belly or joints, and the accuracy of the camera angle can be ensured by an adjustable bracket. The device is equipped with a buffer pad and an anti-vibration design is adopted during installation to ensure that the equipment can still maintain stable operation under the influence of external forces such as wind and traffic vibration.

[0152] The overall structural design of the device is as follows Fig. 9 As shown, first, the external solar panel is used to obtain light energy in real time and store it in the battery, which provides stable energy for all subsequent devices; then, the detection program is started regularly through the built-in timing program of the Raspberry Pi 3b, and instructions are sent to the high-definition camera for data acquisition. After the camera is finished shooting, it can enter the sleep state. Next, the Raspberry Pi analyzes the acquired image data and sends the result information and the power-on instruction to the LoRa gateway. Finally, when the LoRa gateway receives the power-on instruction, it sends the result information and the device number information to the remote terminal. After the sending is completed, it also enters the sleep state. The entire device enters the sleep state and waits for the next scheduled detection process to start. The overall design of the monitoring device of this patent has the characteristics of low cost, low power consumption, and high reliability. It can perform real-time monitoring in key areas of the bridge, conduct a comprehensive assessment of the health status of the bridge, continuously monitor existing diseases, and issue an alarm in time when new diseases are detected, thereby improving the intelligence level of daily bridge maintenance.

[0153] The present invention improves the backbone network part of visual recognition by introducing a self-attention feature extraction mechanism, improves the neck network part of multimodal features by adding a context feature prompt mechanism of natural language keywords, uses the loss function of the optimized regression branch and the regional text comparison loss function to train the cross-modal image text model, designs an information collection system with a sliding window mechanism as the core to standardize image data to a unified standard, and the system hardware devices include solar panels for highway street lights, battery energy storage, fixed surveillance camera monitoring, Raspberry Pi 3b computing and 4G gateway information image transmission. Finally, all hardware devices manage information through a vector database that strengthens information retrieval. The present invention combines the task requirements of actual bridge detection to establish a complete set of low-cost, automated health monitoring systems, achieve accurate detection of key areas of bridge diseases and areas that cannot be effectively detected, reduce labor costs and machine use risks, provide timely and effective bridge health information, and meet the safety and practical application requirements of bridge detection projects.

[0154] At the same time, the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned bridge defect detection method based on the improved multimodal visual language model is implemented.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bridge defect detection method based on an improved multimodal visual language model, characterized in that: The method includes: Acquire a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilever cast section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge; Constructing an improved multimodal visual language target detection model; training the improved multimodal visual language target detection model based on the original bridge image to obtain a trained improved multimodal visual language target detection model as a bridge disease detection model; Acquire a bridge image to be detected, and divide the bridge image to be detected into a plurality of overlapping blocks based on a SAHI sliding window mechanism; Based on the bridge defect detection model, each block is detected in a local area to obtain a regional detection result; based on each regional detection result, a global detection result is generated by splicing and fusion.

2. The bridge defect detection method based on the improved multimodal visual language model according to claim 1 is characterized in that: The improved multimodal visual language target detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual language fusion module on the basis of the existing Yolov10s target detection model.

3. The bridge defect detection method based on the improved multimodal visual language model according to claim 2 is characterized in that: The improved multimodal visual language target detection model includes a feature extraction backbone network, a feature fusion network and a detection head connected in sequence; The feature extraction backbone network introduces an optimized multi-layer link self-attention mechanism based on the existing feature extraction backbone network, so that the feature extraction network can capture global context information at different scales and levels and extract multi-scale visual feature information; The feature fusion network replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of the disease features is enhanced by combining with the pre-defined text feature description information; the feature fusion network outputs the fused image feature information and text feature information; The detection head, by introducing an overall loss function, assists the model in realizing negative feedback network training of the text feature information and the image feature information; the overall loss function includes a first loss function and a second loss function, the first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

4. The bridge defect detection method based on the improved multimodal visual language model according to claim 3 is characterized in that: The feature extraction backbone network includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor and a fifth feature extractor connected in sequence, wherein the fifth feature extractor is a feature extraction module that integrates an optimized multi-layer link self-attention mechanism into a convolutional neural network; The fifth feature extractor includes a first standard convolution layer, a depth-separable convolution layer, a first normalization layer, an optimized multi-layer link self-attention mechanism layer, a second normalization layer and an MLP feature map processing layer; The first standard convolution layer is connected to the depthwise separable convolution layer, the output of the first standard convolution layer is fused with the output of the depthwise separable convolution layer and then connected to the first normalization layer, the first normalization layer is connected to the optimized multi-layer link self-attention mechanism layer, the output of the first standard convolution layer is fused with the output of the depthwise separable convolution layer and then fused with the output of the optimized multi-layer link self-attention mechanism layer and connected to the second normalization layer, the second normalization layer is connected to the MLP feature map processing layer, and the MLP feature map processing layer is then connected to the subsequent standard convolution layer.

5. The bridge defect detection method based on the improved multimodal visual language model according to claim 4 is characterized in that: The execution process of the optimized multi-layer link self-attention mechanism layer is as follows: Taking the feature map as input information, divide the feature map into S*S different regions; Define the eigenvector X r , obtain the query value matrix Q, key value matrix K and value matrix V through linear mapping, and calculate the overall attention Attention(Q,K,V); By building a connectivity graph to locate the relevance of a given area, the attention relationship between different areas is obtained; Perform regional averaging on the query value matrix Q and value matrix V of each region, and derive and calculate the dot product corresponding to each region to obtain the adjacency matrix A r ; By filtering out the adjacency matrix A at the coarse-grained level r The area with the smallest correlation is used to retain the top n areas with the highest correlation, and the routing index matrix I is obtained. r ; The coarse-grained level of attention is calculated through the routing index matrix between different regions, and all relevant key-value matrices K and value matrices V in the attention region are combined to obtain the cluster K g With V g ; K for clustering g With V g Perform attention calculation and local context enhancement LCE(V) derivation, and output the calculation result O=Attention(Q,K g ,V g )+LCE(V).

6. The bridge defect detection method based on the improved multimodal visual language model according to claim 3 is characterized in that: In the feature fusion network, image annotation is redefined as region-text pairs. where t i is the bounding box B i The corresponding text information is a kind of object description text information about the detected target; in this way, the predicted bounding box {B k } and its corresponding object embedding {e k },{e k } represents the embedded representation of the target object description information output by the model.

7. The bridge defect detection method based on the improved multimodal visual language model according to claim 3 is characterized in that: The expression of the overall loss function is: Loss(I)=Loss con +λ I ·Loss WIoU_v3 Loss WIoU_v3 =γ·R WIoU ·Loss IoU Among them, λ I Represents the indicator factor, which is set to 1 when the input image sample I comes from detection or positioning data, and is set to 0 when the input image sample I comes from irrelevant annotation text; Loss WIoU_v3 is the first loss function; Loss IoU represents the general anchor box loss function; W g With H g are the width and height of the smallest outer bounding box respectively; w and h represent the width and height of the prediction box in the target detection result respectively; w gt and h gt Respectively represent the width and height of the real box in the artificial label data; α and δ represent two adaptive hyperparameter constants; β represents the aggregate outlier degree; Loss con is the second loss function; N represents the total number of samples of image-text pairs; exp() is the exponential function; Similarity(e i ,w i ) is the region image representation e i And the corresponding text embedding representation w i The similarity between Similarity(e i ,w j ) is the region image representation e i With the overall text embedding representation w j The similarity between them, where w j Refers to each text representation in the entire text collection.

8. The bridge defect detection system based on the improved multimodal visual language model is characterized by: The system includes: An acquisition unit is used to acquire a historical original image of the bridge, wherein the original image of the bridge includes an image of a cantilevered section, a cast-in-place section, a web, a diaphragm, a closure section, and a concrete box girder of the bridge; A model building and training unit is used to build an improved multimodal visual language target detection model; based on the original bridge image, the improved multimodal visual language target detection model is trained to obtain a trained improved multimodal visual language target detection model as a bridge disease detection model; An image division unit, used for acquiring an image of a bridge to be detected, and dividing the image of the bridge to be detected into a plurality of overlapping blocks based on a SAHI sliding window mechanism; The detection unit is used to detect each image block in a local area based on the bridge disease detection model to obtain a regional detection result; based on each regional detection result, a global detection result is generated by splicing and fusion.

9. The bridge defect detection system based on the improved multimodal visual language model according to claim 8 is characterized in that: The improved multimodal visual language target detection model is improved by introducing an optimized multi-layer link self-attention mechanism and an optimized VLPAN cross-modal visual language fusion module on the basis of the existing Yolov10s target detection model; The improved multimodal visual language target detection model includes a feature extraction backbone network, a feature fusion network and a detection head connected in sequence; The feature extraction backbone network introduces an optimized multi-layer link self-attention mechanism based on the existing feature extraction backbone network, so that the feature extraction network can capture global context information at different scales and levels and extract multi-scale visual feature information; The feature fusion network replaces the original feature fusion network by introducing the optimized VLPAN cross-modal visual language fusion module; the multi-scale visual feature information extracted by the feature extraction backbone network is input into the feature fusion network, the multi-scale features from different feature layers are aggregated, and the recognition ability of the disease features is enhanced by combining with the pre-defined text feature description information; the feature fusion network outputs the fused image feature information and text feature information; The detection head, by introducing an overall loss function, assists the model in realizing negative feedback network training of the text feature information and the image feature information; the overall loss function includes a first loss function and a second loss function, the first loss function refers to the improved regression branch WIoU_v3 loss function, and the second loss function refers to the region-text contrast loss function for text vocabulary embedding representation and image features.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the bridge defect detection method based on the improved multimodal visual language model as described in any one of claims 1 to 7 is implemented.

Images