Building apparent disease detection and evaluation method based on visual language model

Abstract

The invention discloses a building apparent disease detection and evaluation method based on a visual language model, and the method comprises the steps: receiving a high-resolution image and related text description of a building surface through an input module, and converting the high-resolution image and related text description into a format for subsequent processing, so as to guarantee that the model fully obtains visual and language information; the encoder module extracts the multi-scale features of the building image and the semantic features in the text description, thereby enhancing the feature expression capability. The double-branch multi-mode fusion device effectively integrates visual features and text features so as to improve the performance of defect detection. The prior experience learning module optimizes the model performance by storing historical experiences and generating dynamic soft tags. According to the multi-task training mechanism, the comprehensive performance and robustness of the model are improved by learning a plurality of related tasks at the same time, training is carried out in two stages, and effective feature extraction and further optimization are ensured. The efficiency and accuracy of the disease detection method are improved, and the method is suitable for industrial large-scale application and popularization.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of building disease detection, and particularly relates to a building surface disease detection and evaluation method based on a visual language model. BACKGROUND

[0002] During use, buildings often develop various surface diseases over time, such as cracks, peeling, water seepage, etc. These diseases not only affect the appearance of the building, but also can pose a serious threat to its structural safety and use function.

[0003] Traditional disease detection methods mainly rely on manual inspection, which has the problems of low efficiency, poor accuracy, and being easily affected by subjective factors, and is difficult to meet the needs of modern building maintenance management. Therefore, there is an urgent need for an efficient and accurate automated detection method to realize rapid identification and evaluation of building surface diseases. SUMMARY

[0004] The purpose of the present application is to provide a building surface disease detection and evaluation method based on a visual language model to solve the problem of low efficiency and poor accuracy of the disease detection method in the prior art.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] A building surface disease detection and evaluation method based on a visual language model, comprising the following steps:

[0007] Step 1, establishing a building surface disease detection and evaluation model, the building surface disease detection and evaluation model comprising an input module, an encoder module, a dual-branch multi-modal fusioner module and an output module;

[0008] Step 2, inputting a building surface disease image into the input module, the input module inputting image features and text features of the building surface disease image into the encoder module;

[0009] Step 3, the encoder module converting the image features into image multi-scale features, and inputting the attention-enhanced feature image into the dual-branch multi-modal fusioner module after obtaining the attention-enhanced feature image;

[0010] The encoder module converts the text features into text multi-scale features, processes the data, obtains the semantic feature representation of the text multi-scale features, and inputs the semantic feature representation into the dual-branch multi-modal fusioner module;

[0011] Step 4, the dual-branch multi-modal fusioner module processing the attention-enhanced feature image and the semantic feature representation of the text multi-scale features respectively, obtaining visual mixed features and text mixed features;

[0012] Then, the visual and textual hybrid features are fused to obtain multimodal features;

[0013] Step 5: Repeat steps 2-4 to obtain multimodal features from multiple images of building surface defects;

[0014] Step 6: Based on all the multimodal features obtained in Step 5, optimize the building appearance defect detection and evaluation model to obtain a preliminary optimized building appearance defect detection and evaluation model.

[0015] Step 7: Use a multi-task training mechanism to train the preliminarily optimized building appearance defect detection and evaluation model to obtain the trained building appearance defect detection and evaluation model.

[0016] Step 8: Input the building surface defect image to be processed into the trained building surface defect detection and assessment model, and assess the risk of the building surface defect image through the output module.

[0017] The present invention also has the following features:

[0018] Furthermore, the encoder module includes a visual encoder and a text encoder;

[0019] The visual encoder described herein adopts the Swing Transformer model.

[0020] The text encoder described uses a BERT-based Transformer model.

[0021] Furthermore, step 3 includes the following sub-steps:

[0022] Step 31: The visual encoder processes the image features I∈R of the building surface defects image. C×H×W The image undergoes transformation processing to obtain multi-scale features M1-M2. n ;

[0023] Where R represents the attribute of the building's surface defects image;

[0024] C represents the number of channels in the image of building surface defects;

[0025] H represents the height of the image of the building's surface defects;

[0026] W represents the width of the image of the building's surface defects;

[0027] Step 32, combine the multi-scale features M1-M of the image using the following formula. n To form a complete feature Q h ;

[0028]

[0029] wherein LN represents a layer normalization function;

[0030] h represents a layer number;

[0031] represents a weight matrix;

[0032] represents an image feature M n of a complete feature;

[0033] l represents an index variable;

[0034] Step 33, using the complete feature as a query vector, an attention-enhanced feature image is obtained using the following formula :

[0035]

[0036] wherein softmax represents a nonlinear activation function;

[0037] K h represents position index information of a current scale feature;

[0038] V h represents content information of a current scale feature;

[0039] represents a transpose of the position index information of the current scale feature;

[0040] Step 34, a text encoder receives a text feature and converts it into a text multi-scale feature, and obtains a semantic feature representation of the text multi-scale feature according to the text multi-scale feature.

[0041] Further, the dual-branch fusion module comprises a multi-modal interaction module and a fusion module.

[0042] The multi-modal interaction module comprises a visual branch and a text branch, and the visual branch and the text branch each comprise a multi-head self-attention layer and a multi-head cross-attention layer.

[0043] Further, step 4 comprises the following sub-steps:

[0044] Step 41, in the multi-modal interaction module:

[0045] the multi-head self-attention layer of the visual branch extracts a visual feature map related to the building surface disease according to the attention-enhanced feature image;

[0046] the multi-head cross-attention layer of the visual branch introduces the semantic feature representation of the text multi-scale feature;

[0047] Step 42, the multi-head cross-attention layer and the multi-head self-attention layer of the text branch respectively introduce the semantic feature representation of the text multi-scale feature;

[0048] Step 43, in the multi-head self-attention layer of the visual branch, the visual feature map obtained in step 41 is processed using a multi-head self-attention function, as follows:

[0049]

[0050] wherein, represents the visual feature map processed by the multi-head self-attention mechanism;

[0051] MHA represents the multi-head self-attention function;

[0052] represents the visual feature map, and i represents the number of iterations;

[0053] Step 44, in the multi-head cross-attention layer, the visual feature map processed by the multi-head self-attention mechanism is fused with the semantic feature representation using a multi-head cross-attention function, as follows:

[0054]

[0055] wherein, represents the visual feature map fused with the semantic feature representation by the multi-head cross-attention;

[0056] MCA represents the multi-head cross-attention function;

[0057] T XT represents the vector output by the multi-head cross-attention layer of the text encoder;

[0058] Step 45, the visual feature map fused with the semantic feature representation by the multi-head cross-attention is further enhanced using a multi-layer perceptron function to obtain a visual mixed feature, as follows:

[0059]

[0060] wherein, represents the visual mixed feature;

[0061] MLP represents the multi-layer perceptron function;

[0062] represents the vector output by the multi-head cross-attention layer of the text encoder;

[0063] Step 46, the same operation as steps 43-45 is used to process the semantic feature representation of the text multi-scale feature using the text branch to obtain a text mixed feature

[0064] Step 47, the fusion module fuses the visual mixed feature and the text mixed feature to obtain a multi-modal feature.

[0065] Further, step 47 includes the following sub-steps:

[0066] Step 47-1, the fusion module fuses the representation features of the visual mixed feature and the text mixed feature to obtain a joint multi-modal vector and generate a visual and text joint feature F and a query vector q ; IT and a query vector q IT ;

[0067] Step 47-2, respectively linearly transform the visual and text joint feature F IT and the query vector q IT to obtain a query vector Q IT , a key vector K IT and a value vector V IT using the following formula:

[0068]

[0069]

[0070] wherein, and are learnable matrices of size dxd used for linear projection of the feature vectors;

[0071] Step 47-3, calculate the multi-modal feature F M using the following formula:

[0072]

[0073] wherein, softmax represents a nonlinear activation function.

[0074] Further, in step 8, when the output module evaluates the risk of the building apparent disease image, the output module calculates the risk score R according to the disease length xL, the disease width xW, the disease depth xD and the disease area xA corresponding to the building apparent disease image using the following formula:

[0075] R = αxL + βxW + γxD + δxA

[0076] wherein, α represents the weight coefficient of the disease length;

[0077] β represents the weight coefficient of the disease width;

[0078] γ represents a weight coefficient of the disease depth;

[0079] δ represents a weight coefficient of the disease area.

[0080] Further, when the output module evaluates the risk of the building surface disease image in step 8, the building surface disease is divided into the following four levels, and each level corresponds to a risk score R:

[0081] I level, the crack width is not more than 0.2 mm and the length is not more than 1 m, the area of spalling is not more than 0.1 square meters, R <1.0;

[0082] II level, the crack width is between 0.2 mm and 0.5 mm and the length is between 1 m and 3 m, the area of spalling is between 0.1 square meters and 0.5 square meters, 1.0≤R<2.5;

[0083] III level, the crack width is between 0.5 mm and 1 mm and the length is greater than 3 m, the area of spalling is between 0.5 square meters and 1 square meter, 2.5≤R<4.5;

[0084] IV level, the crack width exceeds 1 mm and is a through crack, the area of spalling is greater than 1 square meter, R≥4.5.

[0085] Compared with the prior art, the present application has the following technical effects:

[0086] The building surface disease detection and evaluation method based on the visual language model of the present application improves the efficiency and accuracy of building disease detection through automation technology to solve the problems of low efficiency and poor accuracy of disease detection methods in the prior art. The input module receives high-resolution images of the building surface and related text descriptions and converts them into a format for subsequent processing to ensure that the model fully obtains visual and language information. Secondly, the encoder module extracts multi-scale features of the building image and semantic features in the text description, thereby enhancing the feature expression capability. The dual-branch multi-modal fusioner effectively integrates visual features and text features to improve the performance of defect detection, processes visual and text information respectively, and ensures that the model can accurately identify defect types and locations. At the same time, the prior experience learning module optimizes the model performance by storing historical experience and generating dynamic soft labels, which significantly improves the learning ability and accuracy, especially when dealing with false negative samples. The multi-task training mechanism improves the comprehensive performance and robustness of the model by simultaneously learning multiple related tasks, and trains in two stages to ensure effective feature extraction and further optimization. Through the above methods, the automation level of disease detection is improved, and the efficiency and accuracy of the disease detection method are improved, which is suitable for large-scale use and promotion in industry. DETAILED DESCRIPTION

[0087] The following specific embodiments of the present application are given, it should be noted that the present application is not limited to the following specific embodiments, any equivalent variations made on the basis of the technical solutions of the present application fall within the protection scope of the present application.

[0088] A building surface disease detection and evaluation method based on a visual language model, comprising the following steps: Step 1, establishing a building surface disease detection and evaluation model, the building surface disease detection and evaluation model comprising an input module, an encoder module, a double-branch multi-modal fusion module and an output module; Step 2, inputting a building surface disease image into the input module, the input module inputting image features and text features of the building surface disease image into the encoder module; Wherein, the input module receives high-resolution images of the building surface and related text descriptions to ensure that the model can fully obtain the required visual and language information.

[0089] Step 1, establishing a building surface disease detection and evaluation model, the building surface disease detection and evaluation model comprising an input module, an encoder module, a double-branch multi-modal fusion module and an output module;

[0090] Step 2, inputting a building surface disease image into the input module, the input module inputting image features and text features of the building surface disease image into the encoder module;

[0091] Wherein, the input module receives high-resolution images of the building surface and related text descriptions to ensure that the model can fully obtain the required visual and language information.

[0092] For example, the input image is I and the text description is T, the goal of the input module is to convert I and T into a format that can be processed later.

[0093] Step 3, the encoder module converts the image features into image multi-scale features, and inputs the attention-enhanced feature image into the double-branch multi-modal fusion module;

[0094] The encoder module converts the text features into text multi-scale features and processes the data, and inputs the semantic feature representation of the text multi-scale features into the double-branch multi-modal fusion module;

[0095] Step 4, the double-branch fusion module processes the attention-enhanced feature image and the semantic feature representation of the text multi-scale features respectively, and obtains visual mixed features and text mixed features;

[0096] The double-branch fusion module fuses the visual mixed features and the text mixed features to obtain multi-modal features;

[0097] The core of the embodiment is a double-branch multi-modal fusion engine (DBMFE) that effectively integrates visual and textual features to enhance defect detection performance. Through multi-head self-attention (MSA) layers and multi-head cross-attention (MCA) layers, visual and textual information are processed respectively to ensure that the model can accurately identify defect types and locations.

[0098] Step 5: Repeat steps 2-4 to obtain multi-modal features of multiple building apparent disease images.

[0099] Step 6: Based on all the multi-modal features obtained in step 5, optimize the building apparent disease detection and evaluation model to obtain a preliminary optimized building apparent disease detection and evaluation model.

[0100] The prior experience learning (LPE) module optimizes model performance by storing historical experience and generating dynamic soft labels, especially in handling false negative samples, significantly improving learning ability and accuracy. The multi-task training mechanism improves the comprehensive performance and robustness of the model by simultaneously learning multiple related tasks (such as image-text contrast learning, mask language modeling, etc.), and is trained in two stages to ensure effective feature extraction and further optimization.

[0101] Step 7: Use the multi-task training mechanism to train the preliminary optimized building apparent disease detection and evaluation model to obtain an optimized building apparent disease detection and evaluation model.

[0102] The multi-task training mechanism is prior art and is well known. The embodiment creatively introduces the multi-task training mechanism, which specifically includes image-text contrast learning (ITC), mask language modeling (MLM), mask image modeling (MIM), and image-text matching (ITM) subtasks set for building apparent disease detection tasks, improving the collaborative optimization ability among multiple tasks.

[0103] The training set used for training can use a public training set or be constructed by a person skilled in the art using known methods according to actual needs.

[0104] The data set used in this embodiment is self-collected and constructed, and the data sources include unmanned aerial vehicle aerial images, ground handheld device images and fixed monitoring point images in actual building environment, covering various building types and apparent disease scenes. The building structure parts and disease characteristic information are recorded synchronously during the collection process. The data annotation is completed by professional personnel with experience in structural disease assessment, including disease type label, structure risk level, disease area mask and language description information corresponding to the image. The complete data set has about 12,000 samples, which are divided into training set, verification set and test set according to the ratio of 70%, 15% and 15%, for multi-task supervised training, covering disease identification, risk classification and spatial positioning and other multi-dimensional targets.

[0105] On the basis of obtaining the preliminary optimized building apparent disease detection and evaluation model, the model is further trained by using a multi-task training mechanism. The multi-task mechanism includes but is not limited to disease type identification task, risk level evaluation task and disease area positioning task. In the model structure, a shared-dedicated branch architecture is adopted: a front-end encoder is used to uniformly extract multi-modal features of images and languages, and independent task decoding branches are set at the back end to respectively predict and optimize the three types of tasks. In the training process, the applicant self-built building disease multi-modal data set is adopted, which contains multi-dimensional information such as images, language descriptions, structural risk level labels and spatial positioning masks. The data set is divided into training set, verification set and test set after manual annotation, and is suitable for multi-task supervised learning. Through a joint loss function, each task sub-network is optimized, and finally an optimized building apparent disease detection and evaluation model is obtained.

[0106] Step 8, inputting the building apparent disease image to be processed into the building apparent disease detection and evaluation model, and evaluating the risk of the building apparent disease image through the output module.

[0107] Specifically, the encoder module includes a visual encoder and a text encoder;

[0108] The visual encoder adopts a Swin Transformer model, and the text encoder adopts a Transformer model based on BERT.

[0109] Further, step 3 includes the following sub-steps:

[0110] Step 31, inputting the image features I of the building apparent disease image into the visual encoder to obtain multi-layer image features M1-M C×H×W under multiple scales. n ;

[0111] Wherein, R represents the attribute of the building apparent disease image;

[0112] C represents the number of channels of the building apparent disease image.

[0113] H represents the height of the building apparent disease image;

[0114] W represents the width of the building apparent disease image;

[0115] wherein I represents the input image data, which is a three-dimensional tensor;

[0116] C represents the number of channels of the image, for example, C = 3 in an RGB image;

[0117] H represents the height of the image, and W represents the width of the image.

[0118] After the Swin Transformer model, multi-scale features can be obtained, in the embodiment, there are four layers of multi-scale features:

[0119] M1∈R (HW / 4×4)×c , M1 is the shallowest feature map, with higher resolution and shallower semantics;

[0120] M2∈R (HW / 4×4)×2c , M2 is the second layer feature map, with the same spatial size as M1, but richer semantic information and increased feature dimension;

[0121] M3∈R (HW / 16×16)×4c , M3 is the third layer feature map, with significantly reduced spatial size and higher semantic abstraction;

[0122] M4∈R (HW / 32×32)×8c , M4 is the deepest feature map, with the richest high-level semantic information.

[0123] This step fuses the obtained multi-scale features {M1, M2, M3, M4} to deeply capture the complex content in the building apparent disease image, M1, M2, M3, M4 are multi-scale feature maps output by the Swin Transformer model at different stages, with gradually reduced spatial size and gradually enhanced semantic information.

[0124] Step 32, combining each layer of image features M1-M n to form a complete feature, using the following formula to form a complete feature Q h ;

[0125]

[0126] wherein Ln represents a layer normalization function;

[0127] represents a weight matrix;

[0128] wherein, Specifically, it represents a linear mapping weight matrix used to generate a query vector (Query) in the attention mechanism, which is used to extract task-related attention regions from the fused image features.

[0129] In this embodiment, it is assumed that the input image feature dimension din=256, and the query vector dimension dq=64, which means that each fused feature point is mapped from 256 dimensions to a 64-dimensional query space. The matrix is initialized to a uniform distribution during the training process and is learned through back propagation to adapt to the task guidance characteristics of different attention heads.

[0130] represents the complete feature;

[0131] Step 33, using the following formula to obtain the attention-enhanced feature image with the complete feature as the query vector :

[0132]

[0133] wherein softmax represents a nonlinear activation function;

[0134] K h represents the position index information of the current scale feature;

[0135] V h represents the content information of the current scale feature;

[0136] represents the transpose of the position index information of the current scale feature;

[0137] Step 34, the text encoder receives the multi-scale feature of the text and obtains the semantic feature representation according to the multi-scale feature of the text.

[0138] The working process of the text encoder is to process the input text description through the BERT-based Transformer model. The model initializes the input text through word embedding (Embedding) and position encoding (Positional Embedding) to ensure that the model can understand the position relationship of words in the sentence. Next, the text feature will pass through multiple Transformer encoder layers, each layer containing multi-head self-attention mechanism and feedforward neural network to capture the rich semantic association between words, and through layer normalization to ensure the stability of the features. Finally, the text encoder outputs the semantic feature representation of the text, which is usually represented by a vector corresponding to the classification label, or obtained by aggregating the feature vectors of multiple Tokens, which is used for subsequent multi-modal fusion with visual features to achieve accurate identification and positioning of building surface diseases.

[0139] Specifically, the double-branch fusion module includes a multi-modal interaction module and a fusion module.

[0140] The multi-modal interaction module includes a visual branch and a text branch, and the visual branch and the text branch each include a multi-head self-attention layer and a multi-head cross-attention layer.

[0141] The multi-scale image semantic feature is a structured visual representation generated by the visual branch, which is equivalent to the "attention-enhanced complete feature" and is evolved from Q h image features for step 4 input.

[0142] The text semantic feature F TxT is the output after the original text is processed by the BERT Transformer model, which is the text encoding output of step 3 and is input T XT ; It and the "semantic feature " on the image side are parallel structures and are used as original inputs into the double-branch multi-modal interaction module.

[0143] Note: The "attention-enhanced feature image" output by step 3 is the "visual feature map"

[0144] Further, step 4 includes the following sub-steps:

[0145] Step 41, the multi-head self-attention layer of the visual branch extracts the visual feature map related to the building surface disease according to the feature image;

[0146] The multi-head cross-attention layer of the visual branch introduces the semantic feature representation to improve the recognition effect of the building surface disease detection and evaluation model for building surface diseases;

[0147] Step 42, the multi-head cross-attention layer and the multi-head self-attention layer of the text branch respectively introduce the semantic feature representation; the MSA layer analyzes the semantic relationship in the text and extracts key information to ensure that the model understands the defect type and location described in the text;

[0148] The MCA layer introduces the visual feature into the semantic feature representation to help the model better understand the text description according to the image content.

[0149] Step 43, the visual feature map is processed using a multi-head self-attention mechanism as follows:

[0150]

[0151] Wherein, represents the visual feature map;

[0152] Step 44: The semantic feature representation is fused using the multi-head cross-attention method on the visual feature map after processing by the multi-head cross-attention function, as shown in the following formula:

[0153]

[0154] in, This represents the visual feature map after semantic feature fusion through multi-head cross-attention;

[0155] Step 45: Use a multilayer perceptron function to further enhance the visual feature map after multi-head cross-attention fusion of semantic features to obtain visual hybrid features, as shown in the following formula:

[0156]

[0157] in, It represents visual blending features;

[0158] In the above formula and These are two intermediate feature maps in the processing, with a size of ((hHW / (32×32)))×d.

[0159] MSA, MCA, and MLP are functions of the MSA, MCA, and MLP layers, respectively. Since the visual branch and text module are identical in the multimodal fusion processor, visual fusion features are obtained after processing.

[0160] R ((HW / (32×32)))×d Text blending features

[0161] Step 46: Using the methods from steps 43-45, process the semantic feature representation and obtain the text hybrid features.

[0162] Step 47: The fusion module fuses the visual and textual mixed features to obtain multimodal features.

[0163] Furthermore, the branch fusion module in the dual-branch multimodal fusion unit aims to further fuse the features extracted and processed from the visual branch and the text branch to generate a unified and comprehensive multimodal feature representation.

[0164] First, visual blending features are considered. Text blending features Characteristic features and Cascaded into a joint multimodal vector P IT ∈R 2×d. Then, the joint feature (visual-text joint feature F IT ) and the query vector are respectively linearly transformed to obtain the query vector Q , the key vector K

[0165] and the value vector V for attention calculation.

[0165] Step 47 includes the following sub-steps:

[0166] Step 47-1, the fusion module concatenates the representation features of the visual mixed feature F and the text mixed feature F to obtain the joint multi-modal vector F .

[0167] Step 47-2, the visual and text joint features F iT and the query vector q iT are respectively linearly transformed using the following formula to obtain the query vector Q IT , the key vector K IT and the value vector V IT for attention calculation:

[0168]

[0169] wherein Wq , Wk and Wv are learnable matrices of size dxd for linear projection of feature vectors;

[0170] wherein the weight matrices belong to the known technical category, are standard parameter items for generating vectors in the attention mechanism, and are obtained through training and learning.

[0171] The dimension depends on the input / output feature dimension d, and the specific weight parameters are directly obtained by the model.

[0172] Step 47-3, the multi-modal feature F M is calculated using the following formula:

[0173]

[0174] wherein softmax represents a nonlinear activation function.

[0175] The multi-modal feature F M output by the fusion module finally contains highly fused semantic information of the visual and text modalities, and provides strong and unified feature support for subsequent model tasks such as disease classification and positioning.

[0176] The F Mis the core input of the step 5 priori experience learning module (LPE). The LPE module utilizes F M Matched with historical experience, generate optimized labels or representations to help the model train better to distinguish real defects and misjudgment-prone samples.

[0177] The priori experience learning (LPE) module and the multi-task training mechanism are both known methods in the prior art, which are further described as follows:

[0178] The LPE module is based on the multi-modal fusion feature and the pre-stored historical experience for feature matching and similarity analysis, and constructs dynamic labels or optimized feature representations.

[0179] The priori experience learning (LPE) module is an important part of the building surface disease detection method based on the visual language model, aiming to optimize the model performance by using historical experience and dynamic labels, especially in handling false negative samples. The LPE module effectively improves the learning ability and accuracy of the model through the following steps. The priori experience learning (LPE) module stores and utilizes historical experience to generate dynamic soft labels to optimize the model's handling of negative samples, significantly improving the performance of the building surface disease detection model. Through accurate matching judgment, experience storage and dynamic label generation, the LPE module reduces the impact of false negatives while enhancing the adaptability and accuracy of the model.

[0180] The multi-task training mechanism is a key component of the building surface disease detection method based on the visual language model, aiming to improve the comprehensive performance and robustness of the model by simultaneously learning multiple related tasks. The main tasks of the multi-task training mechanism are: image-text contrastive learning (ITC), mask language modeling (MLM), mask image modeling (MIM), and image-text matching (ITM). The multi-task training mechanism is divided into two stages. In the first stage, the model is trained through the above tasks, the main goal is to establish effective feature extraction capability, and the original hard label is used for training to ensure that the model understands the basic image and text relationship. This stage of task setting lays a foundation for the model, providing solid support for subsequent optimization. After the basic training, the priori experience learning (LPE) module and dynamic soft labels are introduced for optimization. In the second stage, the model continues to perform ITC, MLM, MIM and ITM tasks, but uses updated dynamic soft labels instead of original hard labels.

[0181] The output module is an important part of the building surface disease detection method based on the visual language model, responsible for converting the model's detection results into understandable information.

[0182] As an optional solution, the output module first provides detailed information of the building surface defects detected by the model, including: defect location, defect type, defect severity assessment. The output module will generate a visual result so that the user can intuitively understand the basis of the model's judgment. The summary of the detection results provides an overall view, which is convenient for the user to quickly understand the building status according to the detected defect type and severity.

[0183] The risk score calculation means that the model extracts the quantitative parameters of the disease according to the visual features of the disease, including length, width, depth, area and its location characteristics, and analyzes the association of the disease with the surrounding environment or other diseases. Length refers to the extension length of the crack or disease, in millimeters (mm) or meters (m); width refers to the opening width of the crack or the width of the peeling area; depth is the depth of disease erosion, which is an important indicator for structural safety; area represents the area of the affected area, in square meters (㎡); position is the specific position of the disease on the building outer wall, described using a coordinate system or floor, area.

[0184] Specifically, when the output module evaluates the risk of the building surface disease image in step 8, the output module calculates the risk score R according to the disease length xL, disease width xW, disease depth xD and disease area xA corresponding to the building surface disease image using the following formula:

[0185] R = a xL + b xW + g xD + d xA

[0186] Wherein, a represents the weight coefficient of the disease length;

[0187] b represents the weight coefficient of the disease width;

[0188] g represents the weight coefficient of the disease depth;

[0189] d represents the weight coefficient of the disease area.

[0190] Wherein, different weight distributions can be set according to the disease type or the importance of the building structure, and the following gives a specific embodiment:

[0191] a e [0.1, 0.3];

[0192] b e [0.1, 0.25];

[0193] g e [0.3, 0.5];

[0194] d e [0.2, 0.4];

[0195] In a more specific embodiment, a = 0.2, b = 0.2, g = 0.4, d = 0.2.

[0196] Further, when the output module evaluates the building apparent disease image risk in step 8, the building apparent disease is divided into the following four levels:

[0197] I (light) disease has no obvious impact on structure and use function, such as crack width not more than 0.2mm and length not more than 1m, spalling area not more than 0.1㎡, and slight dampness without obvious water seepage.

[0198] II (moderate) disease may affect the use function and needs to be handled in time, such as crack width between 0.2mm and 0.5mm and length between 1m and 3m, spalling area between 0.1㎡ and 0.5㎡, or seepage with obvious water seepage and may cause indoor dampness.

[0199] III (serious) disease seriously affects the safety of structure and needs to be taken immediately, such as crack width between 0.5mm and 1mm and length more than 3m, spalling area between 0.5㎡ and 1㎡ and falling risk, or seepage continues to leak and may damage internal facilities.

[0200] IV (critical) disease is a great threat to the safety of structure and personnel and must be handled immediately, such as crack width more than 1mm and through crack, spalling area more than 1㎡ and high-altitude falling risk, or seepage leaks a lot and may cause electrical short circuit and structure corrosion.

[0201] As an optional solution, disease data visualization quantitatively analyzes the number, area, and location of diseases in each level through automatic rating of diseases, providing scientific decision support for building maintenance. Visualize the disease data to generate floor plans, elevations, or three-dimensional models, and intuitively display the location and severity of the disease. Use color coding to identify disease levels and generate a heat map of disease distribution to help managers quickly understand the health of the building.

[0202] In addition, generate detailed reports including disease type, characteristic parameters (such as length, width, depth, area, location, etc.), rating results, and repair recommendations to support managers in developing repair and management plans. Disease distribution maps mark the location and rating of each disease on the building's floor or elevation plan, using different colors to represent different levels of disease. Statistical charts use bar charts, pie charts, and other methods to statistically analyze the number and area proportion of various diseases. Combined with three-dimensional modeling technology, provide interactive disease distribution display to help managers fully understand the spatial distribution of diseases.

[0203] As an optional solution, the output module can also include a data storage and management module, which stores all disease data in a database for easy querying, updating, and historical trend analysis. Regular backups of data are made to ensure data security and integrity. Access security for different users is ensured through permission management, supporting remote monitoring and management functions.

Claims

1. A method for detecting and evaluating architectural surface defects based on a visual language model, characterized in that, Includes the following steps: Step 1: Establish a building appearance defect detection and evaluation model, which includes an input module, an encoder module, a dual-branch multimodal fusion module, and an output module. Step 2: Input an image of building surface defects into the input module. The input module inputs the image features and text features of the building surface defects image into the encoder module. Step 3: The encoder module transforms the image features into multi-scale image features, obtains the attention-enhanced feature image, and then inputs it into the dual-branch multimodal fusion module. The encoder module transforms text features into text multi-scale features and then processes the data to obtain the semantic feature representation of the text multi-scale features before inputting it into the dual-branch multimodal fusion module. Step 4: The dual-branch multimodal fusion module processes the semantic feature representations of the attention-enhanced feature images and text multi-scale features separately to obtain visual fusion features and text fusion features. Then, the visual and textual hybrid features are fused to obtain multimodal features; Step 5: Repeat steps 2-4 to obtain multimodal features from multiple images of building surface defects; Step 6: Based on all the multimodal features obtained in Step 5, optimize the building appearance defect detection and evaluation model to obtain a preliminary optimized building appearance defect detection and evaluation model. Step 7: Use a multi-task training mechanism to train the preliminarily optimized building appearance defect detection and evaluation model to obtain the trained building appearance defect detection and evaluation model. Step 8: Input the building surface defect image to be processed into the trained building surface defect detection and assessment model, and assess the risk of the building surface defect image through the output module.

2. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 1, characterized in that, The encoder module includes a visual encoder and a text encoder; The visual encoder described herein adopts the Swing Transformer model. The text encoder described uses a BERT-based Transformer model.

3. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 2, characterized in that, Step 3 includes the following sub-steps: Step 31: The visual encoder processes the image features I∈R of the building surface defects image. C×H×W The image undergoes transformation processing to obtain multi-scale features M1-M2. n ; Where R represents the attribute of the building's surface defects image; C represents the number of channels in the building's surface defects image; H represents the height of the image of the building's surface defects; W represents the width of the image of the building's surface defects; Step 32, combine the multi-scale features M1-M of the image using the following formula. n To form a complete feature Q h ; Where LN represents the layer normalization function; h represents the layer number; Represents the weight matrix; Representing image features M n Complete features; l represents the index variable; Step 33: Using the complete features as the query vector, obtain the attention-enhanced feature image using the following formula. Where softmax represents a non-linear activation function; K h This indicates the location index information of the current scale feature; V h This represents the content information of the current scale features; This represents the transpose of the location index information of the current scale feature; Step 34: The text encoder receives text features and converts them into text multi-scale features, and obtains the semantic feature representation of the text multi-scale features based on the text multi-scale features.

4. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 1, characterized in that, The dual-branch fusion module includes a multimodal interaction module and a fusion module; The multimodal interaction module includes a visual branch and a text branch, both of which include a multi-head self-attention layer and a multi-head cross-attention layer.

5. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 4, characterized in that, Step 4 includes the following sub-steps: Step 41, in the multimodal interaction module: The multi-head self-attention layer of the visual branch extracts visual feature maps related to building appearance defects based on the feature images after attention enhancement; The multi-headed cross-attention layer of the visual branch introduces semantic feature representation of text with multi-scale features; Step 42: The multi-head cross-attention layer and multi-head self-attention layer of the text branches respectively introduce semantic feature representations of multi-scale text features; Step 43: In the visual branch of the multi-head self-attention layer, the visual feature map obtained in step 41 is processed using the multi-head self-attention function, as shown in the following equation: in, This represents the visual feature map after processing by a multi-head self-attention mechanism. MHA stands for Multi-Head Self-Attention Function; This represents the visual feature map, and i represents the iteration number. Step 44, in the multi-head cross-attention layer, the visual feature map processed by the multi-head self-attention mechanism is fused with the semantic feature representation using the multi-head cross-attention function, as shown in the following formula: in, This represents the visual feature map after semantic feature fusion through multi-head cross-attention; MCA stands for Multi-head Cross Attention Function; T XT This represents the vector output by the multi-head cross-attention layer of the text encoder. Step 45: Use a multilayer perceptron function to further enhance the visual feature map after multi-head cross-attention fusion of semantic features to obtain visual hybrid features, as shown in the following formula: in, This represents visual blending features; MLP stands for Multilayer Perceptron Function; This represents the vector output by the multi-head cross-attention layer of the text encoder. Step 46: Using the same operations as steps 43-45, process the semantic feature representation of the multi-scale features of the text using the text branch to obtain the text hybrid features. Step 47: The fusion module fuses the visual and textual mixed features to obtain multimodal features.

6. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 5, characterized in that, Step 47 includes the following sub-steps: Step 47-1, the fusion module combines visual blending features Text blending features Characteristic features and Cascaded into joint multimodal vectors And generate joint visual and textual features F IT and query vector q IT ; Step 47-2, use the following formula to calculate the joint visual and textual features F respectively. IT and query vector q IT Perform linear transformations to obtain the query vector Q used for attention calculation. IT Key vector K IT Sum vector V IT : in, and These are the d×d learnable matrices used for linear projection of the feature vectors; Step 47-3, use the following formula to calculate the multimodal feature F M : Here, softmax represents a non-linear activation function.

7. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 6, characterized in that, In step 8, when the output module assesses the risk of building surface defects images, it calculates the risk score R using the following formula based on the defect length xL, defect width xW, defect depth xD, and defect area xA corresponding to the building surface defects images: R = αxL + βxW + γxD + δxA Where α represents the weighting coefficient for disease length; β represents the weighting coefficient for the width of the disease; γ represents the weighting coefficient for disease depth; δ represents the weighting coefficient for the area affected by the disease.

8. The method for detecting and evaluating architectural surface defects based on a visual language model as described in claim 7, characterized in that, In step 8, when the output module evaluates the risk of building appearance defects images, it divides building appearance defects into the following four levels, each corresponding to a risk score R: Class I: Crack width not exceeding 0.2 mm and length not exceeding 1 m, spalling area not exceeding 0.1 m², R < 1.0; Class II, with crack width between 0.2mm and 0.5mm and length between 1m and 3m, and spalling area between 0.1㎡ and 0.5㎡, 1.0≤R<2.5; Grade III: crack width between 0.5mm and 1mm and length greater than 3m, peeling area between 0.5㎡ and 1㎡, 2.5≤R<4.5; Grade IV: Cracks wider than 1 mm and are penetrating cracks, with a spalling area greater than 1 m², and R ≥ 4.5.

Images