Apparent damage diagnosis method for civil engineering structures based on multimodal large model

Abstract

The present invention proposes a method for diagnosing apparent damage of civil engineering structures based on a multimodal large model. The method includes unified representation and alignment fusion of multimodal data of structural apparent damage, a structural damage identification and reasoning method based on a multimodal thinking chain, multimodal data set construction and large model fine-tuning, and civil engineering structure damage identification and feature description. The present invention uses a multimodal large model to complete the vector encoding of the structural damage image and align it with the text encoding features. By calling the computer vision model, a damage semantic segmentation map is generated as a visual thinking chain, and a language thinking chain is formed by constructing a fine-tuning data set. The pre-trained large language model completes the reasoning of the multimodal information encoding and the generation of structural damage diagnosis. The method uses a multimodal large model as a base and integrates the multimodal thinking chain reasoning method, so that the model can perform human-computer interaction through natural language, efficiently diagnose various types of structural damage, and improve the generalization ability and accuracy of the structural damage diagnosis model.

Description

Technical Field

[0001] The present invention belongs to the technical field of structural damage identification and artificial intelligence large models, and in particular relates to a method for diagnosing apparent damage of civil engineering structures based on a multimodal large model. Background Art

[0002] Over the course of long-term service, structures such as buildings, bridges, and roads inevitably suffer damage from the effects of the external environment. This damage primarily involves cracks, holes, spalling, and exposed rebar in concrete; cracks, shedding of protective coatings, and rust in steel components; and cracks and potholes in asphalt pavement. These damages not only affect the aesthetics of the structures but can also seriously impact their safety and stability. Traditional civil engineering maintenance relies primarily on visual inspections by inspectors, a time-consuming and labor-intensive method that is also susceptible to subjective influence.

[0003] In recent years, computer vision algorithms have demonstrated significant advantages in automated damage detection, significantly improving the efficiency and accuracy of concrete damage identification. These methods, which primarily include image classification, object detection, semantic segmentation, and instance segmentation, can identify damage categories and delineate damaged areas from both qualitative and quantitative perspectives. However, these models struggle to describe characteristic information such as the size and location of damage in detail using natural language, nor can they describe the severity of the damage. Furthermore, in the field of civil engineering, there are a wide variety of structures, such as buildings, bridges, and roads. These structures have diverse background information in real-world environments and possess a wide variety of defects. These characteristics pose significant challenges to the robustness and generalization of existing structural damage identification models, making it difficult for existing small computer vision models to cover a wide range of damage types.

[0004] With Google's introduction of the highly parallelizable Transformer model framework, the parameter size of deep learning models has continued to expand. Pre-trained large language models have emerged with capabilities that smaller models lack, such as following natural language instructions and performing step-by-step reasoning. To extend the powerful language capabilities of large language models to the visual modality, many researchers have attempted to map multimodal features, such as images, into a feature space consistent with language, enabling large models to understand multimodal information and perform learning and reasoning. For example, they use projection matrices to connect image encoders and large language models, training them to eliminate model artifacts caused by semantic misalignment between text and images. These large multimodal models, pre-trained on a large amount of common image-text data, demonstrate strong understanding of both language and vision.

[0005] However, since the damage that occurs during the operation and maintenance of civil engineering structures is far less than in normal circumstances, and the backgrounds of various types of damage are complex and the image collection process is cumbersome, image data containing concrete damage will rarely appear in general pre-training datasets. In addition, in addition to damage categories such as cracks and holes that are widely present in the industrial field, there are also damage types unique to the civil engineering field such as concrete spalling and steel bar exposure and corrosion. The diversity and imbalance of civil engineering structural damage result in the general multimodal large model learning less damage identification field knowledge, making it difficult to directly identify and describe structural damage accurately. Therefore, based on the open source pre-training large model, the present invention eliminates complex background information by integrating multimodal thinking chain reasoning, constructs a text-image multimodal dataset and combines it with an efficient fine-tuning strategy to achieve accurate identification and feature description of civil engineering structural damage. Summary of the Invention

[0006] The present invention aims to address the problems of the prior art and meet practical needs. It proposes a method for diagnosing apparent damage in civil engineering structures based on a multimodal large model. The method is suitable for diagnosing and characterizing damage based on damage images of various civil engineering structures.

[0007] The present invention is achieved through the following technical solutions. The present invention proposes a method for diagnosing apparent damage of civil engineering structures based on a multimodal large model. The method comprises the following steps:

[0008] Step 1: Organize damage images of various structures and supplement them with text descriptions of damage features. Use predefined prompt words and answer labels to standardize training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions.

[0009] Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis. Use a semantic segmentation model to remove background environmental information and complete the visual thought chain reasoning process. Use an image encoder to map the original structural damage image and the semantic segmentation map into a high-dimensional encoding in the same vector space. Use a text-image feature encoding alignment module to connect the semantics. Finally, use a large language model combined with text prompts to complete language thought chain reasoning and diagnosis generation.

[0010] Step 3: Use cross-entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label to optimize the quality of the damage description text of the multimodal large model; use an efficient fine-tuning optimization method based on low-rank adapters to add parallel learnable low-rank matrices to the original model layer to reduce the number of parameters that need to be trained; use multimodal datasets to iteratively train until the multimodal large model converges;

[0011] Step 4: Use the multimodal large model that has completed fine-tuning training to diagnose structural damage; input the structural apparent damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

[0012] Furthermore, the step 1 is specifically as follows:

[0013] Step 1.1: Collect damage images of civil engineering structures and perform data cleaning, rotation, and folding data enhancement processing; manually supplement text descriptions of defect category, shape, and severity damage feature information;

[0014] Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as json files. Each data in the dataset is in the format of {"image", "prompt", "label"}, which are the image path, prompt word, and label respectively.

[0015] Furthermore, in step 2, the structural damage image is normalized. For image I, the processing method is as follows:

[0016]

[0017] The normalized image is input to the downsampling encoder, which includes two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0018] Y = ReLU(X*W+b)

[0019] ReLU(x)=max(0,x)

[0020] Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code;

[0021] The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0022]

[0023] in, is the transposed convolution calculation, W is the transposed convolution kernel, X is the input feature code, b is the bias, and Y is the output semantic segmentation map.

[0024] Furthermore, in step 2, the damaged original image and the semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence t∈R is obtained. N×(P2C) ; The image block is mapped into a vector encoding through the linear projection layer:

[0025] z=[t1E;t2E;…;t N E]+E pos

[0026] Among them, E is the linear projection transformation, E pos Encode the position of the graphic block;

[0027] Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature code X = (x1, x2, ..., x n );

[0028] X is input into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism guides the model to retrieve information related to text semantics from image features; the dot-product attention mechanism calculates the correlation between the query vector Q and the image feature encoding X, and the weighted summation based on the attention weights is used to obtain the final image encoding that is consistent with the text semantics:

[0029]

[0030] Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

[0031] Furthermore, in step 2, combined with the input prompt word, the pre-trained large language model is used to perform multimodal thinking chain fusion reasoning on the image and text encoding. The reasoning process is as follows:

[0032] T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )}

[0033] p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q)

[0034]

[0035]

[0036] Among them, T CoTis the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, and output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

[0037] Furthermore, in step 3, when using the multimodal large model to generate text descriptions, a token encoding is generated each time, the target word in the training data is used as the correct label, and the cross-entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label; the loss function calculation method for the sample is as follows:

[0038]

[0039] Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

[0040] Furthermore, in step 3, a parallel learnable low-rank adapter is added to the multimodal large model. The calculation formula for each layer is as follows:

[0041] h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o

[0042] Where x is the input, h is the output, W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrices A and B. i is the input dimension, o is the output dimension, and r is a hyperparameter representing the rank of the parameter matrix. Before training, A is initialized to 0 and B is initialized with a random Gaussian.

[0043] Combined with the loss function, the optimization goal of the multimodal large model is:

[0044] ΔW=argmin ΔW Loss(W0+ΔW).

[0045] Furthermore, in step four, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damage area and the background; then, the original damage image and the semantic segmentation map are encoded by the image encoder, and the image encoding is aligned with the question text encoding input by the user using the alignment module; finally, the encoding is fused and inferred by the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

[0046] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model are implemented.

[0047] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model.

[0048] Beneficial effects of the present invention:

[0049] 1. The present invention can diagnose and characterize damage conditions based on apparent damage images of civil engineering structures such as buildings, bridges, and roads. The algorithm has fast calculation speed and high accuracy.

[0050] 2. The present invention utilizes a pre-trained multimodal large model as a basis to fully utilize structural health monitoring data and improve the generalization and robustness of the civil engineering damage diagnosis model.

[0051] 3. The present invention utilizes a text-image feature coding alignment module to achieve a unified representation and fusion application of multimodal data of structural damage, so that the structural damage diagnosis model has natural language capabilities and can efficiently perform human-computer interaction based on user questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 This is a flow chart of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model according to the present invention.

[0054] Figure 2 This is a schematic diagram of the structural damage image-text coding alignment module described in the present invention.

[0055] Figure 3 This is a flow chart of the multimodal thinking chain reasoning method described in the present invention.

[0056] Figure 4 This is a schematic diagram of the results of the present invention used for diagnosing apparent damage of various civil engineering structures. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] Combine Figures 1-4 The present invention proposes a method for diagnosing apparent damage of civil engineering structures based on a multimodal large model, the method comprising the following steps:

[0059] Step 1: Organize damage images of various structures and supplement them with text descriptions of damage features. Use predefined prompt words and answer labels to standardize training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions.

[0060] Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis. Use a semantic segmentation model to remove background environmental information and complete the visual thought chain reasoning process. Since the large language model processes text information in the form of vector encoding, use an image encoder to map the original structural damage image and the semantic segmentation map into high-dimensional encoding in the same vector space. Use a text-image feature encoding alignment module to connect the semantics. Finally, use the large language model in combination with text prompts to complete language thought chain reasoning and diagnosis generation.

[0061] Step 3: Use cross-entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label, and optimize the quality of the damage description text of the multimodal large model. Due to the large number of overall model parameters and low efficiency of full parameter training, use an efficient fine-tuning optimization method based on low-rank adapters to add parallel learnable low-rank matrices to the original model layer to reduce the number of parameters that need to be trained. Use multimodal datasets to iteratively train until the multimodal large model converges.

[0062] Step 4: Use the multimodal large model that has completed fine-tuning training to diagnose structural damage; input the structural apparent damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

[0063] The step 1 is specifically as follows:

[0064] Step 1.1: Collect damage images of civil engineering structures and perform data cleaning, rotation, and folding data enhancement processing; manually supplement text descriptions of defect category, shape, and severity damage feature information;

[0065] Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as json files. Each data in the dataset is in the format of {"image", "prompt", "label"}, which are the image path, prompt word, and label respectively.

[0066] In step 2, the structural damage image is normalized. For image I, the processing method is as follows:

[0067]

[0068] The normalized image is input to the downsampling encoder, which includes two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0069] Y = ReLU(X*W+b)

[0070] ReLU(x)=max(0,x)

[0071] Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code;

[0072] The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0073]

[0074] in, is the transposed convolution calculation, W is the transposed convolution kernel, X is the input feature code, b is the bias, and Y is the output semantic segmentation map.

[0075] The computer vision model outputs a semantic segmentation map of the damage area and the background, which is used as a visual reasoning process to eliminate irrelevant information and avoid interference with damage diagnosis.

[0076] In step 2, the damaged original image and semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence is obtained. The image blocks are mapped into vector codes through the linear projection layer:

[0077] z=[t1E;t2E;…;t N E]+E pos

[0078] Among them, E is the linear projection transformation, E pos Encode the position of the graphic block;

[0079] Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature code X = (x1, x2, ..., x n );

[0080] X is input into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism guides the model to retrieve information related to text semantics from image features; the dot-product attention mechanism calculates the correlation between the query vector Q and the image feature encoding X, and the weighted summation based on the attention weights is used to obtain the final image encoding that is consistent with the text semantics:

[0081]

[0082] Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

[0083] In step 2, combined with the input prompt word, the pre-trained large language model is used to perform multimodal thought chain fusion reasoning on the image and text encoding. The reasoning process is as follows:

[0084] T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )}

[0085] p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q)

[0086]

[0087]

[0088] Among them, T CoT is the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, and output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

[0089] The model integrates damage semantic segmentation image encoding through a multimodal thinking chain reasoning process, providing more prior information for damage diagnosis and generating more accurate damage description text.

[0090] In step 3, when using a large multimodal model to generate text descriptions, a token encoding is generated each time, and the target word in the training data is used as the correct label. The cross-entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label. The loss function for the sample is calculated as follows:

[0091]

[0092] Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

[0093] In step 3, a parallel learnable low-rank adapter is added to the multimodal large model. The calculation formula for each layer is as follows:

[0094] h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o

[0095] Where x is the input, h is the output, and W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrices A and B. i is the input dimension, o is the output dimension, and r is a hyperparameter representing the rank of the parameter matrix. Therefore, the newly added learnable parameter ΔW is much smaller than the original parameter W0. Before training, A is initialized to 0 and B is initialized with a random Gaussian.

[0096] Combined with the loss function, the optimization goal of the multimodal large model is:

[0097] ΔW=argmin ΔW Loss(W0+ΔW).

[0098] In step 4, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damage area and the background; then, the original damage image and the semantic segmentation map are encoded through the image encoder, and the image encoding is aligned with the question text encoding entered by the user using the alignment module; finally, the encoding is fused and inferred through the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

[0099] Example

[0100] This embodiment applies the present invention to the diagnosis of apparent damage of various civil engineering structures. Figure 1 A flow chart of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model is given. Figure 2A flow chart of the multimodal thinking chain reasoning method described in the present invention is given. Figure 3 A schematic diagram of the structural damage image-text coding alignment module of the present invention is given. Figure 4 A schematic diagram of the results of the present invention used for diagnosing apparent damage of various civil engineering structures is given.

[0101] Step 1 specifically involves screening images for damage such as concrete cracks, holes, spalling, exposed and corroded steel bars, as well as road potholes, cracks in bridge steel box girders, and debonding of steel component protective coatings. Textual descriptions of these damage characteristics were supplemented to create a multimodal dataset of 11,722 matching sets of structural damage image and textual description information.

[0102] Step 2 specifically involves generating a semantic segmentation map using a computer vision model consisting of two convolutional layers and two transposed convolutional layers. VisualGLM-6B is selected as the base multimodal large model for image encoding, and pre-trained query vectors from its Q-Former module are used to perform feature alignment between the image and text encoding.

[0103] Specifically, step three involves the following: Because the image encoder needs to simultaneously encode the lesion semantic segmentation map, low-rank adapters are added to layers 0-37 of the image encoder to learn the encoding scheme for these new images and better extract lesion features. Furthermore, to improve the accuracy and professionalism of the model description, low-rank adapters are added to layers 0 and 14 of the large language model. The rest of the large language model and the semantic segmentation model are frozen. The entire model was trained for 20 epochs on an A10 graphics card with 24GB of video memory.

[0104] The step 4 is as follows: input the images of concrete exfoliation and exposed reinforcement and steel component protective layer peeling and rusting into the fine-tuned model, such as Figure 4 As shown in the figure, the model first generates a semantic segmentation map of the damage area and the background, and then can conduct human-computer interaction based on user questions to make an accurate diagnosis of the damage category and damage characteristics.

[0105] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model are implemented.

[0106] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model.

[0107] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0109] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0110] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0111] The above is a detailed introduction to the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model proposed in the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for diagnosing apparent damage of civil engineering structures based on a multimodal large model, characterized by: The method comprises the following steps: Step 1: Organize damage images of various structures and supplement them with text descriptions of damage features. Use predefined prompt words and answer labels to standardize training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions. Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis. Use a semantic segmentation model to remove background environmental information and complete the visual thought chain reasoning process. Use an image encoder to map the original structural damage image and the semantic segmentation map into a high-dimensional encoding in the same vector space. Use a text-image feature encoding alignment module to connect the semantics. Finally, use a large language model combined with text prompts to complete language thought chain reasoning and diagnosis generation. Step 3: Use cross-entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label to optimize the quality of the damage description text of the multimodal large model; use an efficient fine-tuning optimization method based on low-rank adapters to add parallel learnable low-rank matrices to the original model layer to reduce the number of parameters that need to be trained; use multimodal datasets to iteratively train until the multimodal large model converges; Step 4: Use the multimodal large model that has completed fine-tuning training to diagnose structural damage; input the structural apparent damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: Step 1.1: Collect damage images of civil engineering structures and perform data cleaning, rotation, and folding data enhancement processing; manually supplement text descriptions of defect category, shape, and severity damage feature information; Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as a json file. Each data entry in the dataset is in the format of {"image","prompt","label"}, which are the image path, prompt word, and label respectively.

3. The method according to claim 1, characterized in that In step 2, the structural damage image is normalized. For image I, the processing method is as follows: The normalized image is input to the downsampling encoder, which includes two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows: Y = ReLU(X*W+b) ReLU(x)=max(0,x) Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code; The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows: in, is the transposed convolution calculation, W is the transposed convolution kernel, X is the input feature code, b is the bias, and Y is the output semantic segmentation map.

4. The method according to claim 3, characterized in that In step 2, the damaged original image and semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence is obtained. The image blocks are mapped into vector codes through the linear projection layer: z=[t1E;t2E;…;t N E]+E pos Among them, E is the linear projection transformation, E pos Encode the position of the graphic block; Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature code X = (x1, x2, ..., x n ); X is input into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism guides the model to retrieve information related to text semantics from image features; the dot-product attention mechanism calculates the correlation between the query vector Q and the image feature encoding X, and the weighted summation based on the attention weights is used to obtain the final image encoding that is consistent with the text semantics: Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

5. The method according to claim 4, characterized in that In step 2, combined with the input prompt word, the pre-trained large language model is used to perform multimodal thought chain fusion reasoning on the image and text encoding. The reasoning process is as follows: T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )} p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q) Among them, T CoT is the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, and output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

6. The method according to claim 1, characterized in that In step 3, when using a large multimodal model to generate text descriptions, a token encoding is generated each time, and the target word in the training data is used as the correct label. The cross-entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label. The loss function for the sample is calculated as follows: Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

7. The method according to claim 6, characterized in that In step 3, a parallel learnable low-rank adapter is added to the multimodal large model. The calculation formula for each layer is as follows: h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o Where x is the input, h is the output, W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrices A and B. i is the input dimension, o is the output dimension, and r is a hyperparameter representing the rank of the parameter matrix. Before training, A is initialized to 0 and B is initialized with a random Gaussian. Combined with the loss function, the optimization goal of the multimodal large model is: ΔW=argmin ΔW Loss(W0+ΔW)。 8. The method according to claim 1, characterized in that In step 4, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damage area and the background; then, the original damage image and the semantic segmentation map are encoded through the image encoder, and the image encoding is aligned with the question text encoding entered by the user using the alignment module; finally, the encoding is fused and inferred through the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images