Civil engineering structure apparent damage diagnosis method based on multi-modal large model

Abstract

The invention provides a civil engineering structure apparent damage diagnosis method based on a multi-modal large model. The method comprises the steps of structure apparent damage multi-modal data unified representation and alignment fusion, a structure damage identification reasoning method based on a multi-modal thinking chain, multi-modal data set construction and large model fine adjustment, and civil engineering structure damage identification and feature description. According to the method, vector coding of a structure damage image is completed by using a multi-modal large model, the structure damage image is aligned with text coding features, a damage semantic segmentation map is generated by calling a computer vision model to serve as a visual thinking chain, and a language thinking chain is formed by constructing a fine tuning data set; and reasoning of multi-modal information coding and generation of structural damage diagnosis are completed by the pre-trained large language model. According to the method, a multi-modal large model is used as a base, and a multi-modal thinking chain reasoning method is fused, so that the model can perform man-machine interaction through a natural language, various structural damages are efficiently diagnosed, and the generalization ability and accuracy of the structural damage diagnosis model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of structural damage identification and artificial intelligence large model, and in particular relates to a method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model. Background Art

[0002] During long-term service, structures such as buildings, bridges, and roads will inevitably be damaged by the external environment, including cracks, holes, spalling, exposed reinforcement in concrete, cracks, shedding of protective layers, rust in steel components, cracks and potholes in asphalt pavement, etc. These damages will not only affect the appearance of the structure, but may also have a serious impact on the safety and stability of the structure. In the process of traditional civil engineering operation and maintenance, inspections are mainly carried out visually by inspectors, which is not only time-consuming and labor-intensive, but also easily affected by the subjective influence of inspectors.

[0003] In recent years, computer vision algorithms have shown great advantages in automatic damage detection, greatly improving the detection efficiency and accuracy of concrete damage identification. Such methods mainly include image classification, target detection, semantic segmentation, and instance segmentation, which can identify damage categories and divide damage areas from qualitative and quantitative perspectives. However, these models are difficult to describe the characteristic information such as the size and location of the damage in detail in the form of natural language, nor can they describe the severity of the damage. In addition, in the field of civil engineering, there are various types of structures such as buildings, bridges, and roads. The background information of these structures in the actual environment is different and there are many types of defects. These characteristics pose great challenges to the robustness and generalization of existing structural damage identification models, making it difficult for existing computer vision small models to cover various types of damage.

[0004] With the introduction of the highly parallelizable model framework Transformer by Google, the parameter scale of deep learning models has continued to expand, and pre-trained large language models have emerged with many capabilities that small models do not have, such as following natural language instructions and step-by-step reasoning. In order to extend the powerful language capabilities of large language models to visual modalities, many researchers have tried to map multimodal features such as images to a feature space consistent with language, so that large models can understand multimodal information and perform learning and reasoning. For example, image encoders and large language models are connected through projection matrices, and training is used to eliminate model illusions caused by semantic misalignment between text and images. These large multimodal models have been pre-trained on a large amount of general image-text data, and have shown strong understanding capabilities for both language and vision.

[0005] However, since the damage to civil engineering structures during operation and maintenance is far less than normal, and the backgrounds of various types of damage are complex and the image collection process is cumbersome, image data containing concrete damage will rarely appear in general pre-training datasets. In addition, in addition to damage categories such as cracks and holes that are widely present in the industrial field, there are also damage types unique to the civil engineering field such as concrete spalling and steel bar exposure and corrosion. The diversity and imbalance of civil engineering structural damage result in less damage identification field knowledge learned by the general multimodal large model, making it difficult to directly accurately identify and describe structural damage. Therefore, based on the open source pre-training large model, the present invention eliminates complex background information by integrating multimodal thinking chain reasoning, constructs a text-image multimodal dataset and combines it with an efficient fine-tuning strategy to achieve accurate identification and feature description of civil engineering structural damage. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art and meet the actual needs, and propose a method for diagnosing apparent damage of civil engineering structures based on a multimodal large model. The method is suitable for diagnosing damage and describing features based on damage images of various civil engineering structures.

[0007] The present invention is implemented by the following technical scheme. The present invention proposes a method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model. The method comprises the following steps:

[0008] Step 1: Organize the damage images of various structures and supplement the text descriptions of damage features. Use predefined prompt words and answer labels to standardize the training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions.

[0009] Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis; use a semantic segmentation model to remove background environmental information and complete the visual thinking chain reasoning process; use an image encoder to map the original structural damage image and the semantic segmentation map into high-dimensional encodings in the same vector space, and use a text-image feature encoding alignment module to connect the semantics, and finally use a large language model in combination with text prompts to complete language thinking chain reasoning and diagnosis generation;

[0010] Step 3: Use cross entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label, and optimize the quality of the multimodal large model damage description text; use an efficient fine-tuning optimization method based on a low-rank adapter to add a parallel learnable low-rank matrix to the original model layer to reduce the number of parameters that need to be trained; use a multimodal dataset to iteratively train until the multimodal large model converges;

[0011] Step 4: Use the fine-tuned multimodal large model to diagnose structural damage; input the structural surface damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and the background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

[0012] Furthermore, the step 1 is specifically as follows:

[0013] Step 1.1, collect damage images of civil engineering structures and perform data cleaning, rotate and fold data enhancement processing; manually supplement text descriptions of defect category, shape and severity damage feature information;

[0014] Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as json files. The format of each data in the dataset is {"image", "prompt", "label"}, which are image path, prompt word and label respectively.

[0015] Furthermore, in step 2, the structural damage image is normalized. For image I, the processing method is as follows:

[0016]

[0017] The normalized image is input into the downsampling encoder, which includes two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0018] Y = ReLU(X*W+b)

[0019] ReLU(x)=max(0,x)

[0020] Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code;

[0021] The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0022]

[0023] in, It is a transposed convolution calculation, W is the transposed convolution kernel, X is the input feature encoding, b is the bias, and Y is the output semantic segmentation map.

[0024] Furthermore, in step 2, the damaged original image and the semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence t∈R is obtained. N×(P2C) ; The image block is mapped into a vector encoding through a linear projection layer:

[0025] z=[t1E;t2E;…;t N E]+E pos

[0026] Among them, E is the linear projection transformation, E pos Encode the position of the graphic block;

[0027] Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature code X = (x1, x2, ..., x n );

[0028] X is fed into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism is used to guide the model to retrieve information related to text semantics from image features; the dot product attention mechanism is used to calculate the correlation between the query vector Q and the image feature encoding X, and the final image encoding that is consistent with the text semantics is obtained by weighted summation according to the attention weight:

[0029]

[0030] Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

[0031] Furthermore, in step 2, combined with the input prompt words, the pre-trained large language model is used to perform multimodal thinking chain fusion reasoning on the image and text encoding. The reasoning process is as follows:

[0032] T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )}

[0033] p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q)

[0034]

[0035]

[0036] Among them, T CoTis the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

[0037] Furthermore, in step 3, when using the multimodal large model to generate text descriptions, a token encoding is generated each time, the target word in the training data is used as the correct label, and the cross entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label; the loss function calculation method of the sample is as follows:

[0038]

[0039] Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

[0040] Furthermore, in step 3, a parallel learnable low-rank adapter is added to the multimodal large model, and the calculation formula for each layer is as follows:

[0041] h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o

[0042] Where x is the input, h is the output, W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrices A and B, i is the input dimension, o is the output dimension, and r is the hyperparameter representing the rank of the parameter matrix. Before training, A is initialized to 0 and B is initialized with a random Gaussian.

[0043] Combined with the loss function, the optimization goal of the multimodal large model is:

[0044] ΔW=argmin ΔW Loss(W0+ΔW).

[0045] Furthermore, in step four, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damaged area and the background; then, the original damage image and the semantic segmentation map are encoded by the image encoder, and the image encoding is aligned with the question text encoding input by the user using an alignment module; finally, the encoding is fused and inferred by the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

[0046] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model when executing the computer program.

[0047] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model.

[0048] Beneficial effects of the present invention:

[0049] 1. The present invention can diagnose and characterize damage conditions based on apparent damage images of civil engineering structures such as buildings, bridges, and roads. The algorithm has fast calculation speed and high accuracy.

[0050] 2. The present invention utilizes a pre-trained multimodal large model as a base, realizes full utilization of structural health monitoring data, and improves the generalization and robustness of the civil engineering damage diagnosis model.

[0051] 3. The present invention utilizes a text-image feature coding alignment module to realize a unified representation fusion application of multimodal data of structural damage, so that the structural damage diagnosis model has natural language capabilities and can efficiently perform human-computer interaction based on user questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 The present invention is a flowchart of the method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model.

[0054] Figure 2 This is a schematic diagram of the structural damage image-text coding alignment module described in the present invention.

[0055] Figure 3 This is a flow chart of the multimodal thinking chain reasoning method described in the present invention.

[0056] Figure 4 It is a schematic diagram of the results of the present invention being used for diagnosing apparent damage of various civil engineering structures. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Combination Figure 1-Figure 4 The present invention proposes a method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model, the method comprising the following steps:

[0059] Step 1: Organize the damage images of various structures and supplement the text descriptions of damage features. Use predefined prompt words and answer labels to standardize the training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions.

[0060] Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis; use a semantic segmentation model to remove background environmental information and complete the visual thinking chain reasoning process; since the large language model processes text information in the form of vector encoding, use an image encoder to map the original image of structural damage and the semantic segmentation map into high-dimensional encoding in the same vector space, and use a text-image feature encoding alignment module to connect the semantics, and finally use the large language model in combination with text prompts to complete the language thinking chain reasoning and diagnosis generation;

[0061] Step 3: Use cross entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label, and optimize the quality of the multimodal large model damage description text; Since the overall number of model parameters is large and the efficiency of full parameter training is low, an efficient fine-tuning optimization method based on a low-rank adapter is used to add a parallel learnable low-rank matrix to the original model layer to reduce the number of parameters that need to be trained; Use a multimodal dataset to iteratively train until the multimodal large model converges;

[0062] Step 4: Use the fine-tuned multimodal large model to diagnose structural damage; input the structural surface damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and the background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

[0063] The step 1 is specifically as follows:

[0064] Step 1.1, collect damage images of civil engineering structures and perform data cleaning, rotate and fold data enhancement processing; manually supplement text descriptions of defect category, shape and severity damage feature information;

[0065] Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as json files. The format of each data in the dataset is {"image", "prompt", "label"}, which are image path, prompt word and label respectively.

[0066] In step 2, the structural damage image is normalized. For image I, the processing method is as follows:

[0067]

[0068] The normalized image is input into the downsampling encoder, which includes two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0069] Y = ReLU(X*W+b)

[0070] ReLU(x)=max(0,x)

[0071] Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code;

[0072] The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows:

[0073]

[0074] in, It is a transposed convolution calculation, W is the transposed convolution kernel, X is the input feature encoding, b is the bias, and Y is the output semantic segmentation map.

[0075] The computer vision model outputs a semantic segmentation map of the damaged area and the background, which serves as a visual reasoning process to eliminate irrelevant information and avoid interference with damage diagnosis.

[0076] In step 2, the damaged original image and the semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence is obtained. The image blocks are mapped into vector encodings through a linear projection layer:

[0077] z=[t1E;t2E;…;t N E]+E pos

[0078] Where E is the linear projection transformation, E pos Encode the position of the graphic block;

[0079] Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature code X = (x1, x2, ..., x n );

[0080] X is fed into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism is used to guide the model to retrieve information related to text semantics from image features; the dot product attention mechanism is used to calculate the correlation between the query vector Q and the image feature encoding X, and the final image encoding that is consistent with the text semantics is obtained by weighted summation according to the attention weight:

[0081]

[0082] Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

[0083] In step 2, combined with the input prompt words, the pre-trained large language model is used to perform multimodal thinking chain fusion reasoning on the image and text encoding. The reasoning process is as follows:

[0084] T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )}

[0085] p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q)

[0086]

[0087]

[0088] Among them, T CoT is the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

[0089] The model integrates the damage semantic segmentation image encoding through the multimodal thinking chain reasoning process, provides more prior information for damage diagnosis, and can generate more accurate damage description text.

[0090] In step 3, when using a multimodal large model to generate text descriptions, a token encoding is generated each time, the target word in the training data is used as the correct label, and the cross entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label; the loss function calculation method of the sample is as follows:

[0091]

[0092] Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

[0093] In step 3, a parallel learnable low-rank adapter is added to the multimodal large model. The calculation formula for each layer is as follows:

[0094] h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o

[0095] Among them, x is the input, h is the output, W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrix A and B, i is the input dimension, o is the output dimension, and r is the hyperparameter representing the rank of the parameter matrix. Therefore, the newly added learnable parameter ΔW is much smaller than the original parameter W0. Before training, A is initialized to 0 and B is initialized with a random Gaussian;

[0096] Combined with the loss function, the optimization goal of the multimodal large model is:

[0097] ΔW=argmin ΔW Loss(W0+ΔW).

[0098] In step 4, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damaged area and the background; then, the original damage image and the semantic segmentation map are encoded by the image encoder, and the image encoding is aligned with the question text encoding input by the user using the alignment module; finally, the encoding is fused and inferred by the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

[0099] Example

[0100] This embodiment applies the present invention to the diagnosis of apparent damage of various civil engineering structures. Figure 1 A flow chart of the method for diagnosing apparent damage of civil engineering structures based on a multimodal large model according to the present invention is given. Figure 2A flow chart of the multimodal thinking chain reasoning method described in the present invention is provided. Figure 3 A schematic diagram of the structural damage image-text coding alignment module of the present invention is given. Figure 4 A schematic diagram of the results of the present invention being used for diagnosing apparent damage of various civil engineering structures is provided.

[0101] The step 1 specifically includes: screening damage images such as concrete cracks, holes, spalling, exposed steel bars, steel bar corrosion, road potholes, bridge steel box girder cracks, and steel component protective layer shedding. The text description of damage feature information is supplemented, and 11,722 sets of mutually matched structural damage image-text description multimodal datasets are established.

[0102] The second step is specifically as follows: a computer vision model including two convolutional layers and two transposed convolutional layers is used to generate a semantic segmentation map. VisualGLM-6B is selected as the basic multimodal large model for image encoding, and the pre-trained query vector in its Q-Former module is used to complete the feature alignment of image-text encoding.

[0103] The specific step three is as follows: Since the image encoder needs to encode the damage semantic segmentation map at the same time, in order to learn the encoding method of these new images to better extract damage features, low-rank adapters are added to the 0-37 layers of the image encoder. At the same time, in order to improve the accuracy and professionalism of the model description, low-rank adapters are added to the 0th and 14th layers of the large language model, and the other parts of the large language model and the semantic segmentation model are frozen. The entire model was trained for 20 epochs on an A10 graphics card with 24GB of video memory.

[0104] The step 4 is as follows: input the images of concrete peeling and exposed reinforcement and steel component protective layer peeling and rusting into the fine-tuned model, such as Figure 4 As shown in the figure, the model first generates a semantic segmentation map of the damaged area and the background, and then can perform human-computer interaction based on user questions to make an accurate diagnosis of the damage category and damage characteristics.

[0105] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model when executing the computer program.

[0106] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for diagnosing apparent damage of civil engineering structures based on a multi-modal large model.

[0107] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0109] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0110] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0111] The above is a detailed introduction to the apparent damage diagnosis method for civil engineering structures based on a multimodal large model proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for diagnosing apparent damage of civil engineering structures based on a multimodal large model, characterized in that: The method comprises the following steps: Step 1: Organize the damage images of various structures and supplement the text descriptions of damage features. Use predefined prompt words and answer labels to standardize the training samples to form a multimodal dataset of civil engineering structure damage with matching images and text descriptions. Step 2: Design a large multimodal model architecture for civil engineering structural damage diagnosis; use a semantic segmentation model to remove background environmental information and complete the visual thinking chain reasoning process; use an image encoder to map the original structural damage image and the semantic segmentation map into high-dimensional encodings in the same vector space, and use a text-image feature encoding alignment module to connect the semantics, and finally use a large language model in combination with text prompts to complete language thinking chain reasoning and diagnosis generation; Step 3: Use cross entropy loss to measure the difference between the predicted probability distribution of the multimodal large model and the correct label, and optimize the quality of the multimodal large model damage description text; use an efficient fine-tuning optimization method based on a low-rank adapter to add a parallel learnable low-rank matrix to the original model layer to reduce the number of parameters that need to be trained; use a multimodal dataset to iteratively train until the multimodal large model converges; Step 4: Use the fine-tuned multimodal large model to diagnose structural damage; input the structural surface damage image into the multimodal large model, and the multimodal large model generates a semantic segmentation map of the damaged area and the background; input the user's question text about the structural damage status, and the multimodal large model uses natural language to describe the damage category and damage characteristics based on the prompt words.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: Step 1.1, collect damage images of civil engineering structures and perform data cleaning, rotate and fold data enhancement processing; manually supplement text descriptions of defect category, shape and severity damage feature information; Step 1.2: Use predefined prompt words and answer labels to standardize the training samples and save them as json files. The format of each data in the dataset is {"image", "prompt", "label"}, which are image path, prompt word and label respectively.

3. The method according to claim 1, characterized in that: In step 2, the structural damage image is normalized. For image I, the processing method is as follows: The normalized image is input into the downsampling encoder, which contains two layers of convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows: Y = ReLU(X*W+b) ReLU(x)=max(0,x) Among them, * is the convolution calculation, W is the convolution kernel, X is the input feature map, b is the bias, and Y is the output feature code; The feature code is input into the upsampling decoder, which contains two layers of transposed convolution calculations with a kernel size of 3 and a stride of 1, and uses the ReLU activation function. The calculation formula for each layer is as follows: in, It is a transposed convolution calculation, W is the transposed convolution kernel, X is the input feature encoding, b is the bias, and Y is the output semantic segmentation map.

4. The method according to claim 3, characterized in that In step 2, the damaged original image and the semantic segmentation map are divided into P×P image blocks. For an image with a resolution of H×W and a number of channels of C, the image block sequence is obtained. The image blocks are mapped into vector encodings through a linear projection layer: z=[t1E;t2E;…;t N E]+E pos Where E is the linear projection transformation, E pos Encode the position of the graphic block; Input z into the Transformer encoder composed of alternating self-attention and multi-layer perceptron blocks for feature extraction, and obtain the image feature encoding X = (x1, x2, ..., x n ); X is fed into the image-text encoding alignment module, which uses the pre-trained query vector Q = (q1, q2, ..., q m ), the cross-attention mechanism is used to guide the model to retrieve information related to text semantics from image features; the dot product attention mechanism is used to calculate the correlation between the query vector Q and the image feature encoding X, and the final image encoding that is consistent with the text semantics is obtained by weighted summation according to the attention weight: Among them, Attention is the attention weight, and VR is the structural damage image encoding that is semantically consistent with the text.

5. The method according to claim 4, characterized in that In step 2, combined with the input prompt words, the pre-trained large language model is used to perform multimodal thinking chain fusion reasoning on the image and text encoding. The reasoning process is as follows: T CoT ={I,(x1,v1,y1),…,(x n ,v n ,y n )} p(A,VR|T,Q)=p(A|T,Q,VR)·p(VR|T,Q) Among them, T CoT is the input thinking chain prompt word, which contains the task instruction I and the sample prompt triple, that is, input x, reasoning process v, output y; Q is the user input question; VR is the structural damage semantic segmentation image encoding, providing visual prior information; A is the structural damage description output by the large language model, a i is the i-th encoding output by the large language model.

6. The method according to claim 1, characterized in that In step 3, when using a multimodal large model to generate text descriptions, a token encoding is generated each time, the target word in the training data is used as the correct label, and the cross entropy loss is used to measure the difference between the model's predicted probability distribution and the correct label; the loss function calculation method of the sample is as follows: Among them, N is the number of samples, J is the predicted sentence length, T is the prompt word, Q is the input question, VR is the visual reasoning result, Encode the jth token of the output of the i-th sample, is the probability distribution of the jth correct code in the i-th sample.

7. The method according to claim 6, characterized in that In step 3, a parallel learnable low-rank adapter is added to the multimodal large model. The calculation formula for each layer is as follows: h=W0x+ΔWx=W0x+ABx A∈R i×r ,B∈R r×o Where x is the input, h is the output, W0 is the original parameter; ΔW is the newly added learnable parameter, which is obtained by multiplying the matrices A and B, i is the input dimension, o is the output dimension, and r is the hyperparameter representing the rank of the parameter matrix. Before training, A is initialized to 0 and B is initialized with a random Gaussian. Combined with the loss function, the optimization goal of the multimodal large model is: ΔW=argmin ΔW Loss(W0+ΔW)。 8. The method according to claim 1, characterized in that In step 4, the structural apparent damage image is input into the multimodal large model, and the multimodal large model calls the pre-trained computer vision model to generate a semantic segmentation map of the damaged area and the background; then, the original damage image and the semantic segmentation map are encoded by the image encoder, and the image encoding is aligned with the question text encoding input by the user using the alignment module; finally, the encoding is fused and inferred by the large language model, and the natural language description of the damage category and damage characteristics is completed according to the prompt words.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images