Structural multi-type apparent damage identification and detection report generation method based on multi-modal large model

Abstract

The invention provides a structure multi-type apparent damage identification and detection report generation method based on a multi-modal large model. According to the method, the image recognition result and a natural language generation technology are fused, so that full-process automation from damage detection to report generation is realized, and the intelligent level of building structure health monitoring is remarkably improved. According to the method, automatic identification and accurate positioning of multi-type damage of the appearance of civil engineering structures such as buildings and bridges can be realized, and objective and normalized maintenance suggestions and detection reports are generated in combination with a vision-language multi-modal modeling method, so that the intelligence and automation level of structural appearance damage detection and operation and maintenance management is improved.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision, intelligent operation and maintenance of civil infrastructure, and visual-language multimodal large models, and in particular to a method for identifying and generating reports of structural multi-type apparent damage based on multimodal large models. Background Technology

[0002] With the continuous development of urbanization and infrastructure construction, the demand for monitoring and maintaining the health status of civil structures such as buildings and bridges is increasing. Structural surface damage (such as cracks, spalling, and corrosion) not only affects the durability and safety of buildings but may also trigger catastrophic accidents. Therefore, how to achieve efficient identification, accurate location, and standardized report generation of structural surface damage has become an important research direction in the field of intelligent operation and maintenance of civil engineering.

[0003] Existing structural apparent damage identification technologies mainly include the following categories:

[0004] (1) Appearance damage identification methods based on image classification: Typical methods include convolutional neural networks (CNN), residual networks (ResNet), EfficientNet, etc., which can distinguish single types of appearance damage such as cracks and peeling, and have the advantage of simple implementation, but it is difficult to recognize multiple types of damage at the same time and the generalization ability is insufficient.

[0005] (2) Appearance damage localization methods based on semantic segmentation: Represented by models such as U-Net, DeepLab, and SegFormer, these methods obtain fine boundary information of the damage region through pixel-level segmentation. However, these methods are usually trained for a single scene, have weak cross-type adaptability, and are easily disturbed in complex backgrounds.

[0006] (3) Multi-type apparent damage identification methods based on target detection: Common frameworks such as YOLO and Faster R-CNN can detect multiple types of damage simultaneously. However, due to the often subtle and diverse scales of damage features, their detection accuracy is still limited, and existing frameworks lack the ability to extend from detection results to diagnosis and report generation.

[0007] In summary, existing structural apparent damage identification technologies have the following main shortcomings:

[0008] First, the identification objects are limited to a single or a few types of damage, which makes it difficult to meet the comprehensive detection needs of multiple damages in complex engineering environments;

[0009] Secondly, classification, segmentation, and detection methods are mostly used independently, lacking a unified intelligent framework, making it difficult to achieve multi-task collaboration;

[0010] Third, the repair recommendations and inspection reports following the identification of building surface damage still rely on manual generation, which has the problems of strong subjectivity and lack of standardization, making it difficult to guarantee the objectivity and consistency of the inspection results.

[0011] In recent years, the rapid development of large-scale vision-language models has provided new ideas for solving the above problems. These models, by jointly modeling image features and text semantics, possess cross-modal understanding and generation capabilities, and perform well in general tasks such as image description, question answering, and report generation. Some studies have attempted to introduce general-purpose large-scale vision-language models such as CLIP, BLIP, and GPT-4V into civil engineering surface damage recognition tasks to achieve the association between damage images and text descriptions. However, because these models are mainly trained on general-purpose data and lack knowledge of the civil engineering domain, their accuracy is insufficient when directly applied to structural surface damage detection, making it difficult to meet actual engineering needs.

[0012] Meanwhile, academics have also explored domain-adaptive visual-language models, combining apparent damage image features with professional knowledge bases and maintenance rule bases to attempt to achieve auxiliary interpretation of detection results and generation of preliminary reports. However, overall, existing research is still in the exploratory stage and has not yet formed an integrated technical system for unified identification, precise localization, and automatic generation of maintenance suggestions and detection reports for multiple types of damage.

[0013] In summary, existing technologies still have significant shortcomings in the unified identification of multiple types of damage to building surfaces, intelligent interpretation of detection results, and generation of standardized reports. There is an urgent need to propose a multimodal intelligent solution that combines visual recognition and language generation to improve the efficiency and intelligence level of structural damage detection and operation and maintenance management.

[0014] To address the aforementioned issues, this invention proposes a method for identifying and generating inspection reports of multiple types of apparent damage to structures based on a multimodal large model. This method enables automated identification and precise location of multiple types of apparent damage to civil engineering structures such as buildings and bridges. Combined with a visual-linguistic multimodal modeling method, it generates objective and standardized maintenance suggestions and inspection reports, thereby improving the intelligence and automation level of structural apparent damage detection and maintenance management. Summary of the Invention

[0015] The purpose of this invention is to address the problems in existing technologies by proposing a method for identifying and generating reports on various types of apparent damage to structures based on a multimodal large model. This method aims to solve systemic challenges such as the automated identification, precise location, and report generation of apparent damage to building structures. It can be widely applied in fields such as building visualization, structural health monitoring, disaster prevention and mitigation, and intelligent construction.

[0016] This invention is achieved through the following technical solution: This invention proposes a method for identifying and generating reports on structural multi-type apparent damage based on a multimodal large model, the method comprising:

[0017] Step 1: For images of various types of apparent damage to building structures, establish a lightweight convolutional neural network classification model based on depthwise separable convolution and inverted residual structure;

[0018] Step 2: Based on image damage classification labels, establish a unified semantic segmentation network for multiple types of apparent damage to building structures under complex backgrounds and strong interference.

[0019] Step 3: Integrate apparent damage classification labels and pixel-level semantic segmentation results to construct a damage quantification method based on image recognition and create a large visual-linguistic multimodal model for image description;

[0020] Step 4: Based on the trained visual-language multimodal large model, integrate image recognition results to achieve intelligent generation of a comprehensive structural damage detection report.

[0021] Furthermore, step one specifically includes:

[0022] Step 11: Construct a multi-label dataset and XML annotation and parsing system;

[0023] Steps 1 and 2: Build a lightweight network based on depthwise separable convolution and optimize it using a binary cross-entropy loss function with Logits; specifically, this includes: constructing a lightweight network architecture that decomposes the standard convolution operation into two parts: depthwise convolution and pointwise convolution.

[0024] In the depthwise convolution stage, each input channel uses a separate convolution kernel for convolution operations, without cross-channel mixing;

[0025]

[0026] In the formula, M and N are the kernel sizes, and k represents the input channel index;

[0027] Pointwise convolution uses 1×1 convolutions to perform linear combinations along the channel dimension:

[0028]

[0029] In the formula, K is the number of input channels, and l represents the output channel index;

[0030] Introducing an inverted residual structure:

[0031]

[0032] In the formula, x is the input feature map with dimensions H×W×C. in, To extend the layer, the number of channels is increased from C. in Expanded to t·C in Where t is the expansion factor, DWConv 3×3 For depthwise convolution, a 3×3 convolution is performed independently on each channel to extract spatial features while keeping the number of channels constant. For the projection layer, compress the number of channels back to the target output channel number C. out x+... represents residual connections. If the input x and output y have the same dimension, then element-wise addition is performed to achieve residual learning.

[0033] Step 13: Build a multi-label inference system based on Sigmoid and visualize the results.

[0034] Furthermore, step two specifically includes:

[0035] Step 21: Construct the Multi-Type Scanning Structure Perception Visual State Space Module DS-SAVSS; this module works collaboratively through four paths, which include: diagonal serpentine scanning traversing pixels through alternating diagonal paths, circular scanning spiraling outward from the image center, dense grid scanning using a high-coverage zigzag path, and horizontal and vertical alternating scanning as a general supplementary path.

[0036] Step 22: Create a multi-scale feature fusion module ICBAM based on an improved convolutional block attention mechanism;

[0037] Steps 2 and 3: Construct a lightweight parallel branch multi-scale deformable convolution module MS-DCNN based on the feature processing and prediction segmentation head using parallel branches and multi-scale deformable convolution;

[0038] Step 24: Use BCE+Dice hybrid loss to improve the robustness of the model to unbalanced pixels, and use 6 indicators to comprehensively evaluate the damage recognition performance.

[0039] Furthermore, in step 22, global pooling is performed on the four-level feature maps extracted by DS-SAVSS to extract global information in the channel dimension and generate weight coefficients for each channel, thereby strengthening the key channel features and suppressing redundant information; the calculation expression is as follows:

[0040] F CA =σ(MLP(GlobalPooling(F) i ))) (9)

[0041] In the formula, F iThe input feature map has dimensions (H×W×C), where H is the height, W is the width, and C is the number of channels; GlobalPooling indicates global pooling of the feature map; MLP is a multilayer perceptron used to map the globally pooled features to channel attention weights; σ is the sigmoid activation function used to map the output to the range [0,1]; F CA It is the output of channel attention, used to enhance or suppress the features of individual channels.

[0042] Furthermore, in steps two and three, a gating mechanism is created to enable dynamic features for each spatial location and channel, enhancing the model's ability to capture details. A bottleneck convolution with low-rank approximation, BottConv, is embedded therein to map the matrix from high-dimensional space to low-dimensional space.

[0043] Gating mechanism:

[0044] X gate =σ(W gate X norm +b gate (12)

[0045] In the formula, σ is the Sigmoid activation function, and W gate and b gate These are the parameters of the gating mechanism.

[0046] Furthermore, step three specifically includes:

[0047] Step 31: Image damage type identification and segmentation result post-processing; specifically including: (1) image binarization, (2) damage region identification, (3) result visualization and (4) table generation;

[0048] Step 32: Create a large visual-language multimodal model and generate a detection report.

[0049] Furthermore, in step 32, image feature encoding is implemented in the visual encoder part to extract discriminative visual representations; in the language encoder part, the Llama2-7B architecture is used as the core of the language model to encode the text input.

[0050] The cross-modal attention module embeds modal type labels into the encoding results of the two modalities respectively, so as to achieve effective cross-modal feature fusion using the self-attention mechanism under the Transformer framework. In the fused sequence, two enhancement mechanisms are introduced: one is the structural multi-type damage vocabulary mask enhancement mechanism, which is used to strengthen the semantic expression of key damage concepts; the other is the structural multi-type damage concept embedding mechanism, which injects domain prior knowledge into the model to improve the semantic understanding of civil engineering scenarios.

[0051] Finally, the dynamic context building module integrates multimodal information and generates descriptive text step by step through an autoregressive generation loop. After output post-processing, the final structural damage detection report is generated.

[0052] Further, in step four, the structural damage classification model and segmentation model, along with their results, are preprocessed to obtain three types of building structure appearance images, as well as corresponding classification results and damage information summary tables. These three types of building structure appearance images are then input into a trained visual-language multimodal large model. Subsequently, the images are feature-encoded using a visual encoder, converting pixel-level visual information into high-dimensional feature vectors. These visual feature vectors are then intelligently matched with a pre-built building structure knowledge base. After knowledge matching is completed, the large model is invoked to generate natural language.

[0053] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for identifying and generating reports of structural multi-type apparent damage based on a multimodal large model.

[0054] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for generating reports on structural multi-type apparent damage based on a multimodal large model.

[0055] The beneficial effects of this invention are:

[0056] This invention proposes a method for identifying and generating reports on multiple types of structural damage based on a multimodal large model. By fusing image recognition results with natural language generation technology, it achieves full automation from damage detection to report generation, significantly improving the intelligence level of building structural health monitoring. It has the following improvements:

[0057] (1) Unified identification and precise localization of multiple types of damage

[0058] The ray sampling module has been optimized, combining segmented sampling and enhanced sampling of key areas of interest, and dynamically adjusting the sampling density to achieve more refined high-resolution rendering and modeling of areas with rich details such as building edges, corners and textures, resulting in a high-precision improvement;

[0059] (2) High-precision segmentation under complex backgrounds and strong interference

[0060] To address common complex background interference (such as shadows, stains, and texture interference) in building structure images, a multi-type scanning structure-aware visual state space module (DS-SASS) and multi-scale deformable convolution techniques are employed to effectively enhance the model's ability to extract subtle damage features. This module ensures accurate segmentation of various damage regions even in complex backgrounds through dynamic path selection and adaptive feature fusion, improving detection robustness.

[0061] (3) Multimodal result output enhances visualization and report generation capabilities.

[0062] The system provides image-probabilistic dual-view visualization and structured text reports, intuitively displaying damage detection results for easy engineering applications. At the same time, it combines a visual-language multimodal large model to automatically generate standardized reports containing damage descriptions, safety assessments, and maintenance recommendations, achieving full automation from detection to report generation and significantly reducing manual intervention.

[0063] (4) Cross-modal knowledge fusion to ensure the professionalism of the report

[0064] By leveraging a pre-built building structure knowledge base, the system intelligently matches visual features with engineering standards, ensuring that generated reports comply with industry norms. The system automatically references relevant standard clauses to provide accurate maintenance suggestions, avoiding human error.

[0065] (5) Domain adaptation overcomes the limitations of general models.

[0066] For civil engineering scenarios, this study addresses the issue of insufficient accuracy of general vision-language models in engineering inspection by training with professional datasets and embedding domain knowledge. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 This is a flowchart of a method for identifying and generating reports on structural and multi-type apparent damage based on a multimodal large model.

[0069] Figure 2 This is a visualization of the recognition results from a lightweight classification model.

[0070] Figure 3 It is a unified segmentation network diagram of multiple types of apparent damage to building structures under complex backgrounds and strong interference.

[0071] Figure 4 This is a network architecture diagram of the structure-aware visual state space module.

[0072] Figure 5 This is a diagram of a lightweight parallel branch multi-scale deformable convolutional module network architecture.

[0073] Figure 6 This is a diagram showing the results of post-processing after concrete crack identification.

[0074] Figure 7 This is a diagram showing the results of post-processing for concrete spalling identification.

[0075] Figure 8 This is a schematic diagram of a large-scale model architecture for a visual language for building health diagnosis that embeds knowledge of structural damage attributes.

[0076] Figure 9 It is a network diagram of the visual basic model.

[0077] Figure 10 This is a network architecture diagram of the cross-modal mapping module.

[0078] Figure 11 This is a flowchart for generating structural damage detection reports.

[0079] Figure 12 This is a schematic diagram of a structural damage detection report.

[0080] Figure 13 This is a schematic diagram of the identification results in the test report. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] Combination Figures 1-13 This invention proposes a method for identifying and generating reports on structural multi-type apparent damage based on a multimodal large model. The specific steps include:

[0083] Step 1: For images of various types of apparent damage to building structures, establish a lightweight convolutional neural network classification model based on depthwise separable convolution and inverted residual structure.

[0084] Specifically, the following steps are included:

[0085] Step 11: Construct a multi-label dataset and XML annotation and parsing system;

[0086] The steps involved constructing a multi-label dataset and an XML annotation and parsing system, specifically including:

[0087] (1) Organize the file system.

[0088] In the file system architecture design, a three-level hierarchical storage structure is adopted to achieve physical isolation and logical management of data. The original images, annotation files, and partition indexes are stored independently, with dedicated partitions for the training set, validation set, and test set. The balance of data distribution is ensured by dynamically adjusting the ratio of background images to images with apparent damage to the building structure.

[0089] To ensure data quality, low-quality images are first automatically filtered based on a resolution threshold (800×600 pixels); secondly, a perceptual hashing algorithm (95% similarity threshold) is used to remove duplicate samples; and then the integrity of the annotations for each apparent damage image is systematically verified to ensure that it contains at least one valid XML annotation object.

[0090] (2) Establish a multi-label coding system.

[0091] The structural apparent damage classification standard establishes a six-category damage mapping system based on international civil engineering codes. For vectorized representation, a multi-dimensional binary vector coding scheme is adopted, with each 224×224 pixel input image corresponding to a 6-dimensional label vector. The vector elements correspond to six states in a fixed order: background, cracks, spalling, weathering, exposed rebar, and corrosion stains. When damage exists, the corresponding position is marked as 1. This coding system supports multiple damage annotations on a single image, such as [0,1,0,1,0,0] indicating the simultaneous presence of cracks and weathering defects.

[0092] (3) Introduce an XML intelligent parsing system.

[0093] The XML intelligent parsing system adopts a streaming processing architecture and achieves high-performance processing through an event-driven SAX parsing scheme. The system supports parallel parsing of 16 XML files, with a processing speed of 1200 files per second, and features intelligent error correction capabilities, automatically fixing common annotation errors such as unclosed tags and missing attribute values.

[0094] In terms of data extraction, the system acquires damage location information and converts it into a standardized coordinate format. It also maps damage categories to a pre-defined six-category coding system and fully records metadata such as image resolution, capturing device, and timestamps. The system also integrates an intelligent verification mechanism that automatically detects semantic errors such as bounding boxes and abnormal aspect ratios, ensuring the accuracy and consistency of the labeled data.

[0095] (4) Perform data augmentation.

[0096] The data augmentation strategy combines conventional rotation, flipping, cropping, and scaling operations to increase image diversity and thus improve the model's robustness. Simultaneously, it has been specifically optimized for identifying the characteristics of apparent damage in concrete.

[0097] At the image processing level, the system dynamically adjusts the gamma value (80-120%) to simulate the appearance of damage under different lighting conditions and uses mesh deformation technology to restore the angle deviation during actual shooting. To more realistically reflect the engineering site environment, the algorithm also injects sensor noise conforming to the Rayleigh distribution and randomly adds occlusion effects to simulate common surface stains and obstructions.

[0098] In the data standardization process, the system first converts the image pixel values ​​from an integer range of 0-255 to a floating-point representation of 0-1, and then uses dedicated statistical parameters (mean 0.412 / 0.409 / 0.386, standard deviation 0.241 / 0.209 / 0.195) to perform channel standardization.

[0099] Channel pixel value = (iu) / v (1)

[0100] In the formula, i represents the pixel value of each of the three RGB channels, u represents the corresponding mean parameter, and v represents the corresponding annotation difference parameter.

[0101] Steps one and two: Establish a lightweight network based on depthwise separable convolutions and optimize it using a binary cross-entropy loss function with Logits; specifically including:

[0102] (1) Construct a lightweight network architecture and decompose the standard convolution operation into two steps: depthwise convolution and pointwise convolution, thereby reducing the amount of computation.

[0103] In the depthwise convolution stage, each input channel uses a separate convolution kernel for convolution operations, without cross-channel mixing.

[0104]

[0105] In the formula, M and N are the kernel sizes (3×3), and k represents the input channel index.

[0106] Pointwise convolution uses 1×1 convolutions to perform linear combinations along the channel dimension:

[0107]

[0108] In the formula, K is the number of input channels, and l represents the output channel index.

[0109] (2) Introduce an inverted residual structure.

[0110]

[0111] In the formula, x is the input feature map with dimensions H×W×C. in , To extend the layer, the number of channels is increased from C. in Expanded to t·C in Where t is the expansion factor (t = 6), DWConv 3×3 For depthwise convolution, a 3×3 convolution is performed independently on each channel to extract spatial features while keeping the number of channels constant. For the projection layer, compress the number of channels back to the target output channel number C. out x+... represents residual connections. If the input x and output y have the same dimension (i.e., stride is 1 and number of channels is the same), then element-wise addition is performed to achieve residual learning.

[0112] (3) Optimize the objective function.

[0113] For multi-label classification tasks, a binary cross-entropy loss with sigmoid is used to directly optimize the unnormalized scores (logits) of the model output. Its mathematical expression is:

[0114]

[0115] In the formula, N is the sample size, and y i ∈{0,1} represents the true value of the i-th label, z i The unnormalized score is the logits (logits) output by the model. The sigmoid function maps logits to probabilities. To avoid numerical overflow when calculating log(σ(z)) or log(1-σ(z)), the loss function is stabilized using the softplus function. This approach avoids floating-point underflow or overflow issues when z reaches its maximum or minimum value, thus improving training stability.

[0116] Step 13: Construct a sigmoid-based multi-label inference system and visualize the results. This includes:

[0117] (1) Use the Sigmoid activation function for multi-label probability transformation.

[0118] In multi-label classification tasks, each category is independent of the others, and the same image can belong to multiple categories simultaneously. Therefore, it is necessary to model the probability of occurrence for each category separately. The logits vector output by the classifier is:

[0119]

[0120] In the formula, z cLet p represent the model's original prediction for the c-th class, where C = 6 is the total number of classes. To convert logits into probabilities, a sigmoid activation function is applied to each component, resulting in a predicted probability vector. c p represents the probability that a sample belongs to class c. c Calculate independently, ∑ is not required c p c =1.

[0121] (2) Visual output of recognition results.

[0122] To comprehensively present the results of structural apparent damage detection, the system employs a visualization output scheme that combines image and probabilistic dual views. For example... Figure 2 As shown, the system interface displays the damage image to be detected on the left, while the right side presents the predicted probability distribution of six types of damage in the form of a bar chart, with each bar corresponding to the confidence level of the corresponding category.

[0123] Step 2: Based on image damage classification labels, establish a unified semantic segmentation network for multiple types of apparent damage to building structures under complex backgrounds and strong interference. This includes the following steps:

[0124] Step 21: Construct a multi-type scanning structure-aware visual state space module (DS-SAVSS); specifically including:

[0125] The structure-aware vision state space module is designed specifically for vision tasks and features a two-dimensional selective scanning mechanism. Its network architecture consists of... Figure 4 As shown, this module works collaboratively through four paths, effectively extracting semantic information of various types of apparent damage while maintaining multi-directional texture continuity. It is suitable for multi-scene structural damage image recognition in complex backgrounds.

[0126] (1) Create multi-type feature scanning paths.

[0127] First, four basic scanning modes are predefined to adapt to different types of damage features: diagonal serpentine scanning traverses pixels via alternating diagonal paths, suitable for capturing linear continuous structures such as cracks; circular scanning spirals outward from the image center, effectively covering blocky distribution areas such as spalling; dense grid scanning uses a high-coverage zigzag path, suitable for extracting point-like diffusion damage features such as corrosion; and alternating horizontal and vertical scanning serves as a general supplementary path, further enhancing overall coverage. At the implementation level, all scanning paths are pre-calculated and stored as pixel coordinate sequences to improve operational efficiency.

[0128] (2) Perform dynamic path selection.

[0129] To achieve dynamic path selection, a lightweight weight learning module is introduced. This module first performs global average pooling on the input features to obtain an image-level contextual representation. Then, a small fully connected network generates weight distributions for four scanning paths, which are then normalized using softmax. Next, based on the learned weights, the most suitable scanning path is dynamically selected for feature processing. During feature processing, the input features are rearranged and processed appropriately according to the selected path, ensuring that dependencies between features are effectively preserved and strengthened. Finally, the processed features are weighted and fused using a pixel attention mechanism to enhance feature expressiveness and improve the performance of subsequent tasks.

[0130] (3) Calculation optimization.

[0131] In terms of computational optimization, a parameter-sharing strategy is adopted, allowing the state-space models of the four paths to share most of the parameters, retaining only the final projection layer as independent. This strategy effectively improves model performance while strictly controlling the number of additional parameters to within 0.3M.

[0132] The input image is first processed through a gating mechanism, where the input data X in Control signal G is generated via the gating module ctrl This is then combined with the input feature map. Information flow is regulated using control signals.

[0133] G ctrl =σ(W gate ·X in +b gate (7)

[0134] In the formula, X in The input feature map has dimensions (H×W×C). W gate and b gate These are the weight matrix and bias term in the gating mechanism, respectively. G ctrl It is the generated control signal, and the Sigmoid function σ outputs a value between [0,1].

[0135] The RGB structurally damaged image, after patch embedding and position encoding, was used as the sequence input DS-SAVSS block. Four layers of DS-SAVSS blocks were used to process the image sequence.

[0136]

[0137] In the formula, w is the input. P represents the spatial state of the hidden layer, and S represents the skip connections used to initialize the input. z kLet be the specific hidden state at time step k; Q and R are matrices with hidden space dimension G and time dimension D, obtained through selective SS2D scanning; u k This represents the output at time step k.

[0138] DS-SAVSS establishes multi-directional adjacency relationships, allowing z k The hidden state captures more complex topological and texture details, while also enabling the output u k More effectively integrate multi-directional features.

[0139] To effectively combine the initial sequence x with the sequence processed through 2D selective scanning, pixel-oriented attention fusion (PAF) is introduced to enhance DS-SAVSS's ability to capture the shape and texture details of architectural structural damage. After selective scanning, residual connections are applied to fuse information, thereby preserving detail and promoting feature flow.

[0140] Step 22: Create a multi-scale feature fusion module (ICBAM) based on an improved convolutional block attention mechanism; specifically including:

[0141] (1) First, global pooling is performed on the four-level feature maps extracted by DS-SAVSS to extract global information in the channel dimension and generate weight coefficients for each channel, thereby strengthening the key channel features and suppressing redundant information. The calculation expression is as follows:

[0142] F CA =σ(MLP(Global Pooling(F) i ))) (9)

[0143] In the formula, F i The input feature map has dimensions (H×W×C), where H is the height, W is the width, and C is the number of channels; Global Pooling indicates global pooling of the feature map, typically global average pooling or global max pooling; MLP is a multilayer perceptron used to map the globally pooled features to channel attention weights; σ is the sigmoid activation function used to map the output to the range [0,1]; F CA It is the output of channel attention, used to enhance or suppress the features of individual channels.

[0144] (2) Improvement of dynamic fusion mechanism.

[0145] The dynamic weight α is generated through a lightweight subnetwork: first, global average pooling is performed on the input image to obtain image-level feature vectors; then, the vectors are passed through two fully connected layers (with ReLU activation in the middle layer) to output scalar values; finally, the sigmoid function constrains the output to the range [0,1]. In implementation, the two MLP branches employ an asymmetric design: MLP_coarse uses fewer hidden layer neurons (e.g., 1 / 4 of the input channels) to reduce computation, while MLP_fine maintains a higher dimensionality to preserve detailed information.

[0146] (3) Establish a spatial attention mechanism.

[0147] The features fused by channel attention are used as input. Spatial saliency information is extracted through convolutional operations, and then spatial weight distribution is further modeled using convolutional layers. Finally, the spatial attention map is generated by normalization using the sigmoid function, thus enhancing key regions. This process is represented as follows:

[0148] F SA =Conv2D(σ(Concat(F) CA ,F CA ))) (10)

[0149] In the formula, F CA F represents the output feature map of the channel attention mechanism; Concat concatenates the feature maps along the channel dimension; Conv2D is a convolution operation used to generate a spatial attention map, σ is the Sigmoid activation function used to normalize the output value to the range [0,1]; SA It is the output of spatial attention, used to weight spatial positions.

[0150] (4) Output calculation.

[0151] Multi-scale features weighted by spatial attention are fused and mapped through a fully connected layer to generate the final prediction output:

[0152]

[0153] In the formula, F i This is the feature map at the i-th scale; there are usually multiple scales to extract features of different sizes; ⊙ represents element-wise multiplication, used to apply spatial attention to each feature map; F SA is the output of the spatial attention module, used to weight the spatial location of the feature map; W is the weight matrix of the final fully connected layer (MLP), used to output the prediction; F final This is the final output feature map.

[0154] Steps two and three: Feature processing and prediction segmentation head construction based on parallel branches and multi-scale deformable convolutions (MS-DCNN). Specifically, this includes:

[0155] (1) Create a gating mechanism to enable dynamic features for each spatial location and channel, enhancing the model’s ability to capture details. Embed a bottleneck convolution (BottConv) with low-rank approximation to map the matrix from high-dimensional space to low-dimensional space and significantly reduce computational complexity.

[0156] Gating mechanism:

[0157] X gate =σ(W gate X norm +b gate (12)

[0158] In the formula, σ is the Sigmoid activation function, and W gate and b gate These are the parameters of the gating mechanism.

[0159] (2) Based on the lightweight gated bottleneck convolution module, the network is expanded into a parallel multi-branch multi-scale structure, combining three types of branches—dilated convolution, deformable convolution, and bottleneck convolution (corresponding to large-scale, medium-scale, and small-scale branches, respectively)—to perform multi-scale enhancement for dynamic feature extraction. Specifically, the small-scale branch uses 3×3 convolution kernels to focus on fine-grained textures such as cracks, the medium-scale branch uses 5×5 convolution kernels to extract local morphological features such as spalling, and the large-scale branch uses 7×7 convolution kernels to perceive the overall corrosion distribution and global structure. All branches maintain a lightweight design, and the computational load is further controlled through grouped convolution and channel compression, thereby improving the ability to represent different scales and multiple types of damage features while maintaining efficiency.

[0160] (3) A gated weight generator is designed to dynamically fuse multi-scale features. First, global average pooling is performed on the outputs of each branch to compress spatial features into channel descriptors. Then, three sets of weight values ​​are learned through a fully connected layer and normalized using the Sigmoid function. Finally, multi-branch features are fused through weighted summation, enabling the network to adaptively adjust the contribution ratio of each branch according to the damage type of the input image. The lightweight parallel branch multi-scale deformable convolutional module network architecture is as follows: Figure 5 As shown.

[0161] Step 24: The robustness of the model to imbalanced pixels is improved using a BCE+Dice hybrid loss method, and the damage recognition performance is comprehensively evaluated using six metrics. Specifically, these include:

[0162] (1) A combination of Focal Loss and Tversky loss is used as the objective function to optimize the network, focusing on alleviating the training difficulties caused by the extreme imbalance between foreground and background pixels. The total loss function is shown below:

[0163] L total =α·L Focal +β·L Tversky (13)

[0164] In the formula, L Focal Focal Loss is used to solve the class imbalance problem, and its expression is:

[0165]

[0166] In the formula, p t α represents the model's predicted probability for the target class. t L represents the class weight coefficient, and γ is the focusing parameter for adjusting the weights of hard samples. Tversky Tversky Loss enhances sensitivity to sparse foreground pixels by adjusting the weights of false positives and false negatives.

[0167]

[0168] In the formula, p i Let g be the predicted value of the i-th pixel. i α represents the true label, β controls the weights of false positives and false negatives, respectively, and ∈ is a smoothing constant.

[0169] (2) The performance of the designed damage identification model was evaluated using six metrics: precision (P), recall (R), F1 score, and accuracy. Optimal Dataset Scale (ODS), Optimal Image Scale (OIS), and Mean Intersection over Union (mIoU) are used. ODS measures the model's fitness on datasets of different scales using a fixed threshold m, while OIS evaluates fitness across image scales using an optimal threshold n. The calculation methods are as follows:

[0170]

[0171] mIoU is used to measure the average ratio of the intersection to the union of the true labels and the predicted results. It is calculated as follows:

[0172]

[0173] In the formula, N is the number of categories; t represents the true label, l represents the predicted value, and p tl This represents the count of pixels classified as l but belonging to t.

[0174] like Figure 3As shown, the constructed unified segmentation model for multiple types of apparent damage to building structures under complex backgrounds and strong interference consists of three core modules: First, the multi-type scanning structure perception visual state space module (DS-SAVSS) enhances the topological perception and semantic extraction capabilities of damages such as cracks and spalling through multi-path dynamic scanning and feature fusion; second, the multi-scale feature fusion module (ICBAM) based on an improved convolutional block attention mechanism fuses channels and spatial attention to achieve an effective combination of macroscopic structural information and microscopic details; finally, the lightweight parallel branch multi-scale deformable convolutional module (MS-DCNN) adaptively captures damage features at different scales through gating mechanisms and multi-branch design. Figure 3 In this context, MLP stands for Multilayer Perceptron, Conv for Convolutional Layer, Block for Convolutional Block, Norm for Normalization, and F1, F2, F3, F4, etc., represent the output channels of the image after DS-SAVSS processing.

[0175] Step 3: Integrate apparent damage classification labels and pixel-level semantic segmentation results to construct a damage quantification method based on image recognition, creating a large-scale visual-linguistic multimodal model for image description. This includes the following steps:

[0176] Step 31: Image damage type identification and segmentation result post-processing; specifically including:

[0177] (1) Image binarization processing.

[0178] In the image processing stage, the system first performs strict binarization on the input mask image, identifying pure black areas as the background and uniformly converting all other pixels into white damaged areas to ensure the accuracy of subsequent analysis. Simultaneously, morphological closing operations are used to eliminate minor noise, preserving the original shape characteristics of the damaged areas, and the original structural image is sized and normalized to ensure strict alignment with the mask image.

[0179] Assume the input is a binary damage mask:

[0180]

[0181] In the formula, B(x,y)=1 indicates that pixel (x,y) belongs to the damaged area.

[0182] (2) Damage area identification.

[0183] After entering the damage identification stage, connected component analysis technology is used to identify all possible damage areas, automatically filtering out noise interference with an area smaller than 20 pixels, and focusing on analyzing the main damage area with the largest area. Through algorithms such as skeletonization and central axis transformation, the system can accurately calculate the geometric parameters such as the length and width of the damage area, converting these pixel-level measurements into actual physical dimensions (pixel conversion ratio set at 0.02mm / px), providing reliable data support for structural safety assessment.

[0184] [1] Connected component extraction and noise filtering.

[0185] Connectivity extraction and noise filtering utilize 8-neighborhood connectivity to decompose B into several connected components.

[0186]

[0187] Pixel area of ​​each connected component:

[0188]

[0189] Small noise is filtered out based on an area threshold τ = 20px:

[0190] C valid ={C k |A(C k )≥τ} (21)

[0191] Select the main damaged connected component (largest area):

[0192]

[0193] [2] Skeleton and central axis extraction.

[0194] Define the main domain C * boundary Euclidean distance transformation

[0195]

[0196] The skeleton set is given by the maximal inscribed circle constraint:

[0197]

[0198] In the formula, Let p be a closed disk with center p and radius r.

[0199] [3] Calculation of geometric parameters.

[0200] Sort the skeleton S topologically as a polyline point sequence. Calculate the damage length (based on the longest geodetic path or main path):

[0201]

[0202] The local width of the skeleton point is given by twice the radius of the inscribed circle:

[0203]

[0204] Commonly used statistics:

[0205]

[0206] (3) Results visualization.

[0207] In terms of result visualization, such as Figure 6 , Figure 7 As shown, the system uses a semi-transparent green highlighting method to overlay the identified damaged areas onto the original structural image. Then, the identification number and area measurement value are marked at the center of the damaged area, with these markings distinguished by different colors.

[0208] (4) Table generation.

[0209] As shown in Tables 1 and 2, key detection data are presented in tabular form, including damage type (provided by the damage identification model), number, and precise area. All report files are saved according to a unified naming convention, maintaining a correspondence with the original and labeled images for easy subsequent retrieval and management.

[0210] Table 1 Summary of Crack Damage Parameter Identification Results for Concrete Structures

[0211] Damage type serial number Crack length / mm Maximum crack width / mm crack 1 8.16 0.20

[0212] Table 2 Summary of Spalling Damage Parameter Identification Results for Concrete Structures

[0213] Damage type serial number <![CDATA[Spalling area / mm 2 > Peeling 1 16.5868

[0214] Step 32: Create a large-scale vision-language multimodal model and generate a detection report. This includes:

[0215] A visual-linguistic multimodal model for structural damage recognition is constructed, and its overall architecture is as follows: Figure 8 As shown. This model mainly consists of the following modules:

[0216] First, in the visual encoder part, image feature encoding is achieved based on the improved "Unified Segmentation Network for Multiple Types of Apparent Damage to Building Structures under Complex Backgrounds and Strong Interference" from step two, extracting discriminative visual representations. In the language encoder part, the Llama2-7B architecture is used as the core of the language model to encode the text input.

[0217] The cross-modal attention module embeds modal type labels into the encoding results of the two modalities, enabling effective cross-modal feature fusion using a self-attention mechanism within the Transformer framework. In the fused sequence, two enhancement mechanisms are introduced: first, a structural multi-type damage lexical mask enhancement mechanism to strengthen the semantic representation of key damage concepts; and second, a structural multi-type damage concept embedding mechanism to inject domain prior knowledge into the model, improving semantic understanding of civil engineering scenarios.

[0218] Finally, the dynamic context building module integrates multimodal information and generates descriptive text step by step through an autoregressive generation loop. After output post-processing, the final structural damage detection report is generated.

[0219] (1) Language basic model.

[0220] The core of the language model is built on the Llama2-7B architecture. First, the input text is converted into a token sequence by a token segmenter and mapped to the corresponding word vector representation. These word vectors then enter a processing flow consisting of 32 Transformer blocks. Each block sequentially performs RMSnorm pre-normalization, multi-head attention computation with RoPE positional encoding, and feedforward network transformation using the SwiGLU activation function.

[0221] A pre-normalization structure is adopted, moving the normalization layer before the computation operation, and normalizing only the root mean square of the input features, thus eliminating the process of calculating the mean. The calculation method is as follows:

[0222]

[0223] In the formula, γ is a learnable scaling parameter, and ∈ represents a local minimum to prevent division by zero. By using RMSNorm and pre-normalization, the computational cost of the model is reduced by approximately 10%-20%, making deep models easier to converge.

[0224] Using RoPE, position information is incorporated into the token's vector representation via a rotation matrix. RoPE natively supports relative positional relationships, eliminating the need for additional calculations of relative positional deviations. For position m and token vector x, the encoded vector is:

[0225] RoPE(x,m)=x·e imθ (28)

[0226] In the formula, θ is a predefined rotation angle, and i is the imaginary unit.

[0227] The model training process consists of two stages: unsupervised pre-training and fine-tuning on an instruction dataset. In the first stage, the language model is pre-trained on a massive unlabeled corpus to learn grammatical rules and common-sense knowledge. An autoregressive language model loss is typically used, aiming to predict the next word given preceding words. The loss function takes the following form:

[0228]

[0229] In the second stage, the pre-trained model undergoes supervised fine-tuning and reinforcement learning based on human feedback to teach the model specific answer formats and knowledge expression norms, enabling the model to output answers that meet human requirements and values.

[0230] (2) Visual Base Model. An improved version is used as the visual encoder, such as... Figure 9 As shown.

[0231] This visual encoder maintains the overall architecture of the previous design, with the main improvement being the introduction of a multi-level feature fusion mechanism during feature extraction. Specifically, the dynamically upsampled features are fused in two paths: the F1 and F2 branches are fused to generate a 256×256 resolution feature map to retain rich detail information; the F3 and F4 branches are fused to output a 64×64 resolution feature map to capture a wide range of contextual features. Finally, the two types of fused features at different scales are serialized and concatenated to form a unified visual representation.

[0232] (3) Cross-modal mapping module.

[0233] The cross-modal mapping layer is a key module connecting the visual encoder and the language model. Its module architecture is as follows: Figure 10 As shown.

[0234] The cross-modal mapping layer is responsible for mapping high-dimensional image features to the text semantic space, enabling visual information to be effectively understood and reasoned about by large language models. The specific process is as follows:

[0235] First, the image feature sequence V = [v1, v2, ..., v] output by the visual encoder is processed. m The text feature sequence L = [l1, l2, ..., l] output by the language encoder. n Add modality type embeddings separately to distinguish different modal sources:

[0236] V' = V + E vis L' = L + E lang (30)

[0237] In the formula, E vis and E langLet m and n represent the learnable type embedding vectors for the visual and linguistic modalities, respectively, where m and n are the lengths of the visual and linguistic sequences, respectively.

[0238] Next, the embedded visual sequence V' is concatenated with the language sequence L' to form a cross-modal joint input sequence S:

[0239] S = [V'; L'] = [v1', ..., v m ',l1',…,l n '] (31)

[0240] The concatenated sequence is input into a standard Transformer block, which utilizes its self-attention mechanism to achieve deep fusion of visual and linguistic features. The computation process of the self-attention mechanism is as follows:

[0241]

[0242] In the formula, Q, K, and V represent the query, key, and value matrices, respectively, obtained from the input sequence S through a linear transformation; d k Let be the dimension of the key vector. Used to scale the dot product result.

[0243] Through iterative processing using multi-layered Transformer blocks, the mapping from visual features to the linguistic semantic space is achieved, providing semantically aligned cross-modal representations for subsequent generation stages.

[0244] Step 4: Based on the trained visual-language multimodal large model, integrate image recognition results to intelligently generate a structural damage detection report with complete elements such as images, text, tables, and formulas. This includes the following steps:

[0245] (1) Image and model preprocessing.

[0246] By using structural damage classification and segmentation models and their results preprocessing, three types of building structure appearance images (original structural image, damage segmentation map, and damage location map) are finally obtained, along with corresponding classification results and damage information summary tables. These three types of building structure appearance images are then input into a pre-trained visual-linguistic multimodal large-scale model.

[0247] (2) Visual feature extraction and encoding.

[0248] The system first encodes the image features using a visual encoder, transforming pixel-level visual information into high-dimensional feature vectors. In this step, the model segments the image into multiple small patches and extracts local features. Then, a self-attention mechanism captures global contextual relationships, ultimately generating feature vectors that comprehensively represent the image's semantics. These features not only include damage type information (such as cracks, spalling, or corrosion) but also implicitly contain the geometric characteristics of the damage (such as crack width and orientation) and its severity. Finally, modality type tags are embedded and the data is serialized.

[0249] (3) Knowledge base matching and damage assessment.

[0250] Subsequently, the system intelligently matches these visual feature vectors with a pre-built building structure knowledge base. The knowledge base stores professional knowledge in the field of structural engineering, including criteria for judging various types of damage, standards for classifying safety levels, and maintenance specifications. By calculating cosine similarity, the system can quickly retrieve the professional knowledge item that best matches the current visual features, triggering the corresponding evaluation rules. For example, when a concrete crack width of 0.6mm is detected, the system automatically associates it with relevant clauses in the "Standard for Reliability Appraisal of Civil Buildings" GB50292-2015 to determine its safety level and extract the corresponding maintenance process requirements.

[0251] (4) Natural language report generation and optimization.

[0252] After knowledge matching is completed, the system invokes a language model to generate natural language. It receives structured data from visual features and a knowledge base, and generates a natural language description conforming to professional standards using engineering prompt word templates. During generation, the model comprehensively considers the visual features of the damage (such as crack morphology), quantitative parameters (such as width), and relevant regulatory clauses, outputting a complete paragraph containing a damage description, safety assessment, and repair recommendations. To ensure professionalism, the system also employs a reinforcement learning optimization mechanism, continuously adjusting the generated content through feedback to avoid misuse of technical terminology. Finally, combined with a damage information summary table, a final inspection report is generated. The structural damage inspection report generation process is as follows: Figure 11 As shown. The structural damage detection report is as follows. Figure 12 As shown, this report is only an illustration, and the specific content in the figure is not limited to any particular purpose and is only used as an example.

[0253] The contents of a partial test report are shown below:

[0254] 1. Damage Description

[0255] The reinforcing steel is severely corroded, with rust on the surface and exposed bars, some of which are quite long. Spalling 1: Area 45.3080 mm. 2The structure is vertical and located on the concrete surface. There are water stains on the concrete surface, possibly due to water seepage. Cracks and misalignment are present on the concrete surface, with some areas showing both.

[0256] 2. Quantitative assessment of damage

[0257] The following provides damage assessment results for concrete and masonry structures, respectively. The specific structure types are unknown; please refer to the information based on your actual situation. The repair recommendations listed in the tables are based on established standards; see section 4, Repair Suggestions, for specific repair recommendations.

[0258] Damage assessment of concrete structures:

[0259] Table 3. Assessment of Spalling in Concrete Structures

[0260] Damage type serial number <![CDATA[Spalling area / mm 2 > Usability rating Repair suggestions Peeling 1 45.3080 bs No measures may be taken

[0261] Note: Damage is classified as BS or CS level if it exists, and the area is greater than 0.1m². 2 The damage was assessed as CS level.

[0262] 3. Structural safety assessment

[0263] The concrete structure is severely damaged, with rust and water stains. Professional assessment and repair are recommended.

[0264] 4. Maintenance Recommendations

[0265] [General Recommendations] Rust: It is recommended to remove rust and then apply a rust-preventive coating. Concrete surfaces: Repair is recommended to prevent further structural damage.

[0266] [Urgent Recommendation] The most urgent repair needs are for rust, as it can cause structural damage and safety issues.

[0267] [Regular Maintenance Recommendations] It is recommended to conduct regular structural health monitoring, promptly address any new damage, and prevent further damage from spreading.

[0268] [Detailed Repair Recommendations] The following specific repair methods are provided for different types of damage:

[0269] Detailed Repair Recommendations for Corroded Steel Bars

[0270] Further determination of the type of steel reinforcement corrosion is needed:

[0271] For mild corrosion (only surface rust, no obvious cross-sectional loss), first use a wire brush and angle grinder with a rust-removing abrasive wheel to thoroughly remove the rust until the metal luster is exposed. Then use compressed air to blow away the dust, and then apply two coats of epoxy steel reinforcement anti-rust paint with a total dry film thickness of not less than 80μm.

[0272] For moderate corrosion (obvious rust layer, cross-sectional loss rate of 5%-10%), the mechanical properties of the steel bars need to be evaluated after rust removal. If they still meet the design requirements, cement-based penetrating crystalline waterproof coating can be applied to the surface of the steel bars, and then the protective layer can be repaired with a higher strength grade of shrinkage-compensating concrete. The repair thickness should not be less than the original protective layer thickness, and it needs to be vibrated and compacted.

[0273] For severe corrosion (section loss rate exceeding 10% or severe pitting corrosion), the severely corroded steel bars must be cut off, and new steel bars of the same specification should be used for lap splicing or welding connection (lap length not less than 35 times the diameter of the steel bar, and ensure full weld during welding). After the connection is completed, rust prevention and protective layer repair should be carried out according to the treatment process for moderate corrosion.

[0274] After all repairs are completed, the repaired area must be watered and cured for no less than 14 days. The surrounding concrete should be checked for cracks, and if necessary, sealed to prevent moisture from seeping in again and aggravating corrosion.

[0275] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for identifying and generating reports of structural multi-type apparent damage based on a multimodal large model.

[0276] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for generating reports on structural multi-type apparent damage based on a multimodal large model.

[0277] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0278] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0279] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0280] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0281] The above provides a detailed description of the method for identifying and generating reports of structural multi-type apparent damage based on a multimodal large model proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for structural multi-type apparent damage identification and detection report generation based on a multi-modal large model, characterized in that, The method comprises: Step one: facing the multi-type apparent damage image of the building structure, a lightweight convolutional neural network classification model based on deep separable convolution and inverted residual structure is established; Step two: based on the image damage classification label, a unified semantic segmentation network of building structure multi-type apparent damage under complex background strong interference is established; Step three: fusion of apparent damage classification label and pixel-level semantic segmentation result, construction of damage quantification method based on image recognition, creation of visual-linguistic multimodal large model of image description; Step four: based on the trained visual-linguistic multimodal large model, the image recognition result is fused to realize the intelligent generation of structure damage detection report with multiple elements.

2. The method of claim 1, wherein, The step one specifically comprises: Step one: construct a multi-label dataset and XML annotation parsing system; Step two: establish a lightweight network based on deep separable convolution, and use a binary cross entropy loss function with Logits for optimization; specifically including: constructing a lightweight network architecture, decomposing the standard convolution operation into two parts of deep convolution and pointwise convolution; In the deep convolution stage, each input channel uses a convolution kernel for convolution operation alone, without cross-channel mixing; In the formula, M and N are the size of the convolution kernel, and k represents the input channel index; Pointwise convolution uses 1*1 convolution to perform linear combination in the channel dimension: In the formula, K is the number of input channels, and l represents the output channel index; An inverted residual structure is introduced: In the formula, x is an input feature map, and the dimension is HxWxC in , For the expansion layer, the number of channels is expanded from C in to txC in , where t is an expansion factor, DWConv 3×3 is a deep convolution, a 3x3 convolution is independently performed on each channel to extract spatial features, and the number of channels remains unchanged, The projection layer compresses the number of channels back to the target output channel number C out , and x+... is a residual connection. If the input x and the output y have the same dimension, element-wise addition is performed to realize residual learning. Step three: construct a multi-label inference system based on Sigmoid, and realize the visual output of the result.

3. The method of claim 1, wherein, The step two specifically comprises: Step two: construct a multi-type scanning structure perception visual state space module DS-SAVSS; the module works through four paths, including: diagonal snake scanning through alternating diagonal line paths to traverse pixels, ring scanning expanding outward from the image center, dense grid scanning using high coverage zigzag path, and horizontal and vertical alternating scanning as a general supplementary path; Step two: create a multi-scale feature fusion module ICBAM based on an improved convolution block attention mechanism; Step two: construct a lightweight parallel branch multi-scale deformable convolution module MS-DCNN based on parallel branch and multi-scale deformable convolution feature processing and prediction segmentation head; Step two: use a BCE+Dice hybrid loss to improve the robustness of the model to unbalanced pixels, and use 6 indicators to comprehensively evaluate the damage identification performance.

4. The method of claim 3, wherein, In step two, global pooling operation is performed on the four level feature maps extracted by DS-SAVSS to extract global information in the channel dimension and generate weight coefficients for each channel, so that the model strengthens key channel features and suppresses redundant information; the calculation expression is as follows: F CA = σ(MLP(Global Pooling(F i ))) (9) In the formula, F i is an input feature map with a dimension of (HxWxC), where H is the height, W is the width, and C is the number of channels; GlobalPooling represents global pooling of the feature map; MLP is a multi-layer perceptron for mapping the globally pooled features to channel attention weights; σ is a Sigmoid activation function for mapping the output to the range [0, 1]; F CA is the output of channel attention, used to enhance or suppress the features of each channel.

5. The method of claim 3, wherein, In step two, a gating mechanism is created to enable dynamic features for each spatial position and channel, enhancing the model's ability to capture details, and embedding a bottleneck convolution BottConv with low rank approximation in it, which maps the matrix from a high-dimensional space to a low-dimensional space; Gating mechanism: X gate = σ(W gate X norm + b gate ) (12) where σ is a Sigmoid activation function, W gate and b gate are parameters of the gating mechanism.

6. The method of claim 1, wherein, The step three specifically comprises: Step three one: image damage type recognition and segmentation result post-processing; specifically including: (1) image binarization processing, (2) damage area recognition, (3) result visualization and (4) table generation; Step three two: creating a visual-linguistic multimodal large model and generating a detection report.

7. The method of claim 6, wherein, In step three two, image feature encoding is implemented in the visual encoder part to extract discriminative visual representations; in the language encoder part, Llama2-7B architecture is used as the core of the language model to encode the text input; The cross-modal attention module embeds the encoding results of the two modalities with modal type labels respectively, so as to realize effective cross-modal feature fusion by using the self-attention mechanism under the Transformer framework; in the fused sequence, two enhancement mechanisms are introduced: one is the structural multi-type damage word mask enhancement mechanism, which is used to strengthen the semantic expression of key damage concepts; the other is the structural multi-type damage concept embedding mechanism, which injects domain prior knowledge into the model to improve the semantic understanding of the civil engineering scene; Finally, the multi-modal information is integrated by the dynamic context construction module, and the description text is generated step by step through the self-recurrent generation cycle, and then the output post-processing step is performed to generate the final structural damage detection report.

8. The method of claim 1, wherein, In step four, three types of building structure appearance images and corresponding classification results and damage information summary tables are finally obtained by the structural damage classification model and the segmentation model and the result preprocessing; the three types of building structure appearance images are input into the trained visual-linguistic multimodal large model; then, the image is feature encoded by the visual encoder to convert the pixel-level visual information into a high-dimensional feature vector; Subsequently, these visual feature vectors are intelligently matched with the pre-built building structure knowledge base; after completing the knowledge matching, the large model is called for natural language generation.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1-8.

10. A computer readable storage medium for storing computer instructions, characterized in that, The computer instructions are executed by the processor to realize the steps of the method of any one of claims 1-8.

Images