A building risk assessment method and system based on a multi-modal hybrid expert network

Abstract

This invention belongs to the field of building safety assessment and artificial intelligence, and discloses a building risk assessment method and system based on a multimodal hybrid expert network. The method includes: constructing a multimodal dataset containing images of wall defects and user-generated verbal descriptions; designing a network architecture integrating visual, textual, and fused expert pools, dynamically scheduling expert sub-networks for specific defects through a gating mechanism; and deeply fusing multimodal features to output risk levels and maintenance recommendations. This invention achieves intelligent conversion from non-professional descriptions to professional assessments, utilizes expert activation paths to provide interpretable analysis, and effectively solves the problems of high professional thresholds, slow response times, and difficulties in utilizing multimodal information in the assessment of old buildings.

Description

Technical Field

[0001] This invention belongs to the field of building safety assessment and artificial intelligence technology, specifically a building risk assessment method and system based on a multimodal hybrid expert network. Background Technology

[0002] Among existing buildings, self-built houses constitute a large proportion. However, most of these houses lack planning permits, professional designs, and final acceptance inspections. Some have even undergone further renovations after completion, potentially posing significant safety hazards and accident risks. These buildings cannot be fully rectified in a short period, and safety inspections are a massive undertaking with limited scope. Furthermore, many self-built house owners lack safety awareness and struggle to correctly assess signs of structural deformation, often relying on their own experience and missing crucial rescue opportunities.

[0003] Furthermore, even if a building was initially planned and designed reasonably and professionally, and passed final acceptance, its function may change due to repeated use by different users throughout its lifespan. Some buildings, due to their special location and commercial value, undergo multiple renovations and alterations, which may not be approved by structural engineers, potentially altering the building's structure. All of these factors can lead to structural safety accidents.

[0004] In existing technologies, building safety risk assessment mainly develops in two directions: one is the traditional method based on statistics and rules, and the other is the machine learning method based on structured data. Traditional methods, such as the Analytic Hierarchy Process (AHP) and Fault Tree Analysis (FTA), heavily rely on expert experience to construct indicator systems, are highly subjective, and are difficult to adapt to the massive and diverse scenarios of old buildings. Although machine learning-based risk assessment methods can utilize the structured attribute data of buildings (such as age, area, and structural type) for modeling, for example, using algorithms such as XGBoost and Random Forest to predict risk levels, they still have significant shortcomings. On the one hand, these methods cannot effectively utilize unstructured data, such as photos of defects taken by residents and descriptive text, which are precisely important sources for early risk detection. On the other hand, existing models are mostly single-modal or simple early / late fusion models, lacking the ability to deeply and adaptively fuse fine-grained visual features of images with complex semantic relationships in text, resulting in weak comprehensive judgment capabilities for multimodal risk clues such as "bulge shape + user description 'recently enlarged'". Moreover, due to the poor model segmentation ability, it is prone to "illusions".

[0005] In recent years, multimodal learning and Mixture of Experts (MoE) networks have made progress in their respective fields. Large multimodal models (such as CLIP) have demonstrated the potential for jointly understanding images and text, but their general design falls short in accuracy for highly specialized tasks like identifying building defects. Mixture of Experts networks have been successfully applied in large language models, significantly improving model capacity and efficiency by dynamically activating specific parameter subsets through gating mechanisms. However, their application in multimodal scenarios, particularly in vision-language tasks requiring deep integration of domain knowledge (such as building structure and material aging), has not yet been reported. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a method and system for risk assessment of old buildings based on a multimodal hybrid expert network. Its purpose is to fully utilize multimodal information such as wall defect images and natural language descriptions provided by non-professional users; through a domain-specific hybrid expert network, it achieves refined identification of building defects and in-depth risk reasoning. This reduces the professional threshold and cost of risk assessment, enabling rapid and automated preliminary risk screening. It provides interpretable assessment results and clarifies the basis for risk assessment.

[0007] To achieve the above objectives, the present invention provides a building risk assessment method and system based on a multimodal hybrid expert network, characterized by comprising the following steps:

[0008] S1: Multimodal data acquisition and preprocessing:

[0009] Data Collection: Collect the following three types of data: (1) On-site multimodal data: digital images of wall defects in old buildings, and natural language text describing the defects provided by users or inspectors. (2) Structured attribute data: basic information about the building, including building age, structural type (brick-concrete / frame), number of floors, historical maintenance records, etc., which are derived from property archives, the Internet or user descriptions. (3) Domain knowledge text data: text corpus used to train and build background knowledge for the model, including unstructured or semi-structured texts such as the "Standards for Rating the Integrity of Buildings", building disease diagnosis manuals, professional inspection reports, and case descriptions in accident investigation reports.

[0010] Data preprocessing: (1) For image data: size standardization, brightness and contrast equalization, and weak annotation of key defect areas for supervised training of visual experts; (2) For on-site text description: word segmentation, removal of irrelevant stop words, and mapping of colloquial expressions (such as "bulge", "peeling") to standard terminology (such as "plaster layer hollow", "coating peeling"), while transforming standardized terms into various colloquial expressions through manual semantic annotation and adversarial neural network (GAN) to further improve text processing capabilities; (3) For domain knowledge text: sentence segmentation and entity recognition (such as "crack", "steel corrosion", "concrete carbonization") using natural language processing tools to construct a corpus for training text experts; (4) For structured data: field unification, missing value filling and normalization processing.

[0011] S2: Constructing a multimodal hybrid expert network for risk assessment

[0012] Visual expert pool construction: training a group ( Each expert is a pool of neural networks specializing in different visual tasks. Each expert shares a Vision Transformer encoder but has an independent output head tailored to a specific defect type. These mainly include:

[0013] Crack expert: Identifies and quantifies the width, length, and morphology of linear and network cracks.

[0014] Bulging and Hollow Area Expert: Detects bulging areas on walls, assesses their area and height, and determines whether they are hollow areas or solid bulges based on image features.

[0015] Peeling and weathering experts: Identify the area and severity of material degradation phenomena such as coating peeling, mortar powdering, and brick weathering.

[0016] Humidity and Leakage Expert: Identifies water stains, mold, and seepage traces based on color and texture features.

[0017] Text expert pool construction: training a group ( Each expert is a natural language processing model specializing in extracting risk information from text across different dimensions, forming a pool of text experts. Each expert undergoes domain-adaptation fine-tuning based on a pre-trained language model (BERT) and a Generative Adversarial Network (GAN). This primarily includes:

[0018] Defect Entity and Attribute Extraction Expert: Identifies the type of defect mentioned in the text (e.g., “crack”), location (e.g., “bottom of load-bearing beam”), and size (e.g., “approximately 30cm long”).

[0019] Time-based dynamic description expert: Analyzes and describes the time information of defect development (such as "newly emerged", "continuously expanding", "patched three years ago").

[0020] Subjective perception and severity quantification expert: understand and quantify users' subjective descriptions (such as "very serious", "a bit damp", "dull sound when tapped").

[0021] Causality and correlation experts: Identify implicit causal relationships in text (such as "water stains appeared after rain") or concurrent issues (such as "white exudate was found at the crack").

[0022] Multimodal Gated Network Design: A learnable routing network that dynamically determines which visual and textual experts to activate and assigns weights based on the image and text features of the current input instance. Specifically:

[0023] Visual gating networks Input is global image features Output a weight vector ( (Number of visual experts), among which Indicates the first The activation weights are assigned to each visual expert. This is implemented using a simple feedforward network with Softmax.

[0024]

[0025] Text Gating Networks Input is text [CLS] labeled features Output the text expert weight vector :

[0026]

[0027] The gating network selects the Top-K experts with the highest weights, K=3 experts are activated, and the rest are masked, thereby achieving efficient utilization of conditional computation and model capacity.

[0028] Dynamic fusion and decision-making level:

[0029] Expert Feature Aggregation: For experts selected by the gating network, calculate the weighted sum of their outputs to obtain visual aggregated features. and text aggregation features .

[0030]

[0031] Cross-modal deep fusion: and Input a cross-attention fusion module. This module allows visual and textual features to query and complement each other (e.g., using information about a "corner" mentioned in the text to enhance the visual features of the corresponding area in the image), generating a unified multimodal joint representation. .

[0032] Risk prediction: Input a multilayer perceptron classifier and output the final risk level probability (e.g., low risk, medium risk, high risk, emergency risk).

[0033] S3: Model Training and Optimization Strategies

[0034] This step involves end-to-end training of MM-MoE, which requires addressing the issues of expert division of labor and load balancing.

[0035] Phased training:

[0036] Expert pre-training phase: Using the data constructed in S1, each vision expert and text expert is pre-trained independently to achieve excellent performance on their respective subtasks.

[0037] Joint training phase of gating network: expert parameters are frozen, and equal weights are used in the initial stage to encourage exploration. The main training is to train the gating network to learn how to select experts based on the input.

[0038] Overall fine-tuning phase: Unfreeze all parameters at a low learning rate and perform end-to-end fine-tuning to optimize the global objective.

[0039] Specialization and balance loss function:

[0040] Total loss function Designed as follows:

[0041]

[0042] Standard cross-entropy loss for primary task risk classification.

[0043] Load balancing loss. This prevents the gating network from always selecting a few "hot" experts while leaving other experts "idle." This is achieved by calculating the frequency with which each expert is selected in a batch of data and penalizing those with excessively low distribution entropy.

[0044] Expert specialization loss. Encourage each expert to have higher confidence and output variability on their areas of expertise, promoting a more rational division of labor among experts.

[0045] S4: Risk Assessment Implementation and Report Generation

[0046] Input and Inference: Users upload photos and descriptive text of defects in the target building. The system performs forward propagation through the MM-MoE network trained in S2 to obtain the risk level.

[0047] Structured report generation: The system integrates model output, interpretability analysis results, and basic building attributes to automatically generate a risk assessment report containing the following:

[0048] Risk level: Clearly indicated.

[0049] Defect details: List the identified defect types, locations, and quantitative indicators.

[0050] Risk Basis: Explain the reasons for the assessment by combining visual evidence and textual description.

[0051] Recommendations: Based on the risk level, we provide a tiered recommendation from "regular observation" to "immediate evacuation and contacting professional agencies".

[0052] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0053] It achieves deep, task-oriented multimodal fusion: through the design of a domain-specific expert pool, the model can not only process images and text simultaneously, but also perform fine-grained alignment and deep reasoning on the visual features and professional semantic descriptions of building defects, which goes beyond simple feature splicing or shallow attention mechanisms.

[0054] Combining powerful modeling capabilities with efficient runtime performance: The MoE architecture allows for a very large total number of model parameters (with numerous experts), but for a single specific input, only a small number of relevant experts need to be activated for computation, achieving "large model capacity and low inference cost," making it suitable for practical deployment.

[0055] The interpretability and credibility of the assessment are significantly improved: the gating mechanism provides a natural path for decision interpretation. Users can clearly know which "specialized sub-modules" (experts) derived the assessment results based on what information (high-weight features), thus enhancing the credibility and acceptability of the results.

[0056] It has good scalability and adaptability: When new types of building defects or risk dimensions emerge, the system's capabilities can be expanded by incrementally adding new experts to the expert pool and training them, without having to redesign or train the entire model, allowing the system to evolve continuously.

[0057] Effectively lowers the professional threshold: Image interpretation and text analysis tasks that require deep professional knowledge are encapsulated in a pre-trained expert network, enabling ordinary users to obtain a preliminary risk assessment that is close to professional level by taking pictures and making simple descriptions, providing a feasible technical solution for large-scale, routine safety screening of old buildings. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall architecture of the building risk assessment system based on a multimodal hybrid expert network according to the present invention;

[0059] Figure 2 This is a detailed structural diagram of the multimodal hybrid expert network (MoE) in an embodiment of the present invention;

[0060] Figure 3 This is a flowchart illustrating the risk assessment method provided in the embodiments of the present invention.

[0061] Figure 4 This is a visual explanation of the model expert selection and risk assessment in an application example. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1: A building risk assessment system based on a multimodal hybrid expert network.

[0064] like Figure 1 As shown, this embodiment provides a complete system implementation method, and the system mainly includes the following modules:

[0065] 1. Data Acquisition and Preprocessing Module

[0066] 1.1 Data Acquisition Unit

[0067] Image Acquisition: Using the camera of a mobile terminal (smartphone, tablet), capture images of defects in building walls, beams, columns, floors, etc. Image format should be JPEG or PNG, with a resolution of at least 1920×1080 pixels.

[0068] Text description collection: Collect user descriptions of defects through text input boxes. Example description: "There is a crack on the east wall of the living room near the ceiling. It started last month and is now about 20 centimeters long. It leaks water when it rains."

[0069] Structured attribute collection: Obtain basic building information through form filling or by connecting to the property management system, including: building age, structural type (brick-concrete, frame, steel structure, etc.), number of floors, building area, and historical maintenance records (time, location, maintenance content).

[0070] Domain knowledge acquisition: Structured and semi-structured professional knowledge texts were collected from normative documents such as the "Technical Standard for Building Structure Testing" (GB / T 50344-2019) and the "Standard for Identification of Dangerous Buildings" (JGJ 125-2016), as well as the investigation reports of building safety accidents over the years, to establish a knowledge corpus.

[0071] 1.2 Data Preprocessing Unit

[0072] Image preprocessing:

[0073] Size uniformity: All input images are scaled to 512×512 pixels. However, important details in the images, such as cracks and damp spots, are easily lost during normal scaling. To ensure the accuracy of subsequent expert model analysis, special details are further processed through high-resolution refocusing of regions of interest (ROI), integration of frequency domain analysis and wavelet transform, and key region protection based on attention mechanisms.

[0074] Color space conversion: Convert the image from RGB space to LAB space and perform histogram equalization on the L channel (luminance) to reduce the impact of uneven lighting.

[0075] Weak annotation generation: The pre-trained general object detection model YOLOv5 is used to initially locate potential defect regions (cracks, bulges, peeling, etc.) in the image and generate bounding boxes as weak supervision signals for subsequent training of vision experts.

[0076] Text preprocessing:

[0077] Word segmentation and part-of-speech tagging: The HanLP tool was used to perform word segmentation, part-of-speech tagging, and dependency parsing on user descriptions.

[0078] Spoken language standardization: Establish a "user spoken language - professional terminology" mapping dictionary. For example, map "bulging" to "plaster layer hollowing" and "peeling" to "coating peeling". For new expressions not covered by the dictionary, use a fine-tuned BERT model to perform semantic similarity matching and recommend the closest professional term.

[0079] Adversarial Data Augmentation: To improve model robustness, a Conditional Generative Adversarial Network (cGAN) is used. Based on standardized professional descriptive text, diverse spoken descriptions conforming to user language habits are generated to augment the training data.

[0080] Structured data preprocessing:

[0081] Missing value handling: For numeric fields (such as building age), fill with the average value of buildings of the same type; for categorical fields (such as structure type), fill with the mode.

[0082] Normalization: Z-score standardization is applied to continuous numerical fields.

[0083] 2. Multimodal Hybrid Expert Network (MM-MoE) Module

[0084] This module is the core computing unit, and its detailed structure is as follows: Figure 2 As shown.

[0085] 2.1 Implementation of the Visual Expert Pool

[0086] The visual expert pool contains 5 experts ( All experts share a ResNet-50 pre-trained on ImageNet as the base feature extractor (Backbone). Each expert feeds their task-specific head network after the feature map (2048×16×16) output by the Backbone.

[0087] expert (Crack Expert): The head is a semantic segmentation network based on U-Net, which outputs crack masks and then performs post-processing to calculate crack length (converted from pixel count) and average width.

[0088] expert (Bump and Hollow Expert): The head contains a Region Proposal Network (RPN) and a classification regression head to detect bump regions and output their bounding boxes, areas, and "hollow probability" (obtained by analyzing the differences between the texture features of the bump region and the surrounding region).

[0089] expert (Peeling and Weathering Expert): Using a multi-label classification head, identify material degradation types such as "coating peeling", "mortar powdering" and "brick weathering", and regress their severity scores (0-1).

[0090] expert (Humidity and Leakage Expert): Using a pre-trained DeepLabv3+ model, fine-tuned specifically on a dataset labeled with water stains and mold, outputting a semantic segmentation map of humidity traces.

[0091] expert (Structural Deformation Expert): When a standard design image exists, the input is a multi-angle photo of the same part or a comparison image with the standard design drawing. Using a stereo vision algorithm, the flatness deviation or tilt angle of the wall surface is calculated.

[0092] 2.2 Implementation of the Text Expert Pool

[0093] The text expert pool contains 4 experts ( All experts share a BERT-base model pre-trained on Chinese Wikipedia and architecture-related texts as the base encoder. Each expert is fed a different task head after a contextual representation (768-dimensional vector) labeled with [CLS].

[0094] expert 1 (Defect Entity and Attribute Extraction Expert): Fine-tunes for sequence labeling tasks (such as BIEO labeling) to identify defect entities (such as “crack”), location entities (such as “east wall”), and size entities (such as “20 cm”) in text.

[0095] expert (Time-based dynamic description expert): As a text classifier fine-tuning tool, it categorizes descriptions into categories such as "newly emerging", "continuously expanding", "stable", and "historical repair".

[0096] expert (Subjective Perception and Severity Measurement Expert): Regressing to the task head, text containing subjective descriptions is mapped to a severity score of 0-1. For example, "somewhat damp" corresponds to 0.3, and "very severe" corresponds to 0.9.

[0097] expert (Causality and Relationship Expert): As a relation classifier fine-tuning tool, it determines whether there is a causal relationship (such as "rain" - "cause" - "seepage") or a concurrent relationship in a sentence.

[0098] 2.3 Implementation of Multimodal Gated Networks

[0099] Visual gating networks First, global average pooling is performed on the feature map output by ResNet-50 to obtain a 2048-dimensional global feature vector. The weights are then passed through a two-layer feedforward neural network (dimensions change from 2048 to 512 to 5), and finally through a softmax layer to output a 5-dimensional weight vector. .

[0100] Text Gating Networks : Directly use the 768-dimensional vector labeled [CLS] from the BERT output As input, the vector passes through a two-layer feedforward network (768→256→4) and a Softmax layer, outputting a 4-dimensional weight vector. .

[0101] Sparse routing: configuration For each input sample, only select The top two vision experts with the highest weighting and The top two text experts with the highest weights participate in the forward computation. The outputs of unselected experts are set to zero, and their corresponding gradients are masked during backpropagation.

[0102] 2.4 Dynamic Integration and Implementation at the Decision-Making Level

[0103] Expert Feature Aggregation:

[0104]

[0105]

[0106] in, It is a 512-dimensional vector. It is a 256-dimensional vector.

[0107] 2. Cross-modal attention fusion module:

[0108] Will and The query, key, and value vectors are obtained by linear projection.

[0109] A single-layer multi-head attention mechanism (4 heads) is adopted to calculate the attention from text to vision, that is, to use text features as a query to retrieve relevant information in visual features.

[0110] Attention-weighted visual features are concatenated with original text features, and then passed through a fully connected layer to output a 768-dimensional fused feature vector. .

[0111] Risk prediction layer:

[0112] Input a three-layer MLP classifier (768→256→64→4) and output the probability distribution of four risk levels (low, medium, high, and urgent) through Softmax.

[0113] 3. Model Training Module

[0114] 3.1 Training Data

[0115] The dataset was trained using a self-built "OBR-MM Dataset" (Multimodal Dataset of Risk in Older Buildings). This dataset contains 50,000 pairs of data (images, text descriptions, structured attributes, and risk level labels). Risk levels were independently labeled by three structural engineers according to the "Standards for Identification of Dangerous Buildings," with disagreements resolved through consultation.

[0116] 3.2 Training Process and Loss Function

[0117] A three-stage training strategy is adopted and implemented in the PyTorch framework.

[0118] Phase 1: Expert Pre-training

[0119] Freeze the shared Backbone parameters and train the task heads of each expert using their respective subtask data.

[0120] Visual experts use segmentation or detection loss (Smooth L1 Loss), while text experts use cross-entropy loss or mean squared error loss.

[0121] Phase Two: Joint Training of Gated Networks

[0122] Freeze all expert parameters (Backbone and Task Header). Train only two gating networks. and .

[0123] The loss function is the total loss. However, at this stage and Set to a smaller value (e.g., 0.1), mainly depending on (Risk classification cross-entropy loss) driven learning.

[0124] Phase 3: Overall End-to-End Fine-Tuning

[0125] Unfreeze all parameters and fine-tune them at a low learning rate (e.g., 1e-5).

[0126] Using the full total loss function:

[0127]

[0128] (Load Balancing Loss): The auxiliary loss proposed in Switch Transformer is adopted. For the gating weights of each expert in a batch (i.e. the probability of being selected), the coefficient of variation of the square of its distribution (CV²) is calculated as a penalty term to encourage uniform use.

[0129] (Expert Specialization Loss): For each expert, calculate the variance of their output across different sample types (classified by major defects). Experts are encouraged to have large output variance (high confidence) in their areas of expertise and small output variance in their areas of weakness. Specifically, for each expert, calculate the ratio of their output vector's variance between sample types to its variance within each group, and maximize the negative logarithm of this ratio.

[0130] Optimizer: The AdamW optimizer is used with an initial learning rate of 3e-4 and a cosine annealing strategy.

[0131] A mutual information minimization constraint is introduced as a specialization loss. By forcibly reducing the cosine similarity between the output feature vectors of different experts, it is ensured that each expert only focuses on its specific defect dimension (e.g., expert A focuses on texture degradation, expert B focuses on geometric deformation), thus avoiding the MoE model from degenerating into a simple ensemble model with single weights.

[0132] At the same time, based on the characteristics of the expert network itself, in the future, other expert types can be added or existing experts can be deleted according to their own needs.

[0133] Example 2: Application Case of Risk Assessment for Old Buildings Based on Multimodal Hybrid Expert Network

[0134] This embodiment uses a specific case of "wall bulging" as an example, such as Figure 4 As shown, this paper elaborates on how the multimodal hybrid expert network proposed in this invention operates in real-world scenarios and generates interpretable risk assessment results.

[0135] 2.1 Case Background and Input

[0136] Residents of an older residential community discovered an anomaly on the wall of their living room and uploaded the information using the mobile application provided by this invention.

[0137] Image data: A photo of a bulge on a wall, clearly showing a noticeable bulge on the wall surface.

[0138] Text description: "There is a bulge in this load-bearing wall in the living room. It seems to have gotten bigger in the last month, and it sounds hollow when you tap it."

[0139] 2.2 Reasoning Process of Multimodal Hybrid Expert Networks

[0140] After receiving the above multimodal input, the system initiates the inference process:

[0141] Step S2.2.1: Feature Extraction and Gated Routing

[0142] First, the visual encoder and the text encoder process the image and text respectively to generate high-level semantic features.

[0143] A visual gating network analyzes image features and determines that the input is highly correlated with visual tasks such as bulge analysis, crack detection, and spatial relationships. Therefore, it generates expert weight vectors, where: bulge analysis expert: 0.6; crack detection expert: 0.20; spatial relationship expert: 0.18; and other experts (such as humidity expert, peeling expert, etc.) have weights close to 0.

[0144] Based on the preset Top-2 sparse activation strategy, the system selects to activate the bulge analysis expert and the crack detection expert.

[0145] A text-gated network analyzes text features to identify risky words, temporal dynamics, and location information within the description, generating a weight vector: Risk word extraction expert: 0.40; Temporal dynamics expert: 0.35; Location description expert: 0.25

[0146] The system activates the three text experts mentioned above.

[0147] Step S2.2.2: Expert Collaborative Analysis and Feature Aggregation

[0148] Activated experts work in parallel, outputting professional analysis results:

[0149] Bulging analysis expert: After performing fine-grained analysis on the input image, the output quantitative result is: "A hollow plaster layer was detected. The bulging area is approximately circular, with a diameter of about 18 centimeters, and there are fine radial textures at the edges."

[0150] Crack detection expert: Scanning image, output: "No through-type main crack found."

[0151] Risk word extraction expert: Analyze the text and extract key risk words: "bulge" and "hollow sound".

[0152] Time Dynamics Expert: Analyzes time information in the text and outputs the label: "Progress (based on: 'increased in size in the last month')".

[0153] Location Description Expert: Parses the location information in the text and outputs the label: "Load-bearing wall (critical structural component)".

[0154] The outputs of the aforementioned experts are encoded as feature vectors and then weighted and aggregated according to their weights to form visual aggregated features. and text aggregation features .

[0155] Step S2.2.3: Cross-modal attention fusion and risk assessment

[0156] The aggregated bimodal features are then fed into the cross-attention fusion module. In this module:

[0157] Textual features serve as the "query," guiding the system to focus on the visual regions in the image most relevant to "load-bearing walls," "bulges," and "edge textures."

[0158] Visual features serve as both "keys" and "values," providing concrete visual evidence for textual descriptions.

[0159] This process achieves deep semantic alignment between "progress" and "newly emerging radial patterns," "hollow sound" and "typical acoustic characteristics of hollowness," and "load-bearing wall" and "central location of the wall," generating a unified joint representation. .

[0160] Finally, the joint representation is input into the risk classifier, and the output probability distribution is: high risk (87% probability), medium risk (10%), and low risk (3%). Based on this, the system determines the risk level to be high risk.

[0161] 2.3 Generation of Interpretable Risk Assessment Reports

[0162] The system not only outputs risk levels, but also automatically generates structured, interpretable reports based on gated routing records and expert input.

[0163] Building Risk Assessment Report

[0164] Assessment conclusion: High risk

[0165] Main basis:

[0166] Core Defect Confirmation: The bulge analysis experts confirmed the presence of large-diameter hollow plaster layers, and the edge texture indicated that the hollow area might be expanding.

[0167] Risk dynamic attributes: Time dynamics experts identified users who explicitly described "growing larger in the last month" as indicating that the defect is progressing and the unstable state exacerbates the risk.

[0168] Location criticality: The location description experts determined that the defect was located in a load-bearing wall, and failure in this part would have a significant impact on the overall structural safety.

[0169] Consistent multimodal evidence: The user's description of "hollow sound when tapped" closely matches the "hollow drum" feature determined by visual analysis, enhancing the credibility of the judgment.

[0170] Exclusionary analysis: Crack detection experts did not find any cracks in the main structure, ruling out the extreme risk of immediate collapse caused by cracks. However, the risk of detachment of the hollow areas themselves remains high.

[0171] Recommendations for handling:

[0172] Emergency measures (within 7 days): Immediately invite professionals to conduct an on-site inspection and assess the safety of the load-bearing wall; set up temporary warning signs in the bulging area to prevent impact or hanging of heavy objects.

[0173] Monitoring Recommendation: Users can take photos and measure the size of the bulge weekly to record the changes through this application.

[0174] Anticipated repairs: Based on the results of the professional survey, partial removal, reinforcement with wire mesh, or overall repair may be required.

[0175] 2.4 Technical advantages disclosed in this embodiment

[0176] This case demonstrates that the present invention has the following advantages over traditional methods:

[0177] Specialized division of labor and efficient computing: The gating network acts like a "dispatcher", precisely calling on the few most relevant experts (5 in this case) rather than all experts to perform calculations, which greatly improves efficiency while ensuring the depth of analysis.

[0178] Deep semantic fusion: The system does not simply concatenate image classification results with text keywords, but allows abstract concepts in the text (such as "progressiveness") to actively query and associate with specific evidence in the image (such as new radial patterns), thus achieving true understanding and reasoning.

[0179] Transparent and credible decision-making: Every step of the risk assessment is verifiable (which expert it comes from, how much weight it carries, what the output is), forming a clear chain of evidence that meets the stringent interpretability requirements of high-risk application scenarios.

[0180] User-friendly language: The system can understand and process non-professional colloquialisms such as "bulge" and "hollow sound," and transform them into professional risk parameters through built-in terminology mapping and expert models, greatly reducing the barrier to entry for users.

[0181] Although some embodiments of the present invention have been shown and described, those skilled in the art will recognize that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and all such solutions fall within the scope of protection of the present invention.

Claims

1. A method for building risk assessment based on multi-modal hybrid expert network, characterized in that, Includes the following steps: S1: Multimodal data acquisition and standardization processing: Collect on-site multimodal data of the building to be evaluated, including images of wall defects, user-generated colloquial descriptions, and building structural attribute information; perform geometric calibration and preprocessing on the images, and map the colloquial descriptions into standard building defect terms; S2: Multimodal feature extraction: Extract global features of the image through a shared visual encoder and extract semantic features of the text through a shared text encoder; S3: Expert dynamic scheduling based on gating mechanism: Construct a visual expert pool containing multiple visual subtask experts and a text expert pool containing multiple text semantic experts; Utilize visual gating network and text gating network to dynamically calculate the activation weight of each expert based on the current input features, and only activate the K experts with the highest weights for calculation based on the Top-K strategy, while the remaining experts are masked. S4: Expert Feature Aggregation and Cross-Modal Alignment: Weighted aggregation of the output features of activated experts is performed to obtain visual aggregated features and text aggregated features; The visual aggregation features and text aggregation features are associated and aligned using a cross-attention mechanism to generate a unified multimodal joint representation vector; S5: Risk Assessment and Report Generation: Input the multimodal joint representation vector into the risk assessment classifier, output the building risk level, and combine the activation weights of the experts with the expert output results to generate an interpretable assessment report containing risk basis.

2. The method of claim 1, wherein, The preprocessing described in step S1 specifically includes: Convert the acquired RGB image to the LAB color space; Histogram equalization is performed on the L channel (luminance channel) in the LAB color space to eliminate the effects of uneven lighting. The image details and textures are refocused using a region of interest (ROI) analysis algorithm to generate a standard input image for subsequent visual expert analysis. A mapping library of "user spoken language - professional terminology" is constructed, and a semantic similarity algorithm based on contrastive learning is used to convert colloquial expressions into standard building disease entity words in real time.

3. The method according to claim 1, characterized in that, The visual subtask expert mentioned in step S3 uses ResNet as the basic feature extractor and connects to different task heads, specifically including at least the following two: Crack Quantitative Analysis Expert: Its task head uses a U-Net-based semantic segmentation network to output crack masks and calculate the pixel-level length and width of cracks; Bump and Hollow Area Analysis Expert: Its task head includes a Region Proposal Network (RPN) to locate the bounding box of the bump region and output the hollow area probability by combining texture features; Humidity and Leakage Analysis Expert: Its task head uses the DeepLab series network for pixel-level semantic segmentation of water stains and mold areas.

4. The method according to claim 1, characterized in that, The text semantic expert mentioned in step S3 uses BERT as the basic encoder and connects to different task heads, specifically including at least the following two: Defect Entity and Attribute Extraction Expert: Employs Sequence Labeling to identify defective entities, location entities, and size entities in text; Subjective perception quantification expert: Using a regression task head, the user's subjective description text is mapped to a severity score between 0 and 1; Causality and Correlation Expert: Employs a relation classification task header to determine the implicit causal logic or concurrent relationships of diseases in text descriptions.

5. The method according to claim 1, characterized in that, The gating mechanism described in step S3 is specifically implemented as follows: Visual gating networks receive global features of images Text-gated networks receive text [CLS] tag features. ; The input features are mapped to logits vectors of dimension equal to the number of experts through a fully connected layer, and the normalized weight vector is calculated using the Softmax function; Set the sparse activation parameter K ( For each input sample, only the output values ​​of the top K experts with the largest weights are retained for subsequent calculations, and the output values ​​of the remaining experts are set to zero.

6. The method according to claim 1, characterized in that, The specific calculation process of the cross-attention mechanism described in step S4 is as follows: Map text aggregation features to query vectors (Query, ), mapping visual aggregation features to key vectors (Key, ) and value vector (Value, ); calculate and The dot product is used to obtain the attention weight map of text semantics to the image region; Using the attention weight map Weighted summation is performed to map key disease features described in the text (such as "radial texture" and "load-bearing wall location") to image features, generating a fused context vector.

7. The method according to claim 1, characterized in that, The multimodal hybrid expert network introduces an expert specialization loss function during the training phase. This loss function is used to constrain the mutual information of the output vectors of different experts, and its mathematical expression includes: Calculate the cosine similarity between the feature vectors output by any two different experts for the same input sample; The squared mean of the cosine similarity is calculated and added as a penalty term to the total loss function; By minimizing the penalty term, the feature vectors of different experts are forced to tend to be orthogonal in the feature space, ensuring that each expert focuses on a unique disease feature dimension and avoiding model degradation.

8. The method according to claim 1, characterized in that, The multimodal hybrid expert network also incorporates a load balancing loss function during the training phase. The loss function calculates the coefficient of variation based on the cumulative activation probability of each expert within a training batch. The coefficient of variation is used as a penalty term to prevent the gated network from over-relying on a few specific experts in the early stages of training, ensuring that all expert networks can obtain sufficient gradient updates.

9. The method according to claim 1, characterized in that, The logic for generating the interpretability assessment report described in step S5 is as follows: Record the Top-K expert indexes and weights output by the gating network during the inference process; If the bulge and hollow bulge analysis expert is activated and its weight exceeds the preset threshold, its output data of "hollow bulge area" and "texture difference" will be extracted and written into the report. If a time-based dynamic description expert is activated, extract their identified "progressive" or "stable" tags and include them in the report's basis. Finally, based on the risk level, a tiered response recommendation is automatically generated, ranging from "regular observation" to "immediate evacuation".

10. A building risk assessment system based on a multimodal hybrid expert network, characterized in that, include: The mobile data acquisition module is used to acquire images of building wall defects, user natural language description text, and building structure attribute data; The data preprocessing module is used to perform image enhancement, weakly supervised annotation generation, text segmentation, and spoken term mapping. The cloud-based expert network processing module is equipped with a multimodal hybrid expert network as described in any one of claims 1 to 9, used to perform feature extraction, expert gating scheduling, cross-modal fusion reasoning, and risk level determination. The report output module is used to receive the cloud processing results and push a structured assessment report containing risk level, defect details and handling suggestions to the user terminal.

Images