Historic building protection decision-making production method based on multi-modal fusion

Abstract

The application discloses a multi-modal fusion historical style building protection decision production method, comprising: obtaining historical documents, real scene photos and structure monitoring data of a target building, performing time sequence calibration and orthographic projection to solve a structured image, performing semantic pre-calibration on the structured orthographic image through a deep light decomposition network, obtaining a damage area mask, and generating a binary damage heat map, dividing a damage time sequence space according to the binary damage heat map and a calibration time sequence, extracting a work method term by using a large language model, obtaining a representation vector through prototype contrast learning, inputting the representation vector into a KAN network model in a heterogeneous parallel mode to perform explicit mapping, and generating a structured decision scheme through soft voting integration. The method utilizes adaptive modal attention, physical information embedding and interpretable mapping to output high-precision and interpretable building protection decisions under small sample conditions.

Description

Technical Field

[0001] This invention relates to the field of historical buildings, and more particularly to a multimodal integrated method for producing decisions on the protection of historical buildings. Background Technology

[0002] As material carriers of regional cultural genes and the continuation of urban historical context, the protection and revitalization of historical buildings is an important requirement in the fields of cultural heritage protection and urban renewal. With the acceleration of urbanization and the increasing demand for cultural inheritance, digital technology has gradually become an emerging driving force for architectural style protection decisions. Among them, OCR recognition of historical documents and semantic extraction technology of contextual knowledge have been widely used to sort out architectural construction techniques and cultural context. By digitally analyzing local chronicles and construction archives, the historical evolution of buildings can be restored. Combined with drone aerial photography and street scene semantic segmentation technology, urban style areas can be divided and architectural forms and damaged areas can be identified. This promotes the development of historical building protection in a multi-dimensional synergy of culture, form and technology, showing broad application prospects in scenarios such as historical block protection planning, industrial heritage revitalization and ancient building restoration. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal fusion method for making decisions on the protection of historical buildings.

[0004] To achieve the above objectives, the present invention is implemented according to the following technical solution: This invention provides a multimodal fusion method for producing decisions on the protection of historical buildings, including: Historical documents, real-scene photos, and structural monitoring data of the target building are acquired. The structural monitoring data is time-series calibrated, and structured orthophotos are calculated based on real-scene photos from different perspectives according to the calibration time series. The structured orthophoto image is semantically pre-labeled using a depth illumination decomposition network to obtain a damage region mask and generate a binarized damage heatmap. Based on the aforementioned binary damage heatmap and the time-series calibration, the damage time-series space is divided, and construction terms within the damage time-series space are extracted from historical documents using a pre-set large language model to obtain construction method entries. Based on the construction method terms, historical features are determined, and based on these historical features, small-sample learning is performed on the structural mechanical properties of the damaged area through prototype comparison learning to obtain a representation vector. The representation vector is input into a heterogeneous parallel configuration of the Kolmogorov-Arnold network model, and the protection requirement is explicitly mapped according to the structural features and the binary damage heat map. Based on the output of the explicit mapping, soft voting integration is performed to generate a structured decision scheme that includes protection strategies, repair strategies, activation strategies, and construction method suggestions.

[0005] Furthermore, the method for obtaining the calibration timing includes: The system acquires the construction date, structural type, historical repair records, and architectural style records of the target building, and performs OCR recognition, simplified / traditional character conversion, and domain dictionary matching to obtain architectural attribute entities that match historical documents. It also performs noise reduction, exposure correction, and lens distortion elimination on real-scene photos from different perspectives to obtain a standardized image matrix. The real-scene photos include drone aerial images, ground close-up photography, and laser scanning point cloud data. Outlier removal, missing value imputation, and sampling rate normalization are performed on the structural monitoring data to obtain a unified time-series monitoring matrix. Change point detection is performed based on the unified time-series monitoring matrix to obtain the structural state time. The set of key calibration time is determined based on the structural state time. The structural state time includes the time of structural response change, the start and end time of steady state interval, and the time of extreme value of environmental load. The structural monitoring data includes time-series signals collected by strain gauges, inclinometers, accelerometers, and temperature and humidity sensors. The acquisition timestamps of the standardized image matrix are matched with the nearest neighbor set of key calibration times to obtain the structural state time labels corresponding to each viewpoint image, thus obtaining the calibration time sequence. Based on the structural state time labels of the calibration time sequence, the structured orthophoto is calculated from the real scene photos.

[0006] Furthermore, the method for obtaining the structured orthophoto includes: Based on the structural state time label, the structural response value of the corresponding time in the unified time series monitoring matrix is ​​used as the initial constraint condition. Based on the standardized image matrix, feature point detection and descriptor extraction are performed by feature point detection algorithm to obtain feature point matching pairs. The detection algorithm includes SIFT algorithm and ORB algorithm. Based on the feature point matching pairs, a random sampling consensus algorithm is used to eliminate mismatches and obtain an accurate matching point set. Based on the accurate matching point set and initial constraints, the camera intrinsic and extrinsic parameters and 3D point coordinates are iteratively optimized by bundle adjustment to obtain a sparse reconstructed point cloud. Based on the sparse reconstructed point cloud, multi-view stereo matching and depth map fusion are performed to obtain a dense 3D point cloud model. Based on the dense 3D point cloud model, a horizontal reference plane is selected. The 3D coordinates of each point cloud are mapped to the 2D image coordinates according to the horizontal reference plane, and digital elevation information is generated. The digital elevation information is used as pixel values ​​to obtain structured orthophotos and metadata.

[0007] Furthermore, the method for obtaining the mask of the damaged area includes: The pixels of the structured orthophoto are normalized, and the normalized structured orthophoto is input into a deep illumination decomposition network. Based on the deep illumination decomposition network, the image is decomposed into illumination components and reflection components through an encoder and decoder structure. The image is reconstructed by element-wise multiplication of the illumination components and reflection components to obtain a de-interference image. Based on the reflection component of the de-interference image, the building material texture and damage features are extracted through the convolutional layer of the U-Net model to obtain pre-labeled semantics. A pixel-level damage probability map is obtained based on the pre-labeled semantics, material texture and damage features. Damage candidate regions are marked based on the damage probability map. Based on the damage candidate region, isolated noise points are removed by morphological opening operation, the closed boundary of the denoised candidate region is extracted, a damage region mask is generated, the damage region mask is aligned with the metadata of the structured orthophoto image and associated with the structural state time label to obtain a binarized damage heatmap, the gray value of the binarized damage heatmap is positively correlated with the damage probability.

[0008] Furthermore, the method for obtaining the aforementioned construction method terminology includes: Based on the target building, the damage temporal space is divided according to the pixel coordinates of the calibration time sequence and the binarized damage heat map. Building attribute entities are extracted from historical documents. An index library is constructed based on the building terminology, time information and spatial location of the building attribute entities. Based on the damage temporal space, the repair records, construction method records and craftsmanship texts of the corresponding components in the historical documents are retrieved through the index library to obtain a subset of documents. The damage temporal space includes the spatial range of the damage area, the structural state time label, the damage probability level and the damage type. The damage type, spatial location, temporal information, and literature subset of the damage time-series space are input into a pre-trained large language model to obtain structured architectural terms, which include component names, construction method entries, material types, and process parameters.

[0009] Furthermore, the method for obtaining the representation vector includes: Historical features are determined using structural architectural terminology based on construction method entries. The spatial location, temporal information, and historical features vector of the damaged area are used as feature indexes. A small support set and query set are constructed based on the feature indexes. Each damage type in the support set contains only 3 to 5 labeled samples. The labeled samples include the structural mechanical features of the damaged area, the visual features of the binarized damage heat map, and the spliced ​​vector of historical features. The structural mechanical features include the stress peak, strain gradient, and vibration frequency of the structural monitoring data. The visual features are material texture and damage features. The material texture includes crack orientation, and the damage features include the damage area. The query set includes samples of the damaged area to be predicted. The structure of the query set is consistent with that of the support set. The representation vector is calculated based on the support set and the query set. The formula for calculating the representation vector is as follows: ; in For the representation vector, For modal indexing, For modal sets, For text modal features, For image modal features, For sensor modal characteristics, For feature mapping function, For modality The projection matrix, For modality eigenvectors, For modality eigenvectors; ; in For query vector, For transpose, The hyperbolic tangent activation function is used. Here is the weight matrix of the gated network. is the bias vector of the gated network. The square of the Mahalanobis distance. The prototype center vector is the modality. The category of historical features The mean, The best matching prototype category index is determined based on the matching of construction method terms; The feature mapping function and the weight matrix of the gated network are optimized through backpropagation until the loss converges to a preset threshold, thereby obtaining a trained prototype contrastive learning model. Based on the sample features of the query set, the prototype contrastive learning model is used to learn the structural mechanical features through small samples to obtain the representation vector of the structural mechanical properties of the damaged area.

[0010] Furthermore, the method for obtaining the output result includes: The B-spline basis functions of the Kolmogorov Arnold network model are: ; in For the first B-spline activation function for 3D input, For B-spline basis function index, The total number of B-spline basis functions. For the first Vidi Learnable weights of each basis function For the first A B-spline basis function in The value at that position is calculated according to the Cox-de Boor recursive formula. For B-spline basis functions with the same index and one order lower, For the next index, a B-spline basis function of reduced order, the node vector of the B-spline basis function... Evenly distributed in the input range, Let B be the order of the spline; The representation vectors are input in parallel to the Kolmogorov-Arnold network models of each branch. Linear projection and B-spline activation are applied to each dimension of the representation vectors, and the explicit mapping value of the protection requirement is calculated to obtain the output result of the explicit mapping. The formula for calculating the explicit mapping value is as follows: ; in To protect the explicit mapping value of demand, Let be the dimension number of the representation vector. For the first A projection weight vector in each dimension. For the first Bias in each dimension; The heterogeneous branches include a structural safety requirement branch, a damage repair requirement branch, and a context protection requirement branch. The input feature dimension of the structural safety requirement branch is obtained through the structural feature dimension, the input feature dimension of the damage repair requirement branch is obtained through the binarized damage heatmap dimension, and the input feature dimension of the context protection requirement branch is obtained through the historical landscape feature dimension.

[0011] Furthermore, the method for the structured decision-making scheme includes: The monitoring frequency of each monitoring data in the structural monitoring data is obtained, and the monitoring frequency is normalized according to different time periods. Based on the normalized monitoring frequency, the distribution change of the explicit mapping output results of each branch in different time periods is statistically analyzed. Based on the mean, variance and extreme values ​​of the distribution change of each branch, soft voting integration is performed to obtain the voting combination of each branch in the same time period. Based on the voting combination, the period with the highest number of votes in the structural safety requirement branch is taken as the starting period of the protection strategy, and the period with the highest number of votes in the damage repair requirement branch is taken as the starting period of the repair strategy. The period with the highest number of votes in the context protection requirement branch is taken as the starting period of the activation strategy. Construction method suggestions are generated based on the construction method entries corresponding to the starting time of each strategy, and a structured decision-making scheme is obtained. The construction method suggestions include applicable conditions and key points of construction.

[0012] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects: This invention reduces the heterogeneous modal differences between the special protection needs shown by multi-dimensional data such as the historical features of buildings and the actual monitoring data by dynamically weighting and fusing text, images and sensor data in a unified feature space. Based on the Kolmogorov-Arnold network model with physical information embedding, structural mechanical features are introduced into the explicit mapping of the B-spline activation function, so that the protection demand prediction meets the repair capabilities of traditional construction methods, providing an accurate data foundation for comprehensive protection decisions. Through soft voting integration and adaptive statistical scaling of heterogeneous parallel networks, uncertainty is quantified according to the temporal distribution changes to obtain the optimal time period under different decision objectives, avoid execution conflicts, and improve the engineering feasibility and building revitalization capabilities of building protection decisions. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the steps of the multimodal fusion method for producing decisions on the protection of historical buildings in this invention. Detailed Implementation

[0014] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0015] Reference Figure 1 As shown, this invention provides a multimodal fusion method for producing decisions on the protection of historical buildings, including: Historical documents, real-scene photos, and structural monitoring data of the target building are acquired. The structural monitoring data is time-series calibrated, and structured orthophotos are calculated based on real-scene photos from different perspectives according to the calibration time series. In the actual assessment, the target building was a brick-and-stone-wood structure Confucian temple. Historical documents, real-world photographs, and structural monitoring data were acquired. The historical documents included a scanned copy of 12 volumes of the "Records of the Repair of the Confucian Temple," which underwent OCR recognition, noise reduction, and distortion correction to obtain 82,000 characters of structured text, preserved in PDF+TXT dual format. 120 aerial photographs were taken around the target building, and 60 ground-based photographs of the walls and beams were taken, resulting in 180 distortion-corrected images. The structural monitoring data included three strain gauges installed on the walls and two tiltmeters installed on the beams, continuously collecting time-series data with UTC timestamps. Outlier removal, missing value interpolation, and sampling rate normalization were performed on the structural monitoring data to obtain a unified time-series monitoring matrix. Three strain exceeding the threshold peak were removed using the 3σ criterion, and five missing data points were filled using linear interpolation. The sampling rate was normalized to 0.2Hz to obtain the unified time-series monitoring matrix. Based on this unified time-series monitoring matrix, a change-point detection algorithm identified three key calibration moments: 3.15. At 08:42, the wall crack propagation abruptly occurred, with a strain increment of 0.028 mm / m. At 14:17 on March 20, the stress peaked due to rain, with a measured strain of 0.035 mm / m. At 09:30 on March 10, the target building marked by the cultural relic inspection was in a steady-state range. The aerial timestamp of March 10 was matched with the key moments using nearest neighbor matching. A structural state label was bound to each image, and 12,456 feature points were extracted from the standardized image matrix using the SIFT algorithm. Based on the feature point matching pairs, mismatches were eliminated using the Random Sampling Consensus Algorithm (RANSAC), retaining 8,923 accurate matching points. Based on the accurate matching point set, the camera intrinsic and extrinsic parameters were optimized using Pix4Dmapper bundle adjustment to generate a dense point cloud of 300 points / m. 2 The target building's platform level is used as the horizontal reference plane. Based on the horizontal reference plane, the three-dimensional coordinates of each point cloud are mapped to the two-dimensional image coordinates. The projection generates an orthophoto with a resolution of 0.01m / pixel, along with metadata including WGS-84 coordinates and structural time tags, thus obtaining a structured orthophoto and metadata. The structured orthophoto image is semantically pre-labeled using a depth illumination decomposition network to obtain a damage region mask and generate a binarized damage heatmap. In the actual evaluation, the pixels of the structured orthophoto were normalized to [0,1], and the size was uniformly 2048×2048. Based on the DCE-Net deep illumination decomposition network, the normalized structured orthophoto was decomposed into illumination components and reflection components. Rainy season shadows and wall reflection interference were removed to obtain the reflection component feature map without illumination interference, which was used as the interference image. Based on the reflection component of the de-interference image, the building material texture and damage features were extracted through the convolutional layer of the U-Net model. The U-Net model has 3 input channels and output channels for damage and background, and the convolution kernel is 3× 3. With dropout set to 0.2, the probability of cracked areas is 0.85 and weathered areas is 0.72. Areas with a damage probability > 0.7 are marked as damage candidate areas. Based on the damage candidate areas, isolated noise points are removed by morphological opening operation. The closed boundaries of the denoised candidate areas are extracted to generate a damage area mask. The morphological processing of one damage candidate area is a 3×3 structure point opening operation to remove 3 isolated noise points. The binarized damage mask is aligned with the metadata of the structured orthophoto image and associated with the 3.15 abrupt change time label to obtain a binarized damage heat map. Based on the aforementioned binary damage heatmap and the time-series calibration, the damage time-series space is divided, and construction terms within the damage time-series space are extracted from historical documents using a pre-set large language model to obtain construction method entries. In the actual assessment, the damage time-series space was divided based on the target building according to the calibrated time sequence and pixel coordinates of the binary damage heatmap. Building attribute entities were extracted based on historical documents. The physical coordinates of the heatmap pixels of one region are (X120.3456, Y31.2345), with a damage area of ​​2.3m². 2 The damage time sequence space is "Space: Southwest eaves wall, Time: 3.15, Type: Brick and stone crack, Level: High risk". Based on the NER entity index library, the text fragment of the southwest wall repair record is matched. The instruction extraction is to input the cultural heritage field fine-tuning ERNIE 3.5, Prompt, to extract the components, construction methods, materials, and process parameters of the repair of cracks in the blue brick wall, and output structured entries. The structured entries are deduplicated and the terminology is unified to generate standard construction method entries. The components are blue brick masonry, the construction method is brick and stone grouting and filling, the material is glutinous rice mortar, including glutinous rice juice: lime: fine sand = 1:3:5, and the process is construction temperature 15-25℃, layered grouting, curing for 7 days, to obtain the construction method entries. Based on the construction method terms, historical features are determined, and based on these historical features, small-sample learning is performed on the structural mechanical properties of the damaged area through prototype comparison learning to obtain a representation vector. In the actual assessment, historical features were determined using structured architectural terminology based on construction method entries. The feature vector F = [0.4 (brick and stone structure), 0.3 (construction age), 0.2 (compressive strength of blue bricks 25MPa), 0.1 (grouting method)], with a dimension of 4. The spatial location, temporal information, and historical feature vector of the damaged area were used as feature indices. Based on the feature indices, a small sample support set and query set were constructed. The support set consisted of two categories: brick and stone cracks and wall weathering, with 4 labeled samples in each category. Each sample contained structural features. The crack orientation, area, and landscape feature vectors were analyzed under stress of 12.5 MPa, strain of 0.03 mm / m, and frequency of 12 Hz. The query set consisted of one sample of the southwest wall to be predicted, with the format consistent with the support set. 32-dimensional embedding vectors were extracted from text, image, and structural modalities. Representation vectors were calculated based on the support and query sets. The average of the fused features for each support set category was calculated to obtain the category prototype vector. The feature mapping function and the weight matrix of the gating network were optimized through backpropagation until the loss converged to a preset threshold 1e. -4 The prototype contrastive learning model is trained and then trained. Based on the sample features of the query set, the prototype contrastive learning model is used to learn the structural mechanical features by small sample, and the representation vector of the structural mechanical properties of the damaged area is obtained. A 64-dimensional fusion representation vector is obtained by reasoning based on the query set. The representation vector is input into a heterogeneous parallel configuration of the Kolmogorov-Arnold network model, and the protection requirement is explicitly mapped according to the structural features and the binary damage heat map. In the actual assessment, the input features of the structural safety requirement branch are the first 16 dimensions of the representation vector, with a spline order of 3, and the mapping target is the structural safety requirement degree. The input features of the damage repair requirement branch are the heat map features of the middle 16 dimensions of the representation vector, with a spline order of 2, and the mapping target is the damage repair requirement degree. The input features of the context protection requirement branch are the landscape features of the last 32 dimensions of the representation vector, with a spline order of 4, and the mapping target is the context protection requirement degree. Based on the Kolmogorov-Arnold network model, the B-spline basis functions and B-spline activations are calculated. The single-branch B-splines of forward propagation and explicit mapping are obtained through the Cox-de-Boor recursive formula, and the explicit mapping value of the protection requirement degree is calculated to obtain the output results of the explicit mapping. The weights of the explicit mapping values ​​are 0.4, 0.4, and 0.2, respectively, and the bias is zero. In the output results, the explicit mapping value of structural safety is 0.78, that of the damage repair branch is 0.82, and that of the context protection branch is 0.65. Based on the output of the explicit mapping, soft voting integration is performed to generate a structured decision scheme that includes protection strategies, repair strategies, activation strategies, and construction method suggestions.

[0016] In the actual assessment, the monitoring frequency of each monitoring data point in the structural monitoring data was obtained. The monitoring frequency was normalized according to different time periods. The original frequency of strain gauges was daily, with a normalized value of 1; the original frequency of inclinometers was weekly, with a normalized value of 0.75; and the original frequency of temperature and humidity was monthly, with a normalized value of 0.5. The mean, variance, and extreme values ​​of the branch outputs for each time period were calculated using a sliding window and used as voting weights. Soft voting integration was then performed to obtain the voting combinations of each branch within the same time period. The highest vote period for structural safety was the crack abrupt change with a strain increment of 0.031 mm / m at 14:27 on April 2nd; the highest vote period for damage repair was the damage propagation moment caused by high humidity before the plum rain season on May 28th; and the highest vote period for context protection was the monitoring steady state marked by manual inspection on June 10th. The highest vote period of each branch was used as the starting time period for the protection strategy. Construction method suggestions were generated based on the construction method entries corresponding to the starting time periods of each strategy, resulting in a structural decision-making scheme. The main defects are weathering of the blue bricks on the southwest eaves wall, with a depth of 2-3mm, vertical cracks 3-5mm wide and 1.2m long, and slight deflection of the front eaves wooden beams, with a deformation of 8mm. The protection strategy is to designate the southwest wall as the core area starting from April 2nd, increase strain gauge monitoring to twice a day, and mark structural safety information weekly. The repair strategy is to carry out composite reinforcement of the southwest wall with glutinous rice mortar and carbon fiber cloth starting from May 28th, first grouting the cracks, with the ratio adjusted to glutinous rice juice: lime: fine sand. =1:4:6, then reinforce with thick carbon fiber cloth, control the construction temperature at 18-22℃, extend the curing period to 10 days, and grout in 3 layers. The activation strategy is to carry out a static exhibition of construction techniques from June 10th, and keep the flow of people density ≤50 people / time. The applicable conditions in the construction method suggestions are brick and stone buildings, cracks ≤5mm, and humidity ≤60%. The key points of construction are that the cracks must be cleaned with high pressure air before grouting, and special structural adhesive must be used for bonding carbon fiber cloth. Ordinary epoxy resin is strictly prohibited as a substitute.

[0017] In this embodiment, the method for obtaining the calibration timing includes: The system acquires the construction date, structural type, historical repair records, and architectural style records of the target building, and performs OCR recognition, simplified / traditional character conversion, and domain dictionary matching to obtain architectural attribute entities that match historical documents. It also performs noise reduction, exposure correction, and lens distortion elimination on real-scene photos from different perspectives to obtain a standardized image matrix. The real-scene photos include drone aerial images, ground close-up photography, and laser scanning point cloud data. Outlier removal, missing value imputation, and sampling rate normalization are performed on the structural monitoring data to obtain a unified time-series monitoring matrix. Change point detection is performed based on the unified time-series monitoring matrix to obtain the structural state time. The set of key calibration time is determined based on the structural state time. The structural state time includes the time of structural response change, the start and end time of steady state interval, and the time of extreme value of environmental load. The structural monitoring data includes time-series signals collected by strain gauges, inclinometers, accelerometers, and temperature and humidity sensors. The acquisition timestamps of the standardized image matrix are matched with the nearest neighbor set of key calibration times to obtain the structural state time labels corresponding to each viewpoint image, thus obtaining the calibration time sequence. Based on the structural state time labels of the calibration time sequence, the structured orthophoto is calculated from the real scene photos.

[0018] In this embodiment, the method for obtaining the structured orthophoto includes: Based on the structural state time label, the structural response value of the corresponding time in the unified time series monitoring matrix is ​​used as the initial constraint condition. Based on the standardized image matrix, feature point detection and descriptor extraction are performed by feature point detection algorithm to obtain feature point matching pairs. The detection algorithm includes SIFT algorithm and ORB algorithm. Based on the feature point matching pairs, a random sampling consensus algorithm is used to eliminate mismatches and obtain an accurate matching point set. Based on the accurate matching point set and initial constraints, the camera intrinsic and extrinsic parameters and 3D point coordinates are iteratively optimized by bundle adjustment to obtain a sparse reconstructed point cloud. Based on the sparse reconstructed point cloud, multi-view stereo matching and depth map fusion are performed to obtain a dense 3D point cloud model. Based on the dense 3D point cloud model, a horizontal reference plane is selected. The 3D coordinates of each point cloud are mapped to the 2D image coordinates according to the horizontal reference plane, and digital elevation information is generated. The digital elevation information is used as pixel values ​​to obtain structured orthophotos and metadata.

[0019] In this embodiment, the method for obtaining the mask of the damaged area includes: The pixels of the structured orthophoto are normalized, and the normalized structured orthophoto is input into a deep illumination decomposition network. Based on the deep illumination decomposition network, the image is decomposed into illumination components and reflection components through an encoder and decoder structure. The image is reconstructed by element-wise multiplication of the illumination components and reflection components to obtain a de-interference image. Based on the reflection component of the de-interference image, the building material texture and damage features are extracted through the convolutional layer of the U-Net model to obtain pre-labeled semantics. A pixel-level damage probability map is obtained based on the pre-labeled semantics, material texture and damage features. Damage candidate regions are marked based on the damage probability map. Based on the damage candidate region, isolated noise points are removed by morphological opening operation, the closed boundary of the denoised candidate region is extracted, a damage region mask is generated, the damage region mask is aligned with the metadata of the structured orthophoto image and associated with the structural state time label to obtain a binarized damage heatmap, the gray value of the binarized damage heatmap is positively correlated with the damage probability.

[0020] In this embodiment, the method for obtaining the construction method terminology includes: Based on the target building, the damage temporal space is divided according to the pixel coordinates of the calibration time sequence and the binarized damage heat map. Building attribute entities are extracted from historical documents. An index library is constructed based on the building terminology, time information and spatial location of the building attribute entities. Based on the damage temporal space, the repair records, construction method records and craftsmanship texts of the corresponding components in the historical documents are retrieved through the index library to obtain a subset of documents. The damage temporal space includes the spatial range of the damage area, the structural state time label, the damage probability level and the damage type. The damage type, spatial location, temporal information, and literature subset of the damage time-series space are input into a pre-trained large language model to obtain structured architectural terms, which include component names, construction method entries, material types, and process parameters.

[0021] In this embodiment, the method for obtaining the representation vector includes: Historical features are determined using structural architectural terminology based on construction method entries. The spatial location, temporal information, and historical features vector of the damaged area are used as feature indexes. A small support set and query set are constructed based on the feature indexes. Each damage type in the support set contains only 3 to 5 labeled samples. The labeled samples include the structural mechanical features of the damaged area, the visual features of the binarized damage heat map, and the spliced ​​vector of historical features. The structural mechanical features include the stress peak, strain gradient, and vibration frequency of the structural monitoring data. The visual features are material texture and damage features. The material texture includes crack orientation, and the damage features include the damage area. The query set includes samples of the damaged area to be predicted. The structure of the query set is consistent with that of the support set. The representation vector is calculated based on the support set and the query set. The formula for calculating the representation vector is as follows: ; in For the representation vector, For modal indexing, For modal sets, For text modal features, For image modal features, For sensor modal characteristics, For feature mapping function, For modality The projection matrix, For modality eigenvectors, For modality eigenvectors; ; in For query vector, For transpose, The hyperbolic tangent activation function is used. Here is the weight matrix of the gated network. is the bias vector of the gated network. The square of the Mahalanobis distance. The prototype center vector is the modality. The category of historical features The mean, The best matching prototype category index is determined based on the matching of construction method terms; The feature mapping function and the weight matrix of the gated network are optimized through backpropagation until the loss converges to a preset threshold, thereby obtaining a trained prototype contrastive learning model. Based on the sample features of the query set, the prototype contrastive learning model is used to learn the structural mechanical features through small samples to obtain the representation vector of the structural mechanical properties of the damaged area.

[0022] In this embodiment, the method for obtaining the output result includes: The B-spline basis functions of the Kolmogorov Arnold network model are: ; in For the first B-spline activation function for 3D input, For B-spline basis function index, The total number of B-spline basis functions. For the first Vidi Learnable weights of each basis function For the first A B-spline basis function in The value at that position is calculated according to the Cox-de Boor recursive formula. For B-spline basis functions with the same index and one order lower, For the next index, a B-spline basis function of reduced order, the node vector of the B-spline basis function... Evenly distributed in the input range, Let B be the order of the spline; The representation vectors are input in parallel to the Kolmogorov-Arnold network models of each branch. Linear projection and B-spline activation are applied to each dimension of the representation vectors, and the explicit mapping value of the protection requirement is calculated to obtain the output result of the explicit mapping. The formula for calculating the explicit mapping value is as follows: ; in To protect the explicit mapping value of demand, Let be the dimension number of the representation vector. For the first A projection weight vector in each dimension. For the first Bias in each dimension; The heterogeneous branches include a structural safety requirement branch, a damage repair requirement branch, and a context protection requirement branch. The input feature dimension of the structural safety requirement branch is obtained through the structural feature dimension, the input feature dimension of the damage repair requirement branch is obtained through the binarized damage heatmap dimension, and the input feature dimension of the context protection requirement branch is obtained through the historical landscape feature dimension.

[0023] In this embodiment, the method for the structured decision-making scheme includes: The monitoring frequency of each monitoring data in the structural monitoring data is obtained, and the monitoring frequency is normalized according to different time periods. Based on the normalized monitoring frequency, the distribution change of the explicit mapping output results of each branch in different time periods is statistically analyzed. Based on the mean, variance and extreme values ​​of the distribution change of each branch, soft voting integration is performed to obtain the voting combination of each branch in the same time period. Based on the voting combination, the period with the highest number of votes in the structural safety requirement branch is taken as the starting period of the protection strategy, and the period with the highest number of votes in the damage repair requirement branch is taken as the starting period of the repair strategy. The period with the highest number of votes in the context protection requirement branch is taken as the starting period of the activation strategy. Construction method suggestions are generated based on the construction method entries corresponding to the starting time of each strategy, and a structured decision-making scheme is obtained. The construction method suggestions include applicable conditions and key points of construction.

[0024] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

Claims

1. A multimodal fusion-based decision-making and production method for the preservation of historical buildings, characterized in that: Includes the following steps: Historical documents, real-scene photos, and structural monitoring data of the target building are acquired. The structural monitoring data is time-series calibrated, and structured orthophotos are calculated based on real-scene photos from different perspectives according to the calibration time series. The structured orthophoto image is semantically pre-labeled using a depth illumination decomposition network to obtain a damage region mask and generate a binarized damage heatmap. Based on the aforementioned binary damage heatmap and the time-series calibration, the damage time-series space is divided, and construction terms within the damage time-series space are extracted from historical documents using a pre-set large language model to obtain construction method entries. Based on the construction method terms, historical features are determined, and based on these historical features, small-sample learning is performed on the structural mechanical properties of the damaged area through prototype comparison learning to obtain a representation vector. The representation vector is input into a heterogeneous parallel configuration of the Kolmogorov-Arnold network model, and the protection requirement is explicitly mapped according to the structural features and the binary damage heat map. Based on the output of the explicit mapping, soft voting integration is performed to generate a structured decision scheme that includes protection strategies, repair strategies, activation strategies, and construction method suggestions.

2. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, The method for obtaining the calibration timing includes: The system acquires the construction date, structural type, historical repair records, and architectural style records of the target building, and performs OCR recognition, simplified / traditional character conversion, and domain dictionary matching to obtain architectural attribute entities that match historical documents. It also performs noise reduction, exposure correction, and lens distortion elimination on real-scene photos from different perspectives to obtain a standardized image matrix. The real-scene photos include drone aerial images, ground close-up photography, and laser scanning point cloud data. Outlier removal, missing value imputation, and sampling rate normalization are performed on the structural monitoring data to obtain a unified time-series monitoring matrix. Change point detection is performed based on the unified time-series monitoring matrix to obtain the structural state time. The set of key calibration time is determined based on the structural state time. The structural state time includes the time of structural response change, the start and end time of steady state interval, and the time of extreme value of environmental load. The structural monitoring data includes time-series signals collected by strain gauges, inclinometers, accelerometers, and temperature and humidity sensors. The acquisition timestamps of the standardized image matrix are matched with the nearest neighbor set of key calibration times to obtain the structural state time labels corresponding to each viewpoint image, thus obtaining the calibration time sequence. Based on the structural state time labels of the calibration time sequence, the structured orthophoto is calculated from the real scene photos.

3. The multimodal fusion-based historical building preservation decision-making production method according to claim 2, characterized in that, A method for obtaining the structured orthophoto includes: Based on the structural state time label, the structural response value of the corresponding time in the unified time series monitoring matrix is ​​used as the initial constraint condition. Based on the standardized image matrix, feature point detection and descriptor extraction are performed by feature point detection algorithm to obtain feature point matching pairs. The detection algorithm includes SIFT algorithm and ORB algorithm. Based on the feature point matching pairs, a random sampling consensus algorithm is used to eliminate mismatches and obtain an accurate matching point set. Based on the accurate matching point set and initial constraints, the camera intrinsic and extrinsic parameters and 3D point coordinates are iteratively optimized by bundle adjustment to obtain a sparse reconstructed point cloud. Based on the sparse reconstructed point cloud, multi-view stereo matching and depth map fusion are performed to obtain a dense 3D point cloud model. Based on the dense 3D point cloud model, a horizontal reference plane is selected. The 3D coordinates of each point cloud are mapped to the 2D image coordinates according to the horizontal reference plane, and digital elevation information is generated. The digital elevation information is used as pixel values ​​to obtain structured orthophotos and metadata.

4. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, A method for obtaining the mask of the damaged area includes: The pixels of the structured orthophoto are normalized, and the normalized structured orthophoto is input into a deep illumination decomposition network. Based on the deep illumination decomposition network, the image is decomposed into illumination components and reflection components through an encoder and decoder structure. The image is reconstructed by element-wise multiplication of the illumination components and reflection components to obtain a de-interference image. Based on the reflection component of the de-interference image, the building material texture and damage features are extracted through the convolutional layer of the U-Net model to obtain pre-labeled semantics. A pixel-level damage probability map is obtained based on the pre-labeled semantics, material texture and damage features. Damage candidate regions are marked based on the damage probability map. Based on the damage candidate region, isolated noise points are removed by morphological opening operation, the closed boundary of the denoised candidate region is extracted, a damage region mask is generated, the damage region mask is aligned with the metadata of the structured orthophoto image and associated with the structural state time label to obtain a binarized damage heatmap, the gray value of the binarized damage heatmap is positively correlated with the damage probability.

5. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, The method for obtaining the aforementioned construction method terminology includes: Based on the target building, the damage temporal space is divided according to the pixel coordinates of the calibration time sequence and the binarized damage heat map. Building attribute entities are extracted from historical documents. An index library is constructed based on the building terminology, time information and spatial location of the building attribute entities. Based on the damage temporal space, the repair records, construction method records and craftsmanship texts of the corresponding components in the historical documents are retrieved through the index library to obtain a subset of documents. The damage temporal space includes the spatial range of the damage area, the structural state time label, the damage probability level and the damage type. The damage type, spatial location, temporal information, and literature subset of the damage time-series space are input into a pre-trained large language model to obtain structured architectural terms, which include component names, construction method entries, material types, and process parameters.

6. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, The method for obtaining the representation vector includes: Historical features are determined using structural architectural terminology based on construction method entries. The spatial location, temporal information, and historical features vector of the damaged area are used as feature indexes. A small support set and query set are constructed based on the feature indexes. Each damage type in the support set contains only 3 to 5 labeled samples. The labeled samples include the structural mechanical features of the damaged area, the visual features of the binarized damage heat map, and the spliced ​​vector of historical features. The structural mechanical features include the stress peak, strain gradient, and vibration frequency of the structural monitoring data. The visual features are material texture and damage features. The material texture includes crack orientation, and the damage features include the damage area. The query set includes samples of the damaged area to be predicted. The structure of the query set is consistent with that of the support set. The representation vector is calculated based on the support set and the query set. The formula for calculating the representation vector is as follows: ； in For the representation vector, For modal indexing, For modal sets, For text modal features, For image modal features, For sensor modal characteristics, For feature mapping function, For modality The projection matrix, For modality eigenvectors, For modality eigenvectors; ； in For query vector, For transpose, The hyperbolic tangent activation function is used. Here is the weight matrix of the gated network. is the bias vector of the gated network. The square of the Mahalanobis distance. The prototype center vector is the modality. The category of historical features The mean, The best matching prototype category index is determined based on the matching of construction method terms; The feature mapping function and the weight matrix of the gated network are optimized through backpropagation until the loss converges to a preset threshold, thereby obtaining a trained prototype contrastive learning model. Based on the sample features of the query set, the prototype contrastive learning model is used to learn the structural mechanical features through small samples to obtain the representation vector of the structural mechanical properties of the damaged area.

7. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, The method for obtaining the output result includes: The B-spline basis functions of the Kolmogorov Arnold network model are: ； in For the first B-spline activation function for 3D input, For B-spline basis function index, The total number of B-spline basis functions. For the first Vidi Learnable weights of each basis function For the first A B-spline basis function in The value at that position is calculated according to the Cox-de Boor recursive formula. For B-spline basis functions with the same index and one order lower, For the next index, a B-spline basis function of reduced order, the node vector of the B-spline basis function... Evenly distributed in the input range, Let B be the order of the spline; The representation vectors are input in parallel to the Kolmogorov-Arnold network models of each branch. Linear projection and B-spline activation are applied to each dimension of the representation vectors, and the explicit mapping value of the protection requirement is calculated to obtain the output result of the explicit mapping. The formula for calculating the explicit mapping value is as follows: ； in To protect the explicit mapping value of demand, Let be the dimension number of the representation vector. For the first A projection weight vector in each dimension. For the first Bias in each dimension; The heterogeneous branches include a structural safety requirement branch, a damage repair requirement branch, and a context protection requirement branch. The input feature dimension of the structural safety requirement branch is obtained through the structural feature dimension, the input feature dimension of the damage repair requirement branch is obtained through the binarized damage heatmap dimension, and the input feature dimension of the context protection requirement branch is obtained through the historical landscape feature dimension.

8. The multimodal fusion-based historical building preservation decision-making production method according to claim 1, characterized in that, The method for the structured decision-making scheme includes: The monitoring frequency of each monitoring data in the structural monitoring data is obtained, and the monitoring frequency is normalized according to different time periods. Based on the normalized monitoring frequency, the distribution changes of the explicit mapping output results of each branch in different time periods are statistically analyzed. Based on the mean, variance and extreme values ​​of the distribution changes of each branch, soft voting integration is performed to obtain the voting combination of each branch in the same time period. Based on the voting combination, the period with the highest number of votes in the structural safety requirement branch is taken as the starting period of the protection strategy, and the period with the highest number of votes in the damage repair requirement branch is taken as the starting period of the repair strategy. The period with the highest number of votes in the context protection requirement branch is taken as the starting period of the activation strategy. Construction method suggestions are generated based on the construction method entries corresponding to the starting time of each strategy, and a structured decision-making scheme is obtained. The construction method suggestions include applicable conditions and key points of construction.

Images