A data integration method for the entire life cycle of a building

Abstract

The present invention provides a method for integrating data throughout the entire life cycle of a building, which belongs to the field of building data fusion technology, including: configuring structured and unstructured data collection agents for different data sources by constructing an innovative architecture including a data acquisition microservice cluster, a service registration center, a service monitoring center, and a resource scheduling center. The system comprehensively collects data from the entire life cycle of the building, and builds a service dependency model through microservice operation indicator monitoring and dynamic resource scheduling. Using deep data preprocessing, standard conversion, feature mapping and other technologies, deep neural networks and spectral clustering algorithms are used to construct data lineage maps and coupling analysis. By configuring a data fusion service cluster, formulating a data fusion execution plan and fallback mechanism, and establishing a fusion quality assessment system, the intelligent collection, processing and management of building data throughout the entire life cycle are realized, solving the current problem of lack of data fusion mechanism and inability to guarantee the accuracy and consistency of fusion results.

Description

Technical Field

[0001] The present invention belongs to the technical field of building data fusion, and specifically relates to a method for integrating building data throughout its entire life cycle. Background Art

[0002] As the digital transformation of the construction industry deepens, the scale of data across the entire building lifecycle is experiencing explosive growth. Massive amounts of structured and unstructured data are generated across various stages, including planning, design, construction, and operations. This data comes from a wide range of sources and formats, including Building Information Models (BIM), Computer-Aided Design (CAD) files, construction logs, monitoring data, and maintenance records. Existing technologies primarily use traditional data warehouses and data lakes for data storage and management, integrating data through manual intervention and simple rule engines.

[0003] In real-world scenarios, construction companies typically use multiple specialized software systems to process business data at different stages. For example, urban planning software may be used for land use analysis during the survey phase, BIM software for 3D modeling during the design phase, project management systems may be used to track progress during construction, and facilities management systems may be used to record maintenance information during the operations phase. These systems often form information silos, lacking effective data linkage and integration. This is particularly true in large-scale construction projects, where different parties utilize varying software systems and data standards, making data integration complex and time-consuming.

[0004] The existing technology has the following major problems: First, data collection efficiency is low, and there is a lack of adaptive data collection mechanisms, which makes it impossible to dynamically adjust the collection strategy according to the characteristics of the data source; second, the data preprocessing capabilities are limited, the processing effect of unstructured data is poor, and it is difficult to extract effective semantic information; third, data standards are not unified, and data conversion rules between different systems need to be manually defined and maintained, which is prone to errors; fourth, data association analysis is not in-depth enough, making it difficult to discover potential associations between data; in summary, the existing technology lacks an intelligent data fusion mechanism and cannot guarantee the accuracy and consistency of the fusion results. Summary of the Invention

[0005] In view of this, the present invention provides a method for integrating data throughout the entire life cycle of a building, which can solve the problem that the existing technology lacks an effective data fusion mechanism and cannot guarantee the accuracy and consistency of the fusion results.

[0006] The present invention is achieved in that:

[0007] The present invention provides a method for integrating data throughout the entire life cycle of a building, comprising the following steps:

[0008] S01. Build an adaptive multi-channel data acquisition microservice system, which includes a data acquisition microservice cluster, a service registration center, a service monitoring center, and a resource scheduling center. The data acquisition microservice cluster is configured with different data acquisition agents for different data source types, and the data acquisition agents include structured data acquisition agents and unstructured data acquisition agents.

[0009] S02. Collect microservice operation indicators from the service monitoring center, the microservice operation indicators including service response time, service throughput, and resource utilization, and generate a first performance measurement result. The resource scheduling center calculates the microservice resource demand based on the first performance measurement result; and allocates resources to the data collection microservice cluster based on the microservice resource demand to obtain a first resource allocation result.

[0010] S03. Obtaining data collection tasks corresponding to the first resource allocation result, the data collection tasks including survey phase data collection tasks, design phase data collection tasks, construction phase data collection tasks, and operation and maintenance phase data collection tasks; collecting building life cycle data according to the data collection tasks, the building life cycle data including survey phase data, design phase data, construction phase data, and operation and maintenance phase data;

[0011] S04. Construct a first service dependency model based on the building lifecycle data. The first service dependency model calculates the microservice dependency level based on the input data and output data of each microservice in the data acquisition microservice cluster to generate a first dependency relationship result. Generate a data acquisition microservice deployment sequence based on the first dependency relationship result.

[0012] S05. Preprocessing the building life cycle data, performing data cleaning operations, duplicate data removal operations, and abnormal data processing operations on structured data in the building life cycle data, and performing text segmentation operations, feature extraction operations, and semantic annotation operations on unstructured data in the building life cycle data to generate a first preprocessed data set;

[0013] S06. Construct a data standard conversion model, the data standard conversion model including structured data conversion rules and unstructured data conversion rules; convert the first preprocessed data set into a first standardized data set according to the data standard conversion model; extract data standard update parameters from the first standardized data set, and update the data standard conversion model according to the data standard update parameters;

[0014] S07. Construct a data feature mapping model, wherein the data feature mapping model uses a tensor decomposition operation to map the first standardized data set to a feature space to obtain a first feature vector set; establish a feature index library for the first feature vector set, and generate a first feature retrieval result;

[0015] S08. Input the first feature retrieval result into a deep neural network model, wherein the deep neural network model uses a multi-head attention calculation unit to extract feature association strength to obtain a first association strength matrix; and establish a data lineage map based on the first association strength matrix;

[0016] S09, converting the data lineage map into a first data association matrix, processing the first data association matrix using a spectral clustering algorithm, and calculating a data coupling degree; determining a data fusion range based on the data coupling degree, and generating a first fusion boundary result;

[0017] S10. Build a fusion priority model based on the first fusion boundary result, generate a data fusion execution plan based on the fusion priority model, and include data alignment rules, conflict handling rules, and consistency verification rules; and establish a data fusion fallback mechanism that triggers a data fallback operation based on fusion exception information.

[0018] S11. Configuring a data fusion service cluster, wherein the data fusion service cluster includes a structured data fusion service and an unstructured data fusion service; invoking the data fusion service cluster to perform data fusion according to the data fusion execution plan, and generating first fusion result data;

[0019] S12. Establish a fusion quality assessment system, which includes integrity assessment indicators, consistency assessment indicators, accuracy assessment indicators and timeliness assessment indicators; evaluate the first fusion result data according to the fusion quality assessment system to generate a first quality assessment report; update the service parameter configuration of the data fusion service cluster according to the first quality assessment report.

[0020] Wherein, the step S10 specifically includes:

[0021] Step 1001: construct a fusion priority evaluation model based on the first fusion boundary result;

[0022] Step 1002: Set data fusion priority calculation rules, including data importance weight and data timeliness weight;

[0023] Step 1003: Generate a data fusion execution plan, including data alignment rules, conflict handling rules, and consistency verification rules;

[0024] Step 1004: Establish a data fusion anomaly detection mechanism, including data consistency check and data integrity check;

[0025] Step 1005: Build a data fusion rollback mechanism to trigger data rollback operations based on fusion exception information;

[0026] Step 1006: Record the data fusion execution log for data fusion process tracking and optimization.

[0027] Wherein, the steps S11 and S12 specifically include:

[0028] Step 1101: Configure a data fusion service cluster, including structured data fusion service and unstructured data fusion service;

[0029] Step 1102: Perform a data fusion operation to generate first fusion result data;

[0030] Step 1103: Establish a fusion quality assessment system, including integrity assessment indicators, consistency assessment indicators, accuracy assessment indicators, and timeliness assessment indicators;

[0031] Step 1104: Perform a multi-dimensional evaluation on the first fusion result data according to the fusion quality evaluation system;

[0032] Step 1105: Generate a first quality assessment report, including data quality analysis results and optimization suggestions;

[0033] Step 1106: Update the service parameter configuration of the data fusion service cluster according to the first quality assessment report.

[0034] Furthermore, the microservice resource requirements in S02 are calculated as follows:

[0035] The microservice resource requirement matrix M is calculated as follows:

[0036]

[0037] Among them, R ij The demand for the i-th service in the j-th resource dimension is calculated as follows:

[0038]

[0039] Where: T ij is the service response time matrix element; Q ij is the throughput matrix element; U ij is the resource utilization matrix element; α1, α2, α3 are weight coefficients; ∈1 is the error term.

[0040] Resource allocation optimization objective function:

[0041]

[0042] Constraints:

[0043]

[0044] Where: A ij Assign matrix elements to resources; C j is the total capacity of the jth type of resources.

[0045] Service dependency calculation in S04:

[0046] The service dependency matrix D is calculated as follows:

[0047]

[0048] Among them, the inter-service dependency intensity d ij The calculation is as follows:

[0049]

[0050] Service call frequency matrix F:

[0051]

[0052] Where: f ij Indicates the number of times service i calls service j per unit time.

[0053] Service deployment sequence optimization objective function:

[0054]

[0055] Constraints:

[0056]

[0057] Where: x i is the deployment decision variable; k is the number of deployment batches.

[0058] Data preprocessing calculation in S05:

[0059] The data outlier detection threshold θ is calculated as follows:

[0060] θ=μ±kσ;

[0061] Where: μ is the sample mean; σ is the sample standard deviation; k is the adjustment coefficient, and its value range is [2,3].

[0062] Calculation of text feature word importance score:

[0063]

[0064] Where: n ij is the number of occurrences of word i in document j; |D| is the total number of documents; |{d:t i ∈d}| is the number of documents containing word i.

[0065] Tensor decomposition feature map in S07:

[0066] The third-order data tensor The CP decomposition is as follows:

[0067]

[0068] Where: r is the weight of the rth component; is the decomposition vector; represents the outer product operation; ε is the residual tensor.

[0069] Eigenvector similarity matrix S calculation:

[0070]

[0071] in:

[0072] Multi-head attention calculation in S08:

[0073] The query matrix Q, key matrix K, and value matrix V are calculated as follows:

[0074] Q=XW Q ,K=XW K ,V=XW V ;

[0075] Where: X is the input feature matrix; W Q ,W K ,W V is the parameter matrix.

[0076] Attention weight matrix A calculation:

[0077]

[0078] Where: d k is the key vector dimension.

[0079] Multi-head attention output Z calculation:

[0080] Z=Concat(head1,...,head h )W O ;

[0081] in:

[0082] Spectral clustering calculation in S09:

[0083] The Laplace matrix L is calculated as:

[0084] L = DW;

[0085] Where: W is the similarity matrix; D is the degree matrix, D ii =∑ j W ij .

[0086] Eigendecomposition:

[0087] Lv=λv;

[0088] Get the eigenvalue λ and eigenvector v.

[0089] Calculation of data coupling matrix C:

[0090]

[0091] Where: c ij =γ1S ij +γ2T ij +γ3P ij +∈3.

[0092] Fusion priority calculation in S10:

[0093] The fusion priority score P is calculated as:

[0094] P i =ω1C i +ω2U i +ω3E i +∈4;

[0095] Where: C i is the coupling score; U i is the data update frequency; E i is the importance of the data; ω1, ω2, ω3 are weight coefficients.

[0096] Semantic similarity calculation in S11:

[0097] The cosine similarity of the text vectors v1 and v2:

[0098]

[0099] Calculation of quality assessment indicators in S12:

[0100] The integrity index CI is calculated as:

[0101]

[0102] Where: δi is the field integrity indicator function; w i is the field weight.

[0103] Calculation of consistency index CON:

[0104]

[0105] Where: x ij ,y ij is the data element to be compared; MAX is the normalization coefficient.

[0106] Accuracy index ACC calculation:

[0107]

[0108] Where: TP, TN, FP, and FN are the number of true positives, true negatives, false positives, and false negatives, respectively.

[0109] Calculation of timeliness index T:

[0110] T=exp(-λ(t c -t u ));

[0111] Where: t c is the current time; t u is the data update time; λ is the timeliness attenuation coefficient.

[0112] Calculation of overall quality score Q:

[0113] Q = αCI + βCON + γACC + δT;

[0114] Where: α, β, γ, δ are weight coefficients, and they satisfy α + β + γ + δ = 1.

[0115] The main considerations for these calculations and matrices are:

[0116] 1. The microservice resource configuration matrix takes into account service performance, resource utilization, and load balancing;

[0117] 2. Feature mapping uses tensor decomposition to achieve dimensionality reduction and feature extraction of multidimensional data;

[0118] 3. The attention mechanism captures long-range dependencies between data through a query-key-value mechanism;

[0119] 4. Spectral clustering clusters data based on graph theory and is more suitable for discovering non-convex clusters;

[0120] 5. Quality assessment adopts a multi-dimensional indicator system to ensure the reliability of data fusion.

[0121] Compared with existing technologies, the proposed method for integrating building lifecycle data integration achieves intelligent and automated data collection by constructing an adaptive multi-channel data collection microservice system. The system automatically selects appropriate collection agents based on the characteristics of different data sources, significantly improving data collection efficiency. Compared with traditional manually configured data collection methods, this method significantly reduces manual intervention, improves the real-time and accuracy of data collection, and significantly enhances the utilization efficiency of system resources through intelligent resource scheduling mechanisms.

[0122] In terms of data preprocessing, this invention designs specialized processing flows for structured and unstructured data, respectively. Deep learning algorithms are employed to achieve high-quality text segmentation and feature extraction, significantly improving the processing of unstructured data. Furthermore, the adaptive update mechanism of the data standard conversion model enables dynamic optimization of data conversion rules as business changes, significantly reducing the manual workload of data standard maintenance and effectively improving the level of automation in data processing.

[0123] This paper innovatively employs tensor decomposition and a multi-head attention mechanism to construct efficient data feature mapping models and deep neural network models. These models accurately capture the relationships between data, significantly improving the integrity and accuracy of the data lineage map. Data coupling analysis based on a spectral clustering algorithm provides a scientific basis for determining the optimal fusion range and effectively improves the accuracy of data fusion boundary delineation.

[0124] Regarding data fusion quality, this paper establishes a comprehensive fusion quality assessment system, encompassing multiple metrics including completeness, consistency, accuracy, and timeliness. Automatic optimization of fusion service parameters significantly improves the accuracy and consistency of fusion results. In particular, the introduction of a fusion fallback mechanism enables the system to promptly detect and address fusion anomalies, significantly enhancing the reliability and stability of the data fusion process.

[0125] Compared with existing technologies, this invention offers significant technological advancements: First, it automates the entire data collection and preprocessing process, significantly improving processing efficiency and data quality. Second, it innovatively applies deep learning technology to significantly enhance the accuracy and depth of data association analysis. Third, it establishes an adaptive data fusion mechanism, significantly improving the quality and reliability of fusion results. Fourth, it introduces comprehensive quality assessment and exception handling mechanisms, significantly enhancing the system's fault tolerance and stability. These advancements effectively address the existing problem of a lack of effective data fusion mechanisms, which prevents the accuracy and consistency of fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] Figure 1 A flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0127] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0128] like Figure 1 The figure shows a flow chart of a method for integrating building lifecycle data provided by the present invention. The specific implementation of each step in the present invention is described in detail below:

[0129] The specific implementation method of step S01 is to first build a service registration center to manage the service registration and service discovery of the data acquisition microservice cluster. The service registration center provides a unified service registration and service discovery mechanism to ensure the interoperability and schedulability between various data acquisition microservices. Secondly, a service monitoring center is built to collect the operating status data of the data acquisition microservice cluster in real time, including indicators such as central processing unit occupancy, memory occupancy, disk read and write rate, and network transmission rate. The service monitoring center can perceive the operating status of microservices in a timely manner and provide a basis for resource scheduling. Finally, a resource scheduling center is built to dynamically allocate computing resources based on monitoring data. The resource scheduling center will reasonably allocate computing resources such as CPU, memory, and storage based on the real-time resource requirements of each microservice to ensure the overall performance of the microservice cluster.

[0130] For structured data sources, deploy structured data collection agents to collect database and file system data. Structured data collection agents can connect to different types of structured data sources, such as relational databases, NoSQL databases, and file systems, and extract the required information from them using customized data collection logic. For unstructured data sources, deploy unstructured data collection agents to collect image data, text data, and audio data. Unstructured data collection agents provide flexible data collection mechanisms for different types of unstructured data sources, capable of extracting unstructured data in various formats.

[0131] In addition, the system has established a microservice monitoring and alarm mechanism. If a microservice's operating indicators exceed preset thresholds, such as excessive CPU utilization or long response times, the monitoring system will automatically trigger dynamic adjustment measures such as resource expansion or service migration to ensure the stable operation of the entire system.

[0132] The specific implementation of step S02 is to first collect the operating indicators of each data collection microservice in the service monitoring center. These indicators include service response time, service throughput, and resource utilization. Service response time refers to the time interval between the initiation of a service request and the completion of the response, reflecting the processing speed of the service. Service throughput refers to the number of service requests completed per unit time, reflecting the processing capacity of the service. Resource utilization is the ratio of used resources to total resources, describing the resource consumption of the service.

[0133] Based on these operating indicator data, a first performance measurement result is generated. Based on this performance measurement result, the resource scheduling center calculates the resource requirements of each data collection microservice.

[0134] Through the above calculations, the resource scheduling center can reasonably allocate various resources such as computing, storage, and network according to the actual operating status of microservices to meet the needs of data collection tasks.

[0135] The specific implementation of step S03 is to first obtain corresponding data collection tasks based on the resource allocation results obtained in the previous step. These tasks include data collection in the survey phase, data collection in the design phase, data collection in the construction phase, and data collection in the operation and maintenance phase.

[0136] Data collection during the survey phase primarily involves acquiring planning text and graphic data, such as urban master plans and architectural plans. Data collection during the design phase focuses on building information model data and design documentation, such as BIM models and design drawings. Data collection during the construction phase includes construction process data and quality inspection data, reflecting the actual construction status. Finally, data collection during the operation and maintenance phase involves equipment operation data and maintenance records, documenting the building's usage.

[0137] After collecting data from each life cycle stage, the system will classify, store, and index the data, which will facilitate subsequent data retrieval and analysis applications.

[0138] The specific implementation of step S04 is to first analyze the input and output data types of each microservice in the data acquisition microservice cluster. This will help understand the data sources processed by each microservice and their dependencies. The frequency of data transmission between microservices will then be counted, reflecting the mutual calls between them.

[0139] Based on the above analysis results, we constructed the first service dependency model. This model includes two elements: service nodes and service relationships. Service nodes represent individual data collection microservices, while service relationships represent the call dependencies between them.

[0140] Next, calculate the degree of dependency between service nodes.

[0141] Based on the calculated dependency strength and dependency direction, a first dependency result is generated. Finally, based on this result, the deployment order of the data collection microservices is determined to minimize the inter-service call overhead.

[0142] Through the above steps, the system established a dependency model for data acquisition microservices and optimized the deployment order of microservices, laying the foundation for subsequent data processing.

[0143] The specific implementation of step S05 is to first perform data cleaning on the structured data. Data cleaning includes handling null values ​​and formatting to ensure data integrity and consistency. Next, duplicate data is removed from the structured data, using a data fingerprint-based method for identification and merging. Finally, outlier processing is performed on the structured data, using statistical analysis methods to identify and correct outliers.

[0144] For unstructured data, we first perform text segmentation. Here, we use a dictionary-based matching method to segment the text into tokens. We then extract the TF-IDF features of these tokens, which reflect the importance of the words in the text. Finally, we perform semantic annotation, mapping the tokens to semantic concepts using a knowledge graph, enhancing the semantic expressiveness of unstructured data.

[0145] Through the above preprocessing operations, the system converts the original building life cycle data into a structured first preprocessed data set, which lays the foundation for subsequent data standardization and feature extraction.

[0146] The specific implementation of step S06 is to first establish structured data conversion rules. This includes field mapping relationships and data type conversion rules for standardizing the original structured data. At the same time, for unstructured data, rules for text semantic mapping and feature vector conversion are established.

[0147] These structured data conversion rules and unstructured data conversion rules are integrated into a unified data standard conversion model, which can automatically convert the first preprocessed data set to generate a first standardized data set.

[0148] During the data conversion process, the system also extracts parameters for updating data standards. These parameters reflect the limitations of the current data standards and areas for improvement. Based on these parameters, the system can dynamically update the data standard conversion model to continuously improve the effectiveness of data standardization.

[0149] Through the above steps, the system has built an adaptive data standard conversion mechanism. It can standardize the original heterogeneous data and lay the foundation for subsequent data fusion.

[0150] The specific implementation of step S07 is to first construct a data feature mapping model. The model includes two parts: a feature decomposition module and a feature reconstruction module. The feature decomposition module uses a tensor decomposition method to map the first standardized data set to a feature space.

[0151] Next, the system will create an index library of feature vector sets and construct a fast retrieval structure based on feature similarity. Through the above feature mapping and index construction, the system can efficiently extract key features from massive amounts of building data, laying the foundation for subsequent association analysis.

[0152] The specific implementation of step S08 involves first constructing a deep neural network model, comprising a feature encoding layer and a feature decoding layer. In the feature encoding layer, a multi-head attention computation unit is used to extract the correlation strength between features. This multi-head attention mechanism enables the system to effectively capture long-range dependencies between input features, enhancing feature representation capabilities.

[0153] The first feature retrieval results are fed into the deep neural network model, and after feature encoding and decoding, a first correlation strength matrix is ​​generated. Based on this matrix, the system constructs a data lineage map that intuitively reflects the associations between data elements.

[0154] The specific implementation of step S09 is to first convert the data lineage map into a first data association matrix in the form of an adjacency matrix, and then analyze the matrix using a spectral clustering algorithm.

[0155] The core idea of ​​spectral clustering is to first construct a Laplacian matrix L = DW, where W is the similarity matrix and D is the degree matrix. L is then subjected to eigendecomposition to obtain eigenvalues ​​and eigenvectors. Based on the eigenvectors, data points can be divided into different clusters.

[0156] Based on this priority assessment model, the system generates a specific data fusion execution plan. This plan includes data alignment rules, conflict resolution rules, and consistency verification rules. Furthermore, the system establishes a data fusion anomaly detection mechanism and a data fusion fallback mechanism to ensure the reliability of the fusion process.

[0157] In addition, the system also records the execution log of data fusion, providing a basis for subsequent process optimization. Through the above measures, the system has built a complete data fusion management and control system.

[0158] The specific implementation of steps S11 and S12 involves first configuring a data fusion service cluster. This cluster includes two categories: structured data fusion services and unstructured data fusion services. When executing data fusion, the system will call the corresponding fusion service based on the fusion execution plan to complete operations such as data alignment, conflict resolution, and consistency verification, generating the first fusion result data.

[0159] In order to evaluate the quality of the fusion results, the system has established a fusion quality evaluation system. The system includes the following indicators:

[0160] 1) Completeness index CI, which reflects the coverage of fused data;

[0161] 2) consistency index CON, which describes the internal consistency of the fused data;

[0162] 3) Accuracy index ACC, which measures the degree of agreement between the fused data and the reference data;

[0163] 4) Timeliness index T reflects the timeliness of the fused data.

[0164] Based on these indicators, the system evaluates the first fusion result data and generates a first quality assessment report. This report not only includes scores for each quality indicator but also provides optimization suggestions. Finally, based on the quality report, the system dynamically adjusts the parameters of the data fusion service to continuously improve the fusion effect.

[0165] Through the above steps, the system has established a closed-loop data fusion quality control mechanism. The fusion results not only meet data standards and consistency requirements, but can also be continuously optimized to ensure the reliability and practicality of the fused data.

[0166] Specifically, the principle of the present invention is:

[0167] 1. Adaptive multi-channel data acquisition

[0168] This paper constructs a data collection system based on a microservices architecture. The system includes three core components: a service registration center, a service monitoring center, and a resource scheduling center. The service registration center manages the registration and discovery of data collection microservices, ensuring interoperability between them. The service monitoring center collects microservice operating metrics in real time, such as CPU, memory, disk, and network usage, to provide a basis for resource scheduling. The resource scheduling center dynamically allocates computing resources based on monitoring data, ensuring efficient execution of data collection tasks.

[0169] The system also deploys structured and unstructured data collection agents for different data sources. The structured data collection agent connects to various databases and file systems to extract the required engineering documents and model data. The unstructured data collection agent focuses on collecting unstructured data such as images, videos, and text. Through this adaptive data collection mechanism, the system can fully perceive the diverse data generated at all stages of a construction project.

[0170] 2. Deep Learning-Driven Correlation Analysis

[0171] After data collection is complete, the present invention utilizes deep learning technology to perform correlation analysis on the data. First, the system standardizes the collected data, including data cleaning and format conversion. Then, tensor decomposition is used to map the high-dimensional data into a feature space, extracting potential correlations between data elements.

[0172] Next, the system built a deep neural network model that, through a multi-head attention mechanism, mined strong dependencies between data elements. The attention mechanism captures long-range dependencies between data elements through a query-key-value approach, and can more accurately depict the semantic connections between data than traditional fully connected networks.

[0173] Finally, based on the above association analysis results, the system generated a detailed data lineage map, which intuitively displayed the inherent connections between data elements. This laid the foundation for subsequent data fusion.

[0174] 3. Data Coupling Analysis Based on Graph Theory

[0175] To determine the scope and strategy for data fusion, this paper uses a spectral clustering algorithm to analyze the coupling relationships between data. First, the system converts the data lineage map into a data association matrix in the form of an adjacency matrix. Then, a Laplacian matrix is ​​constructed and eigendecomposition is performed to obtain the clustering structure of the data elements.

[0176] Based on the clustering results, the system calculated the degree of coupling between data elements. This degree of coupling takes into account multiple factors, including feature similarity, topological distance, and attribute relevance, and comprehensively characterizes the coupling relationship between data. By setting a coupling threshold, the system determines the scope of data fusion and generates a fusion boundary.

[0177] This coupling degree calculation method based on graph theory analysis can effectively discover potential coupling relationships between data, providing an important basis for subsequent accurate data fusion. Compared with traditional rule-based fusion methods, this method is closer to the intrinsic characteristics of the data and can adaptively determine the fusion range.

[0178] 4. Adaptable data fusion system

[0179] For defined fusion boundaries, the present invention constructs an adaptive data fusion system. This system first calculates the data fusion priority based on data importance, update frequency, and coupling strength. During the fusion process, the system automatically completes the data fusion operation based on alignment rules, conflict resolution rules, and consistency verification rules.

[0180] The system also establishes a fusion quality assessment mechanism. It comprehensively evaluates fusion results using multi-dimensional metrics such as completeness, consistency, accuracy, and timeliness. These results are not only provided to users but also used to adjust fusion service parameters and continuously improve fusion quality. Furthermore, any anomalies during the fusion process are monitored and addressed, ensuring the reliability of the fusion results.

[0181] This adaptive fusion mechanism based on quality feedback can effectively cope with the dynamic changes in the construction project data environment, ensuring that the fusion results always meet business needs. Compared with traditional static fusion methods, this method is more intelligent and flexible, significantly improving the reliability of data fusion.

[0182] A specific embodiment 1 of the present invention is provided below. Each step in this embodiment 1 is described in detail as follows:

[0183] The specific implementation of step S01 is as follows:

[0184] First, build a service registry Used to manage the service registration and service discovery of the data collection microservice cluster. The service registration center provides a unified service registration and service discovery mechanism to ensure the interoperability and schedulability between various data collection microservices. The service registration center maintains a service registry This records metadata for each microservice, such as the service name, service address, and service interface. When a data collection microservice starts, it registers its information with the service registry, which then updates the service registry accordingly. During service discovery, microservices can query the registry to obtain access information for the microservices they need.

[0185] Secondly, build a service monitoring center Used to collect data in real time to collect the running status data of the microservice cluster. The service monitoring center periodically collects the CPU usage of each microservice U cpu , memory usage U mem , disk read and write rate R disk , network transmission rate R net And other indicators, generate the microservice operation indicator matrix M:

[0186]

[0187] These monitoring data provide a basis for resource scheduling.

[0188] Finally, build a resource scheduling center Computing resources are dynamically allocated based on monitoring data. The resource scheduling center will reasonably allocate computing resources such as CPU, memory, and storage based on the real-time resource requirements of each microservice to ensure the overall performance of the microservice cluster. The optimization objective function of resource allocation is:

[0189]

[0190] Among them, R ij A is the demand of the i-th microservice on the j-th resource, ij is the amount of resources of type j actually allocated to the i-th microservice. The constraint is that the total amount of resources allocated does not exceed the total capacity C j :

[0191]

[0192] Deploy structured data collection agents for structured data sources Used to collect database data and file system data. It can connect to different types of structured data sources, such as relational databases, NoSQL databases, file systems, etc., and extract the required information from them through customized data collection logic.

[0193] For unstructured data sources, deploy unstructured data collection agents Used to collect image data, text data and audio data. For different types of unstructured data sources, a flexible data collection mechanism is provided, which can extract unstructured data in various formats.

[0194] In addition, the system has established a microservice monitoring and alarm mechanism. Once the operating indicators of a microservice exceed the preset threshold θ, the monitoring system will automatically trigger dynamic adjustment measures such as resource expansion or service migration to ensure the stable operation of the entire system.

[0195] The specific implementation of step S02 is as follows:

[0196] First, the service monitoring center Collect the operating indicators of each data collection microservice, including service response time T, service throughput Q, and resource utilization U. The service response time T reflects the processing speed of the service and can be calculated through monitoring indicators:

[0197]

[0198] in, is the initiation time of the i-th service request, The time when the response to this request is completed.

[0199] The service throughput Q reflects the processing capacity of the service and can be calculated as the number of service requests completed per unit time:

[0200]

[0201] Resource utilization rate U is the ratio of used resources to total resources, which can be directly calculated based on monitoring indicators.

[0202] Based on these operating indicator data, a first performance measurement result P is generated:

[0203]

[0204] Resource Scheduling Center The resource requirement R of each microservice will be calculated based on P. The specific calculation formula is:

[0205]

[0206] Among them, α1, α2, and α3 are the weight coefficients of the corresponding indicators, and ε1 is the error term. The weight coefficients can be adjusted according to actual needs.

[0207] Finally, the Resource Scheduling Center By optimizing the objective function, the final resource allocation result A is obtained:

[0208]

[0209] The specific implementation of step S03 is as follows:

[0210] First, the resource scheduling center According to the resource allocation result A obtained in the previous step, obtain the corresponding data collection task set These tasks include:

[0211] 1) Data collection tasks during the survey phase Including planning text data and planning graphic data;

[0212] 2) Data collection tasks in the design phase Including building information model data and design document data;

[0213] 3) Data collection tasks during the construction phase Including construction process data and quality inspection data;

[0214] 4) Data collection tasks during the operation and maintenance phase Including equipment operation data and maintenance record data.

[0215] For each data collection task There are corresponding data source addresses x t , acquisition period τt and sampling frequency f t .

[0216] After collecting data from each life cycle stage, the system will collect the data of the entire life cycle of the building. Classification storage and index establishment are carried out to facilitate subsequent retrieval and analysis applications.

[0217] The specific implementation of step S04 is as follows:

[0218] First, analyze the data collection microservice cluster The input data type I and output data type O of each microservice form the service metadata description.

[0219] Next, we count the data transmission frequency F between microservices, which reflects the mutual calls between them. The element f of F ij Indicates the number of times service i calls service j per unit time.

[0220] Based on the above analysis results, the first service dependency model is constructed in is the set of service nodes, and ε is the set of service relationship edges. i Represents the i-th data collection microservice, service relationship edge e ij Indicates that service i depends on service j.

[0221] Next, calculate the dependency D between each service node. Dependence d ij It can be calculated by the following formula:

[0222]

[0223] Among them, β1, β2, β3 are the weight coefficients of the corresponding indicators, and ∈2 is the error term.

[0224] Based on the calculated dependency matrix D, the first dependency result is generated Including dependency strength and dependency direction.

[0225] Finally, according to Determine the deployment order x of the data collection microservices to minimize the inter-service call overhead:

[0226]

[0227] Among them, x i is the deployment decision variable, and k is the number of deployment batches.

[0228] The specific implementation of step S05 is as follows:

[0229] For structured data First, perform data cleaning operations. Data cleaning includes:

[0230] 1) Null value processing: For missing values, the mean value μ, median Or fill with the mode η;

[0231] 2) Format unification: unify data in different formats into the same data type.

[0232] Next, we perform a deduplication operation on the structured data. Here, we use a data fingerprint-based method to calculate the fingerprint value h(x) of the data element using the hash function h, and merge duplicate data based on the fingerprint value.

[0233] Finally, perform outlier processing on the structured data. Using statistical analysis methods, we can calculate the mean μ and standard deviation σ of the data, identify data points that satisfy |x - μ| > kσ as outliers, and perform corrections. Here, k is the adjustment coefficient, which ranges from [2 to 3].

[0234] For unstructured data First, perform the text segmentation operation. Here, we use the dictionary matching method to split the text data into word sets.

[0235] Then, the TF-IDF features F of these words are extracted, which reflects the importance of the words in the text:

[0236]

[0237] Among them, n ij For word w i In the document d j The number of occurrences in is the total number of documents,|{d:w i ∈d}| is the word containing word w i The number of documents.

[0238] Finally, semantic annotation is performed on unstructured data. Here, the pre-built knowledge graph is used Mapping word units to semantic concepts enhances the semantic expression ability of unstructured data.

[0239] Through the above preprocessing operations, the system converts the original building life cycle data First preprocess the dataset into a structured

[0240] The specific implementation of step S06 is as follows:

[0241] First, establish structured data conversion rules This includes:

[0242] 1) Field mapping relationship M, mapping the original field name to the standard field name;

[0243] 2) Data type conversion rule T, converts the original data type into the standard data type.

[0244] At the same time, for unstructured data, text semantic mapping rules are established and eigenvector transformation rules

[0245] Integrate these structured data conversion rules and unstructured data conversion rules into a unified data standard conversion model The model can be used to first preprocess the dataset Perform automatic conversion to generate the first standardized data set

[0246] During the data conversion process, the system will also extract some data standard update parameters Θ. These parameters reflect the limitations of the current data standard and areas that need improvement, such as field coverage, data type distribution, and feature vector distribution.

[0247] Finally, the model is transformed according to the updated data standard Θ Continuously improve the effectiveness of data standardization.

[0248] The specific implementation of step S07 is as follows:

[0249] First, build a data feature mapping model It includes two parts: feature decomposition module and feature reconstruction module. The feature decomposition module uses CP decomposition to transform the first standardized data set into Mapped to the feature space. Specifically, a third-order data tensor can be constructed Perform CP decomposition:

[0250]

[0251] Among them, λ r is the weight of the rth component, is the decomposition vector, represents the outer product operation, and ε is the residual tensor.

[0252] This gives us the feature vector set of the data

[0253] Next, the system will create an index library of feature vector sets And build a fast retrieval structure based on feature similarity. The similarity S between feature vectors can be calculated by cosine similarity:

[0254]

[0255] Will and As output, the first feature retrieval result is generated

[0256] Through the above-mentioned feature mapping and index construction, the system can efficiently extract key features from massive building data, laying the foundation for subsequent association analysis.

[0257] The specific implementation of step S08 is as follows:

[0258] First, build a deep neural network model It includes feature encoding layer and feature decoding layer. In the feature encoding layer, a multi-head attention calculation unit is used to extract the correlation strength between features.

[0259] The multi-head attention calculation is divided into the following steps:

[0260] 1) Generate query matrix Q, key matrix K, and value matrix V. The specific formula is:

[0261] Q=XW Q ,K=XW K ,V=XW V ;

[0262] Among them, X is the input feature matrix, W Q ,W K ,W V is the parameter matrix.

[0263] 2) Calculate the attention weight matrix A, the formula is:

[0264]

[0265] Among them, d k is the key vector dimension.

[0266] 3) Calculate the multi-head attention output Z, the formula is:

[0267] Z=Concat(head1,…,head h )W O ;

[0268] in,

[0269] The first feature retrieval result Input the deep neural network model After feature encoding and decoding, the first correlation strength matrix A is finally generated (1) .

[0270] Based on A(1) , the system builds a data lineage map in is the set of data element nodes, and ε is the set of associated edges between elements.

[0271] The specific implementation of step S09 is as follows:

[0272] First, the data lineage map The first data association matrix C converted into the adjacency matrix form (1) .

[0273] Then, the spectral clustering algorithm is used to (1) Specifically, we first construct the Laplace matrix L = DC (1) , where D is the degree matrix, Then, L is decomposed to obtain the eigenvalue λ and eigenvector v. According to the eigenvector, the data element nodes can be divided into different clusters.

[0274] Based on the results of spectral clustering, the system calculates the coupling degree C between data elements:

[0275] c ij =γ1s ij +γ2t ij +γ3p ij +∈3;

[0276] Among them, s ij is the feature similarity, t ij is the topological distance, p ij is the attribute correlation. γ1,γ2,γ3 are the weight coefficients of the corresponding indicators, and ∈3 is the error term.

[0277] By setting the coupling threshold θ c , the system determines the range of data fusion and generates the first fusion boundary result This provides boundary conditions for subsequent data fusion.

[0278] Finally, Verification and optimization are carried out to ensure the rationality of the fusion range.

[0279] The specific implementation of step S10 is as follows:

[0280] First, according to the first fusion boundary result Constructing a fusion priority assessment model The model takes into account the data coupling degree C i , update frequency U i and importance E i Three factors, calculate the fusion priority score P i :

[0281] P i =ω1C i +ω2U i +ω3E i +∈4;

[0282] Among them, ω1, ω2, ω3 are the weight coefficients of the corresponding indicators, and ∈4 is the error term.

[0283] Based on this priority evaluation model The system generates a specific data fusion execution plan The program includes:

[0284] 1) Data alignment rules Used to eliminate inconsistencies in data format and semantics;

[0285] 2) Conflict resolution rules Used to resolve value conflicts between multi-source data;

[0286] 3) Consistency verification rules Used to verify the logical consistency of the fusion results.

[0287] At the same time, the system also establishes a data fusion anomaly detection mechanism and a data fusion rollback mechanism. The former is used to monitor anomalies in the fusion process, and the latter is used to trigger data rollback operations when anomalies occur, ensuring the reliability of the fusion process.

[0288] In addition, the system also records the execution log of data fusion Provide a basis for subsequent process optimization.

[0289] Through the above measures, the system has built a complete data fusion management and control system.

[0290] The specific implementation of steps S11 and S12 is as follows:

[0291] First, configure the data fusion service cluster This cluster includes structured data fusion services and unstructured data fusion services There are two categories. When executing data fusion, the system will execute the fusion plan according to the Call the corresponding fusion service to complete data alignment, conflict resolution, and consistency verification operations to generate the first fusion result data set

[0292] In order to evaluate the quality of the fusion results, the system established a fusion quality evaluation system Q. The system includes the following indicators:

[0293] 1) Completeness index CI, which reflects the coverage of fused data:

[0294]

[0295] Among them, δ i is the field integrity indicator function, w i is the field weight;

[0296] 2) Consistency indicator CON, which describes the internal consistency of the fused data:

[0297]

[0298] Among them, x ij ,y ij is the data element to be compared, MAX is the normalization coefficient;

[0299] 3) Accuracy index ACC, which measures the degree of consistency between the fused data and the reference data:

[0300]

[0301] Among them, TP, TN, FP, and FN are the number of true positives, true negatives, false positives, and false negatives respectively;

[0302] 4) Timeliness index T, which reflects the timeliness of fused data:

[0303] T=exp(-λ(t c -t u ));

[0304] Among them, t c is the current time, t u is the data update time, and λ is the timeliness attenuation coefficient.

[0305] According to the above indicators, the system performs the first fusion result data set Conduct assessment and generate first quality assessment report The report not only includes the scores of various quality indicators, but also provides optimization suggestions.

[0306] Finally, the system reports the quality Dynamically adjust data fusion service clusters parameter configuration to continuously improve the fusion effect.

[0307] Through the above steps, the system has established a closed-loop data fusion quality control mechanism. The fusion results not only meet data standards and consistency requirements, but can also be continuously optimized to ensure the reliability and practicality of the fused data.

[0308] To better understand and implement the present invention, Example 2 of a specific application scenario is provided below: A construction company is undertaking a large-scale commercial complex project, encompassing the entire lifecycle phases of survey, design, construction, and operations and maintenance. The company hopes to fully integrate the massive amounts of heterogeneous data generated at each stage to provide strong support for project management. To this end, the company has decided to adopt the method for integrating and integrating building lifecycle data proposed in this invention to build a data management system.

[0309] Data collection during the survey phase

[0310] During the survey phase, construction companies mainly collected three types of data: planning text data, planning graphic data and survey results data.

[0311] Planning text data includes documents such as urban master plans, regional development plans, and project concept plans. These documents record key information such as the project's positioning, functions, and layout. In order to collect this text data, the company deployed a structured data collection agent. The agent can connect to the file server, periodically grab relevant files, and convert them into a standard text format.

[0312] At the same time, the company also collected a series of planning drawings, such as the overall planning map, phased planning map, land use nature map, etc. These graphic files reflect the layout and zoning of the project in the urban space. For this type of unstructured data, the company deployed an unstructured data collection agent. The agent can recognize common graphic file formats such as CAD and BIM and extract information such as geometry and attributes. It also retrieves survey data, including survey report number, foundation depth / base elevation, foundation design grade, foundation bearing capacity, foundation bearing layer rock and soil type, seasonal frozen ground depth, and groundwater corrosion.

[0313] During the data collection process, the service monitoring center The operating indicators of each collection microservice are collected in real time, such as CPU usage, memory usage, disk read and write speed, etc. Based on these monitoring data, the resource scheduling center Computing resources are dynamically allocated to ensure the stable execution of data collection tasks.

[0314] Table 1 Data collection tasks during the survey phase

[0315] Data Source Data Type Collection cycle Acquisition frequency File Server Planning text and design results weekly every day Graphics Server Planning graphics per month weekly

[0316] Data collection during the design phase

[0317] During the design phase, the company primarily collected three types of data: Building Information Modeling (BIM) data, design document data, and design outcome data. Design outcome data included building name, structural design life, structural type, foundation type, building structural safety rating, seismic fortification category, seismic fortification intensity, shock absorption, seismic isolation, curtain wall type, whether the building was prefabricated, assembly rate, fire resistance rating, fire resistance rating (underground), and building insulation material type.

[0318] BIM data includes three-dimensional entity models, attribute information, relationship information, etc., which can fully describe the geometry, performance and other characteristics of the building. For BIM data, the company has deployed a structured data collection agent. The agent can connect to the BIM server, regularly capture BIM model files, and perform semantic analysis on them.

[0319] Design document data includes professional design drawings, calculation books, construction organization design and other documents, which record the details of the design plan. For this type of unstructured data mixed with text and graphics, the company also deployed a structured data collection agent and unstructured data collection agents Able to extract key information from design documents.

[0320] During the data collection process, the service monitoring center Keep an eye on the running status of each collection microservice. Once the CPU usage of a microservice exceeds 85% or the response time exceeds 500ms, the resource scheduling center will be triggered. Dynamic expansion is performed. This elastic scaling mechanism ensures efficient and stable data collection during the design phase.

[0321] Table 2 Data collection tasks in the design phase

[0322] Data Source Data Type Collection cycle Acquisition frequency BIM Server BIM Model per month weekly File Server Design Documentation weekly every day

[0323] Data collection during construction phase

[0324] During the construction phase, the company mainly collected two types of data: construction process data and quality inspection data.

[0325] Construction process data includes construction progress, construction technology, mechanical equipment parameters, etc., reflecting the dynamic situation of engineering construction. Most of this data is stored in various professional management systems in a structured form. For this type of data, the company has deployed a structured data collection agent. Able to connect to the construction management system and regularly capture relevant data.

[0326] Quality inspection data includes material inspection reports, sub-project acceptance records, safety inspection records, etc., which record the actual measurement of project quality. Most of this data exists in the form of unstructured text. To this end, the company has deployed an unstructured data collection agent. Able to extract key information from quality inspection documents.

[0327] During the data collection process, the service monitoring center Keep an eye on the resource usage of each collection microservice. When the disk read / write rate of a microservice exceeds 100MB / s or the network transmission rate exceeds 50Mbps, the resource scheduling center Storage and network resources will be dynamically increased to avoid resource bottlenecks affecting data collection quality.

[0328] Table 3 Data collection tasks during the construction phase

[0329]

[0330]

[0331] Data collection during the operation and maintenance phase

[0332] During the operation and maintenance stage, enterprises mainly collected two types of data: equipment operation data and maintenance record data.

[0333] Equipment operation data includes the usage parameters, fault information, energy consumption data of various electromechanical equipment, etc., reflecting the real-time operation status of the building. For this type of structured data, the enterprise has deployed a structured data collection agent. Able to connect to the building equipment monitoring system and capture relevant data regularly.

[0334] Maintenance record data includes daily inspection reports, equipment maintenance records, quality inspection reports, etc., which record the maintenance status of the building. Most of this data exists in the form of unstructured text. To this end, the company deployed an unstructured data collection agent Extract key information from maintenance record documents.

[0335] During the data collection process, the service monitoring center Continuously track the service response time of each collection microservice. When the response time of a microservice exceeds 1 second, the resource scheduling center It will dynamically adjust resource allocation, improve service processing capabilities, and ensure the timeliness of operation and maintenance data collection.

[0336] Table 4 Data collection tasks in the operation and maintenance phase

[0337] Data Source Data Type Collection cycle Acquisition frequency Equipment monitoring system Equipment operation per hour per minute File Server Maintenance records per month weekly

[0338] Data preprocessing and standardization

[0339] Through the above collection, the company has accumulated a massive amount of heterogeneous data from survey to operation and maintenance. To further process this data, the company adopted the data preprocessing and standardization method proposed in this invention.

[0340] First, the company cleaned the structured data. Missing values ​​were filled with the mean μ, and data with varying formats was converted to CSV format. Furthermore, the company used a fingerprint-based approach to identify and remove duplicate data. Furthermore, the company processed outliers in the structured data, removing any points that deviated from the μ ± 3σ range.

[0341] For unstructured data, the company first performs text segmentation operations, dividing text data such as planning texts, design documents, and quality inspection reports into word sets. The company then extracted the TF-IDF features F of these tokens, reflecting the importance of the words in the text. Finally, the company used a pre-built knowledge graph to semantically annotate the unstructured data, enhancing the data's semantic expression capabilities.

[0342] After the above preprocessing, the enterprise converts the original building life cycle data into a structured first preprocessing data set

[0343] Next, the company built a unified data standard conversion model Used to Further transformed into the first standardized dataset The model includes:

[0344] 1) Structured data conversion rules Including field mapping relationship M and data type conversion rules T;

[0345] 2) Unstructured data conversion rules Including text semantic mapping and feature vector conversion;

[0346] 3) From The extracted data standard updates parameters Θ, such as field coverage, data type distribution, etc.

[0347] According to the above rules, the enterprise automatically completes arrive At the same time, the company also uses Θ to continuously optimize the data standard conversion model Improved the effectiveness of data standardization.

[0348] Through the above preprocessing and standardization, the company has built a unified building data asset for the entire life cycle, laying the foundation for subsequent in-depth analysis and integrated applications.

[0349] Data association analysis

[0350] Based on the standardized dataset The company carried out data association analysis with the aim of discovering the intrinsic connections between data elements and providing a basis for subsequent data fusion.

[0351] First, the enterprise built a data feature mapping model It includes feature decomposition module and feature reconstruction module. The feature decomposition module uses CP decomposition to Mapping to feature space:

[0352]

[0353] Among them, λ r is the weight of the rth component, a r ,b r ,c r is the decomposition vector, is the outer product operation, and ε is the residual tensor. In this way, we get the eigenvector set of the data

[0354] Enterprise based on Constructed a feature vector index library And the cosine similarity S between each feature vector is calculated:

[0355]

[0356] Will and S as output, generating the first feature retrieval result

[0357] Next, the company built a deep neural network model The multi-head attention mechanism is used to analyze the correlation strength between data elements. The query matrix Q, key matrix K and value matrix V are calculated as follows:

[0358] Q=XW Q ,K=XW K ,V=XW V ;

[0359] Among them, X is the input feature matrix, W Q ,W K ,W V is the parameter matrix. The calculation formula of the attention weight matrix A is:

[0360]

[0361] Z=Concat(head1,…,head h )W O ;

[0362] in,

[0363] Will enter The model finally generates the first correlation strength matrix A (1) Based on A (1) , the company has built a detailed data lineage map This laid the foundation for subsequent data fusion.

[0364] Data coupling analysis

[0365] In order to determine the scope and strategy of data fusion, the enterprise adopted a data coupling analysis method based on graph theory.

[0366] First, the enterprise will map the data lineage The first data association matrix C converted into the adjacency matrix form (1) Then, the enterprise constructed the Laplace matrix L = DX (1) , where D is the degree matrix, Next, the enterprise performs eigendecomposition on L to obtain the eigenvalue λ and eigenvector v. Based on v, the enterprise divides the data element nodes into different clusters.

[0367] Based on the results of spectral clustering, the enterprise calculated the coupling degree C between data elements:

[0368] c ij =γ1s ij +γ2t ij +γ3p ij +∈3;

[0369] Among them, s ij is the feature similarity, t ij is the topological distance, p ij is the attribute correlation. γ1,γ2,γ3 are the weight coefficients of the corresponding indicators, and ∈3 is the error term.

[0370] By setting the coupling threshold θ c =0.8, the enterprise determined the scope of data fusion and generated the first fusion boundary result This provides an important basis for subsequent data fusion.

[0371] Data fusion and quality control

[0372] Based on the above analysis results, the enterprise built an adaptive data fusion system.

[0373] First, the enterprise based on the first fusion boundary results The fusion priority P of each data element is calculated:

[0374] P i =0.4·C i +0.3 U i +0.3·E i +0.2;

[0375] Among them, C i is the data coupling score, U i is the data update frequency, E i The importance of the data.

[0376] Next, the enterprise configured the data fusion service cluster Including structured data fusion services and unstructured data fusion services According to the fusion priority P and fusion execution plan (including alignment rules Conflict resolution rules and consistency check rules ), enterprise call The data fusion operation was performed to generate the first fusion result dataset

[0377] In order to evaluate the quality of fusion results, the company has established a fusion quality assessment system The system includes the following indicators:

[0378] 1) Integrity Index CI:

[0379]

[0380] Among them, δ i is the field integrity indicator function, w i is the field weight.

[0381] 2)Consistency index CON:

[0382]

[0383] Among them, x ij ,y ij is the data element to be compared, and MAX is the normalization coefficient.

[0384] 3) Accuracy index ACC:

[0385]

[0386] Among them, TP, TN, FP, and FN are the number of true positives, true negatives, false positives, and false negatives, respectively.

[0387] 4) Timeliness index T:

[0388] T=exp(-λ(t c -t u ));

[0389] Among them, t c is the current time, t u is the data update time, and λ is the timeliness attenuation coefficient.

[0390] According to the above indicators, enterprises An assessment was conducted and the first quality assessment report was generated The report showed that the completeness of the fusion results was 92%, the consistency was 85%, the accuracy was 90%, and the timeliness was 93%. The overall quality score was 91 points.

[0391] At the same time, the report also gives some optimization suggestions, such as further improving data alignment rules, optimizing conflict handling algorithms, etc. Updated the data fusion service cluster parameter configuration to continuously improve the fusion effect.

[0392] In addition, the company has also established a data fusion anomaly detection mechanism and a data fusion rollback mechanism to ensure the reliability of the fusion process. Once a problem with data consistency or integrity is found, the system will trigger a rollback operation to restore the data to the previous stable state. The company also records the execution log of data fusion. Provide a basis for subsequent process optimization.

[0393] Through the above-mentioned fusion quality control measures, the company has built a highly adaptive data fusion system to ensure the reliability and practicality of the fusion results.

[0394] It should be noted that the variables involved in the description of the present invention are explained in detail as shown in Table 5 below.

[0395] Table 5 Variable explanation table

[0396]

[0397]

[0398] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A method for integrating data throughout the entire life cycle of a building, characterized by: The following steps are involved: S01. Build an adaptive multi-channel data acquisition microservice system, which includes a data acquisition microservice cluster, a service registration center, a service monitoring center, and a resource scheduling center. The data acquisition microservice cluster is configured with different data acquisition agents for different data source types, and the data acquisition agents include structured data acquisition agents and unstructured data acquisition agents. S02. Collect microservice operation indicators from the service monitoring center, where the microservice operation indicators include service response time, service throughput, and resource utilization, and generate a first performance measurement result. The resource scheduling center calculates the microservice resource demand based on the first performance measurement result. Allocate resources to the data acquisition microservice cluster according to the microservice resource demand, and obtain a first resource allocation result; S03. Obtaining data collection tasks corresponding to the first resource allocation result, the data collection tasks including survey phase data collection tasks, design phase data collection tasks, construction phase data collection tasks, and operation and maintenance phase data collection tasks; collecting building life cycle data according to the data collection tasks, the building life cycle data including survey phase data, design phase data, construction phase data, and operation and maintenance phase data; S04. Construct a first service dependency model based on the building lifecycle data. The first service dependency model calculates the microservice dependency level based on the input data and output data of each microservice in the data acquisition microservice cluster to generate a first dependency relationship result. Generate a data acquisition microservice deployment sequence according to the first dependency result; S05. Preprocessing the building life cycle data, performing data cleaning operations, duplicate data removal operations, and abnormal data processing operations on structured data in the building life cycle data, and performing text segmentation operations, feature extraction operations, and semantic annotation operations on unstructured data in the building life cycle data to generate a first preprocessed data set; S06. Construct a data standard conversion model, the data standard conversion model including structured data conversion rules and unstructured data conversion rules; convert the first preprocessed data set into a first standardized data set according to the data standard conversion model; extract data standard update parameters from the first standardized data set, and update the data standard conversion model according to the data standard update parameters; S07. Construct a data feature mapping model, wherein the data feature mapping model uses a tensor decomposition operation to map the first standardized data set to a feature space to obtain a first feature vector set; establish a feature index library for the first feature vector set, and generate a first feature retrieval result; S08. Input the first feature retrieval result into a deep neural network model, wherein the deep neural network model uses a multi-head attention calculation unit to extract feature association strength to obtain a first association strength matrix; and establish a data lineage map based on the first association strength matrix; S09, converting the data lineage map into a first data association matrix, processing the first data association matrix using a spectral clustering algorithm, and calculating the data coupling degree; Determine the data fusion range according to the data coupling degree, and generate a first fusion boundary result; S10. Build a fusion priority model based on the first fusion boundary result, generate a data fusion execution plan based on the fusion priority model, and include data alignment rules, conflict handling rules, and consistency verification rules; and establish a data fusion fallback mechanism that triggers a data fallback operation based on fusion exception information. S11. Configuring a data fusion service cluster, where the data fusion service cluster includes a structured data fusion service and an unstructured data fusion service; Calling the data fusion service cluster to perform data fusion according to the data fusion execution plan to generate first fusion result data; S12. Establish a fusion quality assessment system, which includes integrity assessment indicators, consistency assessment indicators, accuracy assessment indicators and timeliness assessment indicators; evaluate the first fusion result data according to the fusion quality assessment system to generate a first quality assessment report; update the service parameter configuration of the data fusion service cluster according to the first quality assessment report.

2. The method according to claim 1, characterized in that The step S01 specifically includes: Step 101: Build a service registration center to manage service registration and service discovery of the data acquisition microservice cluster; Step 102: Build a service monitoring center to collect the operating status data of the data collection microservice cluster; Step 103: construct a resource scheduling center for dynamically allocating computing resources according to the operating status data; Step 104: deploying a structured data collection agent for the structured data source, wherein the structured data collection agent is used to collect database data and file system data; Step 105: deploying an unstructured data acquisition agent for the unstructured data source, wherein the unstructured data acquisition agent is used to acquire image data, text data, and audio data; Step 106: Establish a microservice monitoring and alarm mechanism to trigger resource expansion and service migration based on the service operation status data.

3. The method according to claim 1, characterized in that The step S02 specifically includes: Step 201: Collect microservice operation indicators of the service monitoring center, wherein the microservice operation indicators include CPU occupancy rate, memory occupancy rate, disk read / write rate, and network transmission rate; Step 202: Calculate the service response time based on the microservice operation indicator. The service response time is the time interval from the initiation of the service request to the completion of the response. Step 203: Calculate the service throughput based on the microservice operation indicator, where the service throughput is the number of service requests completed per unit time. Step 204: Calculate resource utilization based on the microservice operation indicator, where the resource utilization is the ratio of used resources to total resources. Step 205: Generate a first performance measurement result according to the service response time, the service throughput, and the resource usage rate; Step 206: Calculate the microservice resource requirements based on the first performance measurement result and generate a first resource allocation result.

4. The method according to claim 1, wherein The step S03 specifically includes: Step 301: Obtain a data collection task corresponding to the first resource allocation result, wherein the data collection task includes a data source address, a collection period, and a sampling frequency; Step 302: Collect survey phase data, including planning text data, planning graphic data, and survey result data; Step 303: Collect design phase data, including building information model data, design document data, and design result data; Step 304: Collect construction phase data, including construction process data and quality inspection data; Step 305: Collect operation and maintenance data, including equipment operation data and maintenance record data; Step 306: Classify, store and index the collected building life cycle data.

5. The method according to claim 1, characterized in that The step S04 specifically includes: Step 401: Analyze the input data type and output data type of each microservice in the data acquisition microservice cluster; Step 402: Count the data transmission frequency between the microservices in the data collection microservice cluster; Step 403: Build a first service dependency model, where the first service dependency model includes service nodes and service relationships. Step 404: Calculate the microservice dependency based on the service node and the service relationship; Step 405: Generate a first dependency result, where the first dependency result includes dependency strength and dependency direction; Step 406: Generate a data acquisition microservice deployment sequence based on the first dependency result.

6. The method according to claim 1, characterized in that The step S05 specifically includes: Step 501: Perform data cleaning operations on structured data, including null value processing and format unification. Step 502: performing a deduplication operation on the structured data, wherein the deduplication operation is based on data fingerprint calculation; Step 503: performing an abnormal data processing operation on the structured data, wherein the abnormal data processing operation is based on statistical analysis; Step 504: Perform a text segmentation operation on the unstructured data, wherein the text segmentation operation is based on dictionary matching; Step 505: Perform feature extraction on the unstructured data, where the feature extraction is based on word frequency statistics. Step 506: Perform semantic annotation operations on unstructured data, where the semantic annotation operations are based on the knowledge graph.

7. The method according to claim 1, characterized in that The step S06 specifically includes: Step 601: Establish structured data conversion rules, which include field mapping relationships and data type conversion rules; Step 602: Establish unstructured data conversion rules, wherein the unstructured data conversion rules include text semantic mapping and feature vector conversion; Step 603: construct a data standard conversion model according to the structured data conversion rules and the unstructured data conversion rules; Step 604: Use the data standard conversion model to convert the first preprocessed data set to generate a first standardized data set; Step 605: extracting data standard update parameters from the first standardized data set; Step 606: Update the data standard conversion model according to the data standard update parameters.

8. The method according to claim 1, characterized in that The step S07 specifically includes: Step 701: construct a data feature mapping model, wherein the data feature mapping model includes a feature decomposition module and a feature reconstruction module; Step 702: Perform a tensor decomposition operation on the first standardized data set to extract data features; Step 703: Generate a first feature vector set according to the data features; Step 704: Establish a feature dimension index of the first feature vector set; Step 705: construct a feature retrieval structure based on the feature dimension index; Step 706: Generate a first feature retrieval result using the feature retrieval structure.

9. The method according to claim 1, characterized in that The step S08 specifically includes: Step 801: construct a deep neural network model, wherein the deep neural network model includes a feature encoding layer and a feature decoding layer; Step 802: Setting a multi-head attention calculation unit, wherein the multi-head attention calculation unit includes a query vector generation module, a key vector generation module, and a value vector generation module; Step 803: Input the first feature retrieval result into the deep neural network model for feature association analysis; Step 804: extracting feature correlation strength using the multi-head attention calculation unit; Step 805: Generate a first correlation strength matrix according to the feature correlation strength; Step 806: Establish a data lineage map based on the first association strength matrix.

10. The method according to claim 1, characterized in that The step S09 specifically includes: Step 901: Matrix the data lineage map to generate a first data association matrix; Step 902: Analyze the first data association matrix using a spectral clustering algorithm; Step 903: Calculate the data coupling degree between the nodes in the first data association matrix; Step 904: Set a data coupling threshold, and determine a data fusion range based on the data coupling threshold; Step 905: Generate a first fusion boundary result according to the data fusion range; Step 906: Verify and optimize the first fusion boundary result.

Images