Multi-source heterogeneous data fusion method, system and equipment for urban physical examination database

Abstract

The invention relates to the field of data fusion and processing, and discloses a multi-source heterogeneous data fusion method, system and equipment for an urban physical examination database, and the method comprises the following steps: S1, cleaning urban physical examination data from a plurality of heterogeneous data sources, and removing repeated data and wrong data; s2, extracting key features from the cleaned data; s3, the extracted key features are mapped to a unified data model, and the problem of inconsistent data formats is solved; s4, detecting and solving data conflicts of the same attribute in different data sources, and generating consistent data; s5, constructing an urban building safety evaluation model based on the consistency data, and analyzing potential safety hazards of the building; and S6, outputting an analysis result to a city management system to assist in decision making. Through an automatic identification and matching algorithm, urban physical examination data from different sources and in different formats are integrated into a unified data model, and the low-efficiency process of preprocessing and cleaning depending on manual rules in a traditional method is avoided.

Description

Technical Field

[0001] The present invention relates to the field of data fusion and processing, and in particular to a method, system and equipment for fusion of multi-source heterogeneous data of a city physical examination database. Background Art

[0002] In the process of urbanization, the establishment and application of urban health examination databases have become an important part of smart urban management. In existing technologies, urban health examination data usually comes from multiple heterogeneous data sources, such as building archives from urban planning departments, inspection reports from fire departments, road monitoring data from transportation departments, and real-time information collected by sensors and drones. These data differ in format, collection method, and storage method. For example, building archives may be structured tabular data, monitoring data may be time series numerical values, and drone imagery is unstructured image or video data. To achieve data integration and analysis, existing methods generally use data preprocessing, cleaning, and fusion techniques. Specifically, the data fusion process involves manually formulating rules to convert and clean the data format, and then processing the fused data through statistical analysis or machine learning models to support application scenarios such as urban building safety supervision and planning optimization.

[0003] However, existing technologies have significant drawbacks in the fusion of multi-source, heterogeneous data, primarily manifested in low automation and insufficient accuracy. Traditional methods rely on manually formulated rules to handle format differences and conflicts between different data sources. For example, it requires manual comparison of height data in building archives with measured data and the establishment of fusion rules for each. This approach is not only time-consuming—processing a city's building data can take weeks, for example—but also prone to errors due to subjective human judgment, such as overlooking updates from certain data sources or misjudging data priorities. Furthermore, due to the complexity and diversity of urban health check data, manual rules struggle to fully cover all scenarios. For example, they are unable to effectively handle inconsistencies between dynamic changes in drone imagery and static archival data, resulting in reduced accuracy of the fused data and potentially high data processing error rates exceeding 40%. These issues limit the efficiency and reliability of data analysis in urban management, making it difficult to meet the demands for rapid and accurate decision-making. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a multi-source heterogeneous data fusion method, system and equipment for urban physical examination databases, which solves the problem of low efficiency caused by traditional methods relying on manually formulated rules to handle format differences and conflicts between different data sources.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-source heterogeneous data fusion method for a city physical examination database comprises the following steps: S1. Clean the city physical examination data from multiple heterogeneous data sources to remove duplicate and erroneous data; S2, extract key features from the cleaned data; S3. Map the extracted key features to a unified data model to resolve data format inconsistencies. S4. Detect and resolve data conflicts of the same attribute in different data sources to generate consistent data; S5. Build an urban building safety assessment model based on consistent data to analyze building safety hazards; S6. Output the analysis results to the city management system to assist decision making.

[0006] Preferably, in step S2, the key features include one or more of building location, structure type, and fire escape conditions.

[0007] Preferably, in step S3, the unified data model is based on a geographic information system coordinate system, and standardizes the association of feature data through spatial positioning.

[0008] Preferably, in step S4, the conflict detection and resolution is performed by comparing the differences of the same attribute and automatically correcting them according to preset rules, wherein the preset rules include giving priority to the latest data or data from the most authoritative source.

[0009] Preferably, it also includes: S7. Before step S1, real-time data stream processing technology is used to receive and process real-time data streams from multiple data sources to ensure the immediacy of data fusion.

[0010] Preferably, in step S7, the real-time data stream processing technology includes: using a message queue to receive the real-time data stream, removing outliers through a real-time filter, and using a streaming computing framework to perform feature extraction and data mapping.

[0011] Preferably, it also includes: S8. Before step S1, the uploaded data is encrypted and attached with a timestamp and hash value through a trusted data sharing mechanism based on blockchain, and stored in the blockchain node to ensure the authenticity and integrity of the data source.

[0012] Preferably, the multi-source heterogeneous data fusion system of the urban physical examination database includes: Data collection module, used to collect urban physical examination data from multiple heterogeneous data sources; A data processing module, configured to execute steps S1 to S4 to generate consistency data; An analysis module is used to execute step S5 to construct an urban building safety assessment model and analyze building safety hazards; A result display module is used to execute step S6 and output the analysis results to the city management system in a visual manner; The modules interact with each other via a network communication interface.

[0013] Preferably, it also includes: A real-time processing module, configured to execute step S7, receiving and processing the real-time data stream using a real-time data stream processing technology; The blockchain module is used to execute step S8 and store the encrypted data using blockchain technology.

[0014] Preferably, an electronic device comprises: processor; a memory storing executable instructions; When the processor executes the executable instructions, the multi-source heterogeneous data fusion method of the city physical examination database is implemented.

[0015] The present invention provides a method, system, and device for fusion of multi-source heterogeneous data in a city physical examination database. This method has the following beneficial effects: 1. This invention integrates urban health examination data from different sources and formats (such as building archives, sensor monitoring data, and drone images) into a unified data model through automatic recognition and matching algorithms, avoiding the inefficient process of relying on manual rule preprocessing and cleaning in traditional methods.

[0016] 2. The present invention can effectively handle data format inconsistencies through feature extraction and data mapping steps, while the conflict detection and resolution mechanism automatically corrects differences through preset rules (such as prioritizing the latest data), reducing errors caused by human intervention.

[0017] 3. This invention introduces real-time data stream processing technology, using message queues (such as Apache Kafka) and streaming computing frameworks (such as Spark Streaming) to instantly process and fuse real-time data streams from sensors, drones, etc. Compared with traditional batch processing methods, the data update cycle is shortened from hours to seconds.

[0018] 4. The present invention adopts a trusted data sharing mechanism based on blockchain, which ensures the authenticity and integrity of multi-source data by encrypting, timestamping and storing data with hash values. The data uploaded by each data source is recorded in the distributed ledger. Even if a data source is tampered with or fails, other nodes can still verify the original data through consensus. This mechanism effectively reduces fusion errors caused by data forgery or loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the method for fusion of multi-source heterogeneous data of the city physical examination database of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Please see the attached Figure 1 The embodiment of the present invention provides a multi-source heterogeneous data fusion method for a city physical examination database, comprising the following steps: S1. Clean the city physical examination data from multiple heterogeneous data sources to remove duplicate and erroneous data; S2, extract key features from the cleaned data; S3. Map the extracted key features to a unified data model to resolve data format inconsistencies. S4. Detect and resolve data conflicts of the same attribute in different data sources to generate consistent data; S5. Build an urban building safety assessment model based on consistent data to analyze building safety hazards; S6. Output the analysis results to the city management system to assist decision making.

[0022] Specifically, urban health check databases contain a vast amount of data from various departments and devices, such as building archives from urban planning departments, inspection reports from fire departments, road monitoring data from transportation departments, and real-time imagery and environmental data collected by drones or sensors. These data differ significantly in format, acquisition time, and accuracy. For example, building archives may store static information in a table format, while drone imagery is a dynamic sequence of images. The cleaning process in step S1 specifically includes a data integrity check, such as checking for missing data fields (e.g., null building height or location coordinates), duplicate records (e.g., the same building is registered repeatedly in different archives), and the removal of outliers (e.g., negative building age records). After cleaning, step S2 uses natural language processing (NLP) techniques to extract semantic features from the text data, such as identifying descriptions like "fire escape occupied" in fire reports. Image recognition technology is also used to extract building outlines and illegal construction information from aerial images. The unified data model in step S3 is not limited to GIS but can also be extended to time series databases to support the spatiotemporal correlation of dynamic data, such as comparing a building's annual safety monitoring data with its real-time status. During conflict detection in step S4, a weighted scoring mechanism is employed. For example, each data source is assigned a credibility weight (e.g., government archives are weighted 0.8, while reports from residents are weighted 0.4). Conflicts are resolved through weighted averaging or voting. The evaluation model in step S5 can be a hybrid model based on a convolutional neural network (CNN) and a long short-term memory network (LSTM), capable of integrating spatial features and time series data to predict building collapse or fire risks. The output of step S6 includes an interactive map interface, annotating high-risk areas, and generating a detailed report containing the type of hazard, its urgency, and recommended measures, such as "Fire escape in a certain building is blocked; immediate clearance is recommended."

[0023] In step S2, the key features include one or more of building location, structure type, and fire escape conditions.

[0024] Specifically, extracting key features is a core step in data fusion. In urban health check scenarios, building locations can be represented by latitude and longitude coordinates or street addresses, such as "No. 123, XX Road" or "39.9042°N, 116.4074°E." Structural types, including residential, commercial, and industrial buildings, can be directly retrieved from categorized fields in building archives or inferred through image analysis, such as identifying window distribution patterns in high-rise buildings. Fire escape conditions include information such as width, occupancy (e.g., for parked vehicles or debris), and the presence of obstacles. This data may be derived from field measurements by fire departments or pixel analysis of drone imagery. In practice, feature extraction also includes secondary features, such as the fire resistance rating of building materials (concrete, steel, or wood), the number of windows (related to ventilation and escape capabilities), and the condition of the roof (whether water accumulation or damage is present). This extraction process may combine multiple algorithms, such as using the YOLO algorithm to detect objects in fire escapes from images, or using text mining techniques to extract keywords such as "leakage" and "cracks" from resident feedback. The diversity and comprehensiveness of these features ensure the accuracy of subsequent analysis, such as predicting the difficulty of fire evacuation based on the status of fire escape routes.

[0025] In step S3, the unified data model is based on the geographic information system coordinate system and performs standardized association of feature data through spatial positioning.

[0026] Specifically, geographic information system (GIS) coordinate systems not only provide two-dimensional coordinates (latitude and longitude) but also support three-dimensional modeling, such as recording building heights and topographic features (such as whether they are located on sloped land). In implementation, each data source is first assigned a unique spatial identifier, for example, by converting addresses in building archives into standardized GIS grid codes. Then, geographic data from different data sources is aligned using coordinate transformation algorithms (such as WGS84 to CGCS2000). For example, a "fire lane on a certain road section" in a fire report can be mapped to a specific line segment in GIS using the road name, while drone imagery can be directly located using image geographic metadata (such as EXIF information). Data mapping also incorporates the temporal dimension, such as serializing historical inspection data for a particular building according to timestamps to create a dynamic spatial-temporal distribution map. Furthermore, GIS models support multi-scale analysis, such as displaying fire lane conditions at the street level and building density distribution at the city level. The advantage of this mapping approach is that it not only resolves format differences but also facilitates subsequent spatial statistical analysis, such as calculating the concentration of high-risk buildings within a region.

[0027] In step S4, conflict detection and resolution are performed by comparing differences in the same attribute and automatically correcting them according to preset rules. The preset rules include giving priority to the latest data or the data from the most authoritative source.

[0028] For example, if the height of a building is recorded as 50 meters in archived data, while drone data shows it as 48 meters, the system will first detect this discrepancy and determine the most recent data by comparing timestamps (e.g., drone data collected in 2025, while archived data is from 2020). Pre-set rules can also include determining the authority of the data source, such as prioritizing official government records over resident reports; or assessing the accuracy of the data collection equipment, such as prioritizing laser rangefinder data over estimates taken from mobile phones. In complex situations, such as when multiple data sources simultaneously provide up-to-date but conflicting data (e.g., fire escape occupancy), the system will incorporate machine learning models to train conflict resolution strategies based on historical data, such as using support vector machines (SVMs) to determine the most likely correct value. Furthermore, after a conflict is resolved, the system will log each revision, including the source of the conflict, the rationale for the revision, and the outcome (e.g., "Fire escape width revised from 2 meters to 1.8 meters based on drone imagery, dated March 15, 2025"), to facilitate subsequent audits and algorithm optimization.

[0029] Also includes: S7. Before step S1, real-time data stream processing technology is used to receive and process real-time data streams from multiple data sources to ensure the immediacy of data fusion.

[0030] Specifically, in implementation, data sources may include sensor networks (such as stress sensors installed on buildings), traffic cameras, real-time video streams from drones, and instant feedback from residents via mobile apps. For example, when sensors detect an abnormally high stress on a building wall, the data is immediately transmitted to the system via the 5G network. Simultaneously, a nearby drone is triggered to capture updated images to verify any external damage. The specific process for step S7 involves: first, a high-throughput data pipeline is established using Apache Kafka, with the data streams from each data source assigned to independent topics (e.g., "building sensors" and "drone images"). Real-time filters then remove outliers based on statistical rules (e.g., mean ±3 standard deviations), such as sensor jumps caused by interference. Spark Streaming then performs windowed processing on the data stream (e.g., aggregating every 5 seconds), extracting features, and performing mapping. This real-time nature enables the system to rapidly update data and trigger alerts in the event of an emergency (e.g., building damage after an earthquake), for example, identifying damaged buildings and notifying emergency response departments within 10 seconds of an earthquake.

[0031] In step S7, the real-time data stream processing technology includes: using a message queue to receive the real-time data stream, removing outliers through a real-time filter, and using a stream computing framework to perform feature extraction and data mapping.

[0032] Specifically, the Apache Kafka message queue supports distributed deployment and can process millions of data records per second, such as simultaneously receiving monitoring data and high-definition video streams from thousands of sensors across the city. Real-time filters not only remove outliers but also identify missing data. For example, if a sensor has no signal for five consecutive seconds, it is automatically marked as faulty and maintenance personnel are notified. The streaming computing framework Spark Streaming supports multiple window types, such as sliding windows (updated every second) and tumbling windows (aggregated every minute), to meet the needs of different scenarios, such as short windows for fire monitoring and long windows for analyzing building aging trends. The feature extraction process also incorporates pre-trained models, such as using ResNet to extract building appearance features from video streams or using BERT to extract semantic information from resident feedback text. Data mapping is accelerated through in-memory computing, for example, caching GIS coordinates in a Redis database to reduce disk I / O latency. This efficient real-time processing mechanism significantly improves system response time, for example, reducing fire hazard detection time from minutes to seconds.

[0033] Also includes: S8. Before step S1, the uploaded data is encrypted and attached with a timestamp and hash value through a trusted data sharing mechanism based on blockchain, and stored in the blockchain node to ensure the authenticity and integrity of the data source.

[0034] Specifically, in implementation, each data source (such as the fire department or building management unit) is treated as a node in the blockchain network. When data is uploaded, a hash value is first generated using the SHA-256 algorithm. This hash value, combined with a timestamp (e.g., "2025-03-31 10:00:00"), forms a digital signature to ensure the data is tamper-proof. For example, after fire tunnel monitoring data is uploaded, its hash value is stored in the distributed ledger. Even if a node is hacked, other nodes can still verify the data's authenticity through consensus algorithms (e.g., PBFT). The encryption process utilizes asymmetric encryption techniques, such as the RSA algorithm, with the data provider using a private key for encryption and the recipient using a public key for decryption, ensuring secure transmission. Furthermore, smart contracts are used to automatically manage data access rights, for example, by limiting access to "real-time monitoring data to the fire department only," and recording the time and identity of each access. The advantage of this mechanism is that it not only prevents data falsification but also reduces the trust cost associated with human intervention, for example, by eliminating the need for third-party audits to verify data integrity.

[0035] The multi-source heterogeneous data fusion system of the urban physical examination database includes: Data collection module, used to collect urban physical examination data from multiple heterogeneous data sources; A data processing module, configured to execute steps S1 to S4 to generate consistency data; An analysis module is used to execute step S5 to construct an urban building safety assessment model and analyze building safety hazards; A result display module is used to execute step S6 and output the analysis results to the city management system in a visual manner; Among them, data exchange is carried out between modules through the network communication interface.

[0036] Specifically, the data acquisition module supports a variety of interfaces, such as receiving sensor data via the MQTT protocol, accessing government archives via HTTP API, and downloading drone imagery via FTP. The data processing module's implementation consists of multiple subunits: the preprocessing unit uses Python scripts to batch clean data, such as removing missing values using the Pandas library; the feature extraction unit combines OpenCV and TensorFlow to process image and text data, such as extracting building boundaries from aerial imagery; the mapping unit relies on GIS software (such as ArcGIS) for spatial alignment; and the conflict resolution unit runs conflict detection algorithms, such as K-means clustering to analyze data distribution and identify outliers. The analysis module includes multiple built-in machine learning models, such as online learning algorithms (such as incremental random forests) that support real-time prediction and dynamically adjust parameters based on new data. The results presentation module provides a web interface and mobile application, such as using ECharts to generate heat maps showing the distribution of high-risk buildings across the city or using push notifications to alert managers to emergency hazards in specific areas. Network communication between modules is based on RESTful APIs or WebSockets, ensuring low latency and high concurrency, such as handling thousands of simultaneous users.

[0037] Also includes: A real-time processing module, configured to execute step S7, receiving and processing the real-time data stream using a real-time data stream processing technology; The blockchain module is used to execute step S8 and store the encrypted data using blockchain technology.

[0038] Specifically, the real-time processing module's hardware support includes high-performance server clusters, such as GPU-equipped compute nodes, to accelerate Spark Streaming's stream processing. Its software implementation includes fault-tolerance mechanisms, such as automatically switching to a backup partition if a Kafka partition fails, ensuring data is not lost. The blockchain module relies on distributed storage technologies, such as IPFS (InterPlanetary File System), to store large data files (such as video streams), while only hash values are uploaded to the blockchain to reduce storage costs. In practical applications, such as monitoring the safety of a high-rise building, the real-time processing module updates sensor data every second and generates a risk score. Meanwhile, the blockchain module records the source and time of each data update, such as "Sensor ID: A123, Data: Stress value 50 MPa, Time: 2025-03-31 10:05:00, Hash: 0xabc123." The advantage of this dual-module collaboration is that it ensures real-time data availability while providing traceability and trustworthiness, allowing for rapid data retrieval during accident investigations.

[0039] An electronic device, comprising: processor; a memory storing executable instructions; Among them, when the processor executes the executable instructions, it implements the multi-source heterogeneous data fusion method of the urban physical examination database.

[0040] Specifically, the electronic device can be a server, edge computing node, or portable terminal deployed at a city management center. For example, when used as a server, the processor is a multi-core CPU (such as an Intel Xeon) or a GPU (such as an NVIDIA A100), supporting parallel processing of large data streams. Storage includes an SSD (solid-state drive) and RAM (memory), with capacities of up to 1TB and 128GB, respectively, to store Kafka queue data and temporary computation results. When used as an edge computing node, the device may be an embedded device installed on-site (such as a Raspberry Pi), synchronizing data with the cloud via a 4G / 5G network. The executable instructions in the memory are written in Python or Java, and may include, for example, data cleaning scripts, feature extraction modules, and model training code. A specific example of the execution process is: after receiving sensor data from a building, the processor runs a cleaning algorithm to remove noise, extracts features, and maps them to a GIS model. This generates a safety score, which is then transmitted over the network to the management center. This flexibility makes the device suitable for a variety of scenarios, such as deploying edge devices in remote areas or running high-performance servers in data centers.

[0041] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source heterogeneous data fusion method for urban physical examination database, characterized by: The following steps are involved: S1. Clean the city physical examination data from multiple heterogeneous data sources to remove duplicate and erroneous data; S2, extract key features from the cleaned data; S3. Map the extracted key features to a unified data model to resolve data format inconsistencies. S4. Detect and resolve data conflicts of the same attribute in different data sources to generate consistent data; S5. Build an urban building safety assessment model based on consistent data to analyze building safety hazards; S6. Output the analysis results to the city management system to assist decision making.

2. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 1 is characterized in that: In step S2, the key features include one or more of building location, structure type, and fire escape conditions.

3. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 1 is characterized in that: In step S3, the unified data model is based on a geographic information system coordinate system and performs standardized association of feature data through spatial positioning.

4. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 1 is characterized in that: In step S4, the conflict detection and resolution is performed by comparing the differences of the same attribute and automatically correcting them according to preset rules. The preset rules include giving priority to the latest data or the data from the most authoritative source.

5. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 1 is characterized in that: Also includes: S7. Before step S1, real-time data stream processing technology is used to receive and process real-time data streams from multiple data sources to ensure the immediacy of data fusion.

6. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 5 is characterized in that: In step S7, the real-time data stream processing technology includes: using a message queue to receive the real-time data stream, removing outliers through a real-time filter, and using a stream computing framework to perform feature extraction and data mapping.

7. The multi-source heterogeneous data fusion method of the urban physical examination database according to claim 1 is characterized in that: Also includes: S8. Before step S1, the uploaded data is encrypted and attached with a timestamp and hash value through a trusted data sharing mechanism based on blockchain, and stored in the blockchain node to ensure the authenticity and integrity of the data source.

8. The multi-source heterogeneous data fusion system of the city physical examination database is characterized by: The multi-source heterogeneous data fusion method for the urban physical examination database according to any one of claims 1 to 7 comprises: Data collection module, used to collect urban physical examination data from multiple heterogeneous data sources; A data processing module, configured to execute steps S1 to S4 to generate consistency data; An analysis module is used to execute step S5 to construct an urban building safety assessment model and analyze building safety hazards; A result display module is used to execute step S6 and output the analysis results to the city management system in a visual manner; The modules interact with each other via a network communication interface.

9. The multi-source heterogeneous data fusion system for urban physical examination database according to claim 8 is characterized in that: Also includes: A real-time processing module, configured to execute step S7, receiving and processing the real-time data stream using a real-time data stream processing technology; The blockchain module is used to execute step S8 and store the encrypted data using blockchain technology.

10. An electronic device, characterized in that: The multi-source heterogeneous data fusion method for the urban physical examination database according to any one of claims 1 to 7 comprises: processor; a memory storing executable instructions; When the processor executes the executable instructions, the multi-source heterogeneous data fusion method of the city physical examination database is implemented.

Images