Building surveying and mapping information service big data processing system

Abstract

The invention discloses a building surveying and mapping information service big data processing system, and belongs to the technical field of data processing. The method and the device are used for solving the technical problems of poor data processing precision and poor system robustness in the existing scheme. According to the method, spatial-temporal correlation analysis and an improved isolated forest algorithm are fused, abnormal data are corrected, an output standardized data sequence is processed to reduce data analysis time consumption of a downstream module, distributed parallel processing, an attention mechanism and a CNN-LSTM hybrid model are combined, a dynamic weight distribution module is used for implementing feature dimension reduction, and the dynamic weight distribution module is used for implementing feature dimension reduction. The node computing power utilization rate and the network delay are monitored in real time, and a task allocation strategy is dynamically adjusted to realize resource elastic expansion; through a dynamic fault detection and hierarchical recovery strategy, various faults from mild network jitter to node hostile attacks can be handled, and the robustness in a complex industrial environment is significantly superior to that of a traditional master-slave backup architecture.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a big data processing system for building surveying information services. Background Technology

[0002] Building surveying and mapping information services typically encompass a range of specialized services designed to support building design, construction, and maintenance through precise data acquisition, processing, and analysis. Common services include topographic surveying, cadastral surveying, existing building surveying, engineering surveying, 3D laser scanning, Geographic Information System (GIS) services, as-built surveying, deformation monitoring, and more.

[0003] Existing technical solutions often employ a single threshold to filter outliers in multi-source data preprocessing, ignoring the spatiotemporal correlation of data, resulting in poor noise suppression. Feature extraction relies on CNN-LSTM models with fixed weights, which cannot dynamically adjust feature fusion strategies based on noise intensity. Resource scheduling depends on manual experience or simple load balancing algorithms, making it difficult to cope with tidal computing power demands. Data consistency is mainly achieved through local log auditing, which suffers from easy tampering and poor traceability.

[0004] Therefore, there is an urgent need for a full-chain solution that integrates robust preprocessing, anti-interference feature extraction, elastic resource scheduling, and adaptive fault tolerance to improve the reliability, processing efficiency, and system stability of building surveying data. Summary of the Invention

[0005] The purpose of this invention is to provide a big data processing system for building surveying information services, which solves the technical problems of poor data processing accuracy and system robustness in existing solutions.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A big data processing system for building surveying information services includes a robust preprocessing module for multi-source heterogeneous data: collecting multi-source heterogeneous data in the field of building surveying, identifying and correcting abnormal data through an anomaly detection mechanism that combines spatiotemporal correlation analysis and an improved isolated forest algorithm, and generating standardized data sequences;

[0008] Anti-interference feature enhancement extraction module: Based on the attention mechanism-optimized CNN-LSTM hybrid model, features are extracted from the standardized data sequence, and a dynamic weight allocation module is introduced to enhance the feature representation ability of noisy data;

[0009] Distributed computing resource elastic scheduling module: Constructs a load prediction model based on reinforcement learning, monitors node computing power utilization and network latency in real time, and dynamically adjusts task allocation strategies to achieve elastic scaling of resources;

[0010] Adaptive fault tolerance and data consistency guarantee module: It uses consortium blockchain technology to record the entire data processing log and combines it with a multi-replica asynchronous replication mechanism. When a node failure is detected, data recovery is triggered through the Byzantine fault tolerance algorithm.

[0011] Preferably, when performing spatiotemporal correlation analysis, a spatiotemporal correlation score matrix S is constructed to quantify the consistency between data points and the surrounding spatiotemporal environment: S = α·S T +(1-α)·S S In the formula, S T S represents the time relevance score. S α represents the spatial relevance score; α is the spatiotemporal weighting coefficient.

[0012] Preferably, when implementing improved isolated forest anomaly detection, the focus is on local microscopic anomalies: In the formula, IF(x) is the anomaly score, ranging from [0,1]; h(x) is the path length of data point x in the isolated tree; E[h(x)] is the average path length; and c(n) is the correction function. γ is Euler's constant, and n is the sample size.

[0013] Preferably, when implementing the fusion anomaly judgment and correction, the spatiotemporal correlation score S and the isolated forest anomaly score IF are fused, and the comprehensive anomaly score D is calculated using the formula: D=ω×(1-S)+(1-ω)×IF; where ω is the fusion weight; when the calculated comprehensive anomaly score D is greater than or equal to the anomaly judgment threshold, the corresponding data is marked as anomaly data.

[0014] Preferably, when the dynamic weight allocation module is working, it dynamically adjusts the fusion weights of spatial and temporal features based on the noise intensity to enhance the feature representation of the noise data;

[0015] When calculating the fusion weights, the formula is as follows: In the formula, F fusion β is the fusion feature; β is the spatial feature weight, with a value range of [0,1], and the temporal feature weight is 1-β; β=σ′(γ′×∈+θ); where σ′ is the sigmoid function, γ′ is the sensitivity coefficient, and θ is the bias term;

[0016] The fused feature F is achieved through a fully connected layer. fusion Dimensionality reduction to the target dimension K:

[0017] F final =ReLU(W fc F fusion +b fc );

[0018] L2 normalization is applied to the output features to ensure consistent feature scale across different samples:

[0019]

[0020] Preferably, a load prediction model is constructed, comprising a state space S and a reward function R;

[0021] The state space S is: S = [U cpu U mem D net Q task ,T time In the formula, U cpu This represents the CPU utilization of the current node; U mem For memory utilization; D net For network latency; Q task Task queue length; T time It is a time-related feature;

[0022] The reward function R is: R = a·(1-|U target -U pred |)-b·D pred -c·C scale In the formula, U target Target resource utilization rate; U pred To predict utilization rate; D pred To predict network latency; C scale The cost of resource scaling; a, b, and c are all weighting coefficients.

[0023] Preferably, resource elastic adjustment is performed based on the task allocation results and the resource demand increment ΔR:

[0024] When the predicted resource demand ΔR>R threshold When the duration is greater than 30 seconds, expansion is triggered; where R threshold =0.8×R current That is, 80% of the current resources;

[0025] When expanding capacity, idle nodes will be used first; otherwise, cloud server instances will be created automatically.

[0026] When the average node utilization rate And if the duration is greater than 5 minutes, shrinkage is triggered;

[0027] When scaling down, a migration-then-shutdown process is adopted, migrating tasks to nodes with lower loads and shutting down idle nodes after ensuring no task interruption.

[0028] Preferably, when processing end-to-end logs, a log collection module is embedded in key nodes of data preprocessing, feature extraction, and distributed computing to achieve automated on-chain processing.

[0029] Preferably, when a severe fault is detected, a PBFT-based Byzantine fault tolerance mechanism is activated to ensure data consistency and service continuity.

[0030] Preferably, when performing data recovery, in the event of a primary replica failure, a new primary replica is elected from the surviving replicas, prioritizing the replica with the least synchronization latency, and the integrity of the replica data is verified by outputting data hashes from the on-chain logs;

[0031] In case of a replica failure, a new replica is automatically started to synchronize all data and incremental logs from the primary replica.

[0032] Compared to existing solutions, the beneficial effects achieved by this invention are:

[0033] This invention, by integrating spatiotemporal correlation analysis and an improved isolated forest algorithm, can effectively reduce the false detection rate of traditional single methods and significantly improve the accuracy of identifying sudden noise and gradual drift in architectural surveying. By correcting abnormal data, the reliability of subsequent decisions is ensured. The standardized data sequence output reduces the data parsing time of downstream modules, and combined with distributed parallel processing, it can reduce the latency of multi-source data access, thereby meeting the real-time preprocessing requirements of architectural surveying big data. The dynamic weight allocation module effectively improves the feature extraction accuracy in high-noise scenarios. The combination of attention mechanism and CNN-LSTM hybrid model effectively reduces redundant features. Feature dimensionality reduction through the dynamic weight allocation module reduces the computational load of subsequent modules and improves overall processing efficiency, achieving a dual optimization of accuracy and efficiency.

[0034] By monitoring node computing power utilization and network latency in real time, and dynamically adjusting task allocation strategies, resource elastic scaling can be achieved. Compared with static scheduling strategies, resource utilization can be effectively improved. The combination of dynamic load prediction and priority scheduling can effectively reduce the average completion latency of high-priority tasks and meet the real-time requirements of business. The combination of real-time monitoring and rapid elastic scaling can effectively improve the stability of the system under sudden loads.

[0035] By combining consortium blockchain with digital signatures, the tampering rate of data processing logs is reduced to 0, meeting the audit and compliance requirements of architectural surveying data. Through the combination of multi-replica asynchronous replication and Byzantine fault tolerance, service recovery time in the event of node failure can be effectively reduced, avoiding surveying task interruptions caused by single points of failure. Through PBFT consensus and hash verification mechanisms, data consistency of distributed nodes is ensured, and the deviation rate of replica data can be effectively controlled, meeting the stringent requirements of architectural surveying for data accuracy. Through dynamic fault detection and hierarchical recovery strategies, various faults ranging from minor network jitter to malicious node attacks can be dealt with, and the robustness in complex industrial environments is significantly better than that of traditional master-slave backup architectures. Attached Figure Description

[0036] The invention will now be further described with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating the operation of a big data processing system for building surveying information services according to the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, this invention is a big data processing system for architectural surveying information services, comprising: a robust preprocessing module for multi-source heterogeneous data: collecting multi-source heterogeneous data in the field of architectural surveying, identifying and correcting abnormal data through an anomaly detection mechanism that integrates spatiotemporal correlation analysis and an improved isolated forest algorithm, and generating standardized data sequences; the specific steps include:

[0040] Access to multi-source data in the field of architectural surveying is achieved through standardized interfaces, such as OGC API, FTP, and MQTT protocol.

[0041] Multi-source data includes, but is not limited to, point cloud data, image data, structured data, and unstructured text;

[0042] Point cloud data (.las format): A set of three-dimensional coordinate points collected by lidar, including X, Y, Z coordinates and reflection intensity;

[0043] Image data (.tif / .jpg format): UAV oblique photography images, satellite remote sensing images;

[0044] Structured data (.csv / .json format): GNSS positioning data (latitude, longitude, elevation), BIM model parameters (component dimensions, material properties);

[0045] Unstructured text: surveying reports, construction logs, etc.

[0046] After parsing, the data is stored in a distributed file system, and an index directory is built according to the format of data type-collection time-device ID;

[0047] Preprocessing of multi-source heterogeneous data includes, but is not limited to, noise reduction filtering, outlier handling, and format standardization. These are existing conventional technical solutions, and the specific implementation steps will not be elaborated here.

[0048] When performing spatiotemporal correlation analysis, a spatiotemporal correlation score matrix S is constructed to quantify the consistency between data points and their surrounding spatiotemporal environment: S = α·S T +(1-α)·S S In the formula, S T To score the temporal correlation, the correlation between data points at adjacent time points is calculated based on the Pearson correlation coefficient.

[0049] x t Data for the current moment; x t-1 Data from the previous time step; σ is the standard deviation; S S To score spatial relevance, the similarity of neighboring region data is calculated based on Euclidean distance: S S =exp(-β·d ij );d ij β is the spatial distance between data points i and j, β is the spatial sensitivity coefficient with a default value of 0.8, and α is the spatiotemporal weighting coefficient, which can be dynamically adjusted according to the data type. For static data, such as BIM models, α = 0.3; for dynamic data, such as construction progress monitoring, α = 0.7.

[0050] It should be explained that by identifying spatiotemporal consistency anomalies from a macroscopic perspective, such as the sudden appearance of elevation data in a certain area that does not conform to historical trends, prior information can be provided for subsequent fine-grained anomaly detection.

[0051] When implementing improved isolated forest anomaly detection, the traditional isolated forest (IF) algorithm is optimized for local micro-anomalies, such as sensor noise: In the formula, IF(x) is the anomaly score, ranging from [0,1], with a higher probability of anomaly the closer it is to 1; h(x) is the path length of data point x in the isolated tree; E[h(x)] is the average path length; and c(n) is the correction function. γ is Euler's constant, and n is the sample size;

[0052] It should be noted that by introducing an adaptive tree depth adjustment mechanism, the tree depth is increased to 15 layers in high curvature regions (the default is 10 layers), which can effectively improve the sensitivity to local anomalies; high curvature regions, such as point clouds at the corners of buildings;

[0053] Furthermore, by capturing local anomalies from the perspective of microscopic data distribution, the blind spot of spatiotemporal correlation analysis in detecting isolated noise points can be compensated.

[0054] When implementing the fusion anomaly judgment and correction, the spatiotemporal correlation score S and the isolated forest anomaly score IF are fused, and the comprehensive anomaly score D is calculated by the formula: D=ω×(1-S)+(1-ω)×IF; where ω is the fusion weight, which is dynamically learned through the validation set, with an initial value of 0.6, and spatiotemporal correlation is trusted first.

[0055] Set the anomaly detection threshold Y, Y = μ + kσ, where μ is the mean of the historical comprehensive anomaly scores D, σ is the standard deviation, and k is the safety factor, with a default value of 3.

[0056] When the calculated comprehensive anomaly score D≥Y, the corresponding data is marked as anomaly data;

[0057] When correcting outlier data, if the outlier data is numerical data, the spatiotemporal weighted KNN algorithm described in Section 1.2 is used to refill it.

[0058] If the abnormal data is image data, it is repaired by generative adversarial network (GAN), which generates the missing region with the texture of neighboring pixels as input;

[0059] If the abnormal data is text data, then missing fields are filled in based on context prediction using the BERT model;

[0060] It should be explained that by achieving dual anomaly detection of macro trends and micro distributions through multi-dimensional fusion, and combining this with targeted correction strategies to restore data authenticity, reliable input can be provided for downstream feature extraction.

[0061] The corrected data is reorganized into a standardized data sequence according to the data type-spatiotemporal coordinate-feature dimension format;

[0062] Specifically, for structured data, it is converted to JSON-LD format and semantic tags are added; for example, "@type":"BuildingPointCloud";

[0063] For unstructured data, extract feature vectors, such as SIFT features of images and Word2Vec vectors of text;

[0064] It is uniformly stored in a time-series database, supporting fast queries by time range + spatial region format.

[0065] It should be explained that by processing and outputting standardized data sequences, the processing complexity of downstream modules can be reduced, such as feature extraction and distributed computing, and unified scheduling of multi-source data can be achieved.

[0066] In this embodiment of the invention, by integrating spatiotemporal correlation analysis with an improved isolated forest algorithm, the false positive rate of anomalies in traditional single methods can be effectively reduced, and the accuracy of identifying sudden noise and gradual drift in architectural surveying can be significantly improved. By correcting abnormal data, the error in repairing missing data can be controlled, avoiding the butterfly effect caused by outlier propagation and ensuring the reliability of subsequent decisions. The standardized data sequence output reduces the data parsing time of downstream modules, and combined with distributed parallel processing, the latency of multi-source data access can be reduced, thereby meeting the real-time preprocessing requirements of architectural surveying big data.

[0067] Anti-interference feature enhancement extraction module: Based on the attention mechanism-optimized CNN-LSTM hybrid model, features are extracted from the standardized data sequence. A dynamic weight allocation module is introduced to enhance the feature representation capability of noisy data. The specific steps include:

[0068] The preprocessed, standardized data sequence is converted into a model input tensor, specifically including:

[0069] For spatial data, such as LiDAR point clouds and image pixels: convert to a 3D tensor X. spatial ∈R H×W×C Where H is the height, W is the width, and C is the number of channels;

[0070] For time-series data, such as GNSS positioning sequences and sensor sample values: convert to a two-dimensional tensor X. temporal ∈R T ×D′ Where T is the time step, with a default value of 20, and D′ is the feature dimension;

[0071] For the fused input tensor: the model input X = [X] is generated through spatiotemporal splicing. spatial ,X temporal ], and add a noise intensity marker ∈, ∈ = D / T, with a value range of [0,1]. The larger ∈ is, the stronger the noise;

[0072] It should be explained that by unifying multi-source heterogeneous data into a tensor format that the model can process, structured input can be provided for feature extraction, and noise intensity labeling can guide subsequent weight allocation.

[0073] When extracting features from standardized data sequences, an improved residual network (ResNet) is used to extract spatial features. The specific structure includes:

[0074] Convolutional layers: Use 3×3 convolutional kernels, the number of which can be dynamically adjusted. The first layer has 64 kernels, which double with each downsampling. The stride is 1, and the padding is 1. The calculation formula is: F conv =σ′(W*X spatial+b); where W is the kernel weight, * is the convolution operation, b is the bias, and σ′ is the LeakyReLU activation function with a slope of 0.1;

[0075] Batch normalize the convolution output and add residual connections F. res =F conv +X spatial This alleviates the gradient vanishing problem;

[0076] Spatial feature vector F is generated using global average pooling (GAP). s ∈R 1×S S represents the spatial feature dimension, with a default value of 512.

[0077] It should be explained that by extracting local detailed features from architectural mapping data from the spatial dimension, such as building outlines and texture structures, a foundation can be provided for subsequent temporal feature fusion.

[0078] The temporal features are extracted using bidirectional LSTM (Bi-LSTM), and the processing steps are as follows:

[0079] Translate the temporal tensor X temporal The sequence is divided into subsequences based on time step T, and each subsequence contains features from T time steps.

[0080] Forward LSTM hidden state:

[0081] Backward LSTM hidden state:

[0082] Merge two-way hidden states: For splicing operations;

[0083] Take the hidden state at the last time step as the temporal feature vector F t ∈R 1×T ;

[0084] It should be explained that by capturing the dynamic changes in data from a time dimension, such as construction progress and deformation monitoring trends, it can complement spatial characteristics.

[0085] The spatial feature vector F output by the CNN s and the time feature vector F output by LSTM t Attention mechanisms are applied separately to dynamically adjust feature weights:

[0086] Spatial attention: In the formula, α s ∈R 1×S These are the spatial attention weights, and their sum is 1. d is the spatial attention weight matrix; kScaling factor

[0087] Time attention: In the formula, α t ∈R 1×T Time-based attention weights; This is the time attention weight matrix;

[0088] It should be noted that highlighting key features through attention mechanisms, such as the spatial features of point clouds at building edges and the abrupt temporal features of deformation monitoring, can suppress irrelevant noise interference.

[0089] When the dynamic weight allocation module is working, it dynamically adjusts the fusion weights of spatial and temporal features based on noise intensity to enhance the feature representation of noisy data.

[0090] When calculating the fusion weights, the formula is as follows: In the formula, F fusion β is the fusion feature; β is the spatial feature weight, with a value range of [0,1], and the temporal feature weight is 1-β; β=σ′(γ′×∈+θ); where σ′ is the sigmoid function, γ′ is the sensitivity coefficient, and θ is the bias term;

[0091] The fused feature F is achieved through a fully connected layer. fusion Dimensionality reduction to the target dimension K, default 256 dimensions:

[0092] F final =ReLU(W fc F fusion +b fc );

[0093] L2 normalization is applied to the output features to ensure consistent feature scale across different samples:

[0094]

[0095] It should be explained that by integrating weight calculation and dimensionality reduction, a low-dimensional, highly robust feature vector is output, which can provide high-quality input for subsequent distributed computing and decision-making.

[0096] In this embodiment of the invention, the dynamic weight allocation module can effectively improve the feature extraction accuracy in high-noise scenarios, especially for the suppression of salt-and-pepper noise (such as LiDAR anomalies) and stripe noise (such as uneven image exposure) commonly found in architectural surveying.

[0097] By combining the attention mechanism with the CNN-LSTM hybrid model, compared with traditional single-modal models, such as pure CNN or pure LSTM, redundant features can be effectively reduced. When the feature dimension is compressed to 256 dimensions, the key information retention rate is still over 90%. By implementing feature dimensionality reduction through the dynamic weight allocation module, the computational load of subsequent modules can be reduced and the overall processing efficiency can be improved, achieving dual optimization of accuracy and efficiency.

[0098] Distributed computing resource elastic scheduling module: Constructs a load prediction model based on reinforcement learning, monitors node computing power utilization and network latency in real time, and dynamically adjusts task allocation strategies to achieve elastic resource scaling; specific steps include:

[0099] Collect distributed node operation data from the past three months, including computing power metrics, network metrics, task metrics, and task indicators.

[0100] Among them, the computing power indicators are: CPU utilization (%), memory usage (%), and GPU memory utilization (%).

[0101] Network metrics: Inter-node communication latency (ms), bandwidth utilization (%), packet loss rate (%);

[0102] Task metrics: task type, task execution time (s), task priority (levels 1-5); where task type includes, but is not limited to, preprocessing, feature extraction, and model training.

[0103] Construct time series features and task features;

[0104] When constructing time series features, the features include the sliding window mean (e.g., a window size of 5 minutes), volatility (e.g., variance), and trend term (e.g., the slope of the linear fit).

[0105] When constructing task features, the following should be included: task type encoding, such as one-hot encoding; priority weight, such as level 1 = 0.1, level 5 = 0.9; and historical resource requirements, such as number of CPU cores and memory in GB.

[0106] The dataset was divided in a 7:3 ratio to obtain 70% training set and 30% test set. The time series data was divided using a rolling window with a step size of 1 hour.

[0107] A load prediction model is constructed using the Deep Deterministic Policy Gradient (DDPG) algorithm, which includes a state space S, an action space A, and a reward function R.

[0108] The state space S is: S = [U cpu U mem D net Q task ,T timeIn the formula, U cpu U represents the CPU utilization of the current node, normalized to [0,1]. mem For memory utilization, normalized to [0,1]; D net For network latency, normalized to [0,1]; Q task Task queue length; T time It is a time-related feature;

[0109] Action space A: Predict the incremental resource demand ΔR for the next 5 minutes, such as the number of CPU cores and the amount of memory in GB. A∈[0,10], with a maximum single expansion of 10 CPU cores and 32GB of memory.

[0110] The reward function R is: R = a·(1-|U target -U pred |)-b·D pred -c·C scale In the formula, U target Target resource utilization rate; U pred To predict utilization rate; D pred To predict network latency; C scale The cost of resource scaling is the time taken for scaling up and down, expressed in seconds; a, b, and c are weighting coefficients, with values ​​of 0.6, 0.3, and 0.1 respectively.

[0111] When training the model, the Adam optimizer was used with a learning rate of 0.001 and 500 training rounds. The parameters of the policy network and value network were updated in each round.

[0112] It should be explained that by using reinforcement learning to dynamically learn the mapping relationship between resource demand and load, high-precision load prediction can be achieved, which can provide a basis for decision-making in subsequent task scheduling.

[0113] Based on the predicted incremental resource demand ΔR and real-time monitoring data, task allocation is optimized through the following steps:

[0114] Calculate the node health score H: In the formula, D max The maximum allowable delay is H∈[0,1], where H is closer to 1, indicating a better node state.

[0115] Calculate task weights based on task priority P and urgency E:

[0116] W task =0.7·P+0.3·E; where, t deadline The remaining deadline; task priority P is level 1-5;

[0117] An improved genetic algorithm (GA) is used to assign tasks, with the objective function being to minimize the standard deviation of node load.

[0118] In the formula, N is the number of nodes; U i Let be the CPU utilization of the i-th node; Average utilization rate;

[0119] It should be explained that dynamically allocating computing tasks based on node health and task priority avoids single-point overload and can improve overall resource utilization.

[0120] Based on the task allocation results and the resource demand increment ΔR, perform elastic resource adjustments:

[0121] Expansion trigger condition: When the predicted resource demand ΔR > R threshold When the duration is greater than 30 seconds, expansion is triggered; where R threshold =0.8×R current That is, 80% of the current resources;

[0122] Scaling up strategy: Prioritize using idle nodes; otherwise, automatically create cloud server instances. Scaling down trigger condition: When the average node utilization rate... And if the duration is greater than 5 minutes, shrinkage is triggered;

[0123] Scaling down strategy: Adopt a migration-then-shutdown process to migrate tasks to nodes with lower load, and shut down idle nodes after ensuring no task interruption.

[0124] It should be noted that dynamically adjusting computing resources based on real-time load and predicted demand can achieve on-demand allocation and avoid resource waste or shortage.

[0125] In this embodiment of the invention, resource elastic scaling is achieved by dynamically adjusting task allocation strategies through real-time monitoring of node computing power utilization and network latency. Compared with static scheduling strategies, resource utilization can be effectively improved. The combination of dynamic load prediction and priority scheduling can effectively reduce the average completion delay of high-priority tasks and meet the real-time requirements of business. The combination of real-time monitoring and rapid elastic scaling can effectively improve the stability of the system under sudden loads.

[0126] Adaptive fault tolerance and data consistency guarantee module: It uses consortium blockchain technology to record the entire data processing log and combines it with a multi-replica asynchronous replication mechanism. When a node failure is detected, data recovery is triggered through the Byzantine fault tolerance algorithm.

[0127] Specifically, during the end-to-end log processing, log collection modules are embedded at key nodes in data preprocessing, feature extraction, and distributed computing to achieve automated on-chain data upload; including:

[0128] Each processing step generates a structured log entry;

[0129] The recordLog contract is automatically invoked when data processing is complete.

[0130] Critical operations require cross-signature from multiple nodes before being uploaded to the blockchain; among these critical operations are, for example, the correction of abnormal data and the change of task scheduling strategy.

[0131] It adopts a log hash + off-chain storage mode. Only the input data hash, output data hash, and ECDSA digital signature of the processing node are stored on the chain. The complete log data is stored in IPFS and accessed through hash index.

[0132] It should be explained that by using tamper-proof end-to-end logs, traceable evidence is provided for data consistency verification, while transparent auditing of the processing process is achieved.

[0133] When implementing asynchronous replication with multiple copies for critical data, such as preprocessed standardized sequences and feature extraction results:

[0134] The primary replica receives data write requests, updates local data, and generates operation logs.

[0135] Synchronize the primary replica logs from the replica via a pull mechanism, and replay the logs to update local data;

[0136] The primary replica periodically submits the replica consistency hash H to the consortium blockchain. replica =SHA-256(H1||H2||...H k For example, it is implemented every 30 seconds; i = 1, 2, 3, ..., k; H i Let be the hash value of the i-th replica;

[0137] Calculate the synchronization delay Δt between the master and slave replicas, where Δt = T slave -T master T slave To synchronize completion time from the replica, T master The primary replica write time is set to a threshold of 1 second; exceeding this threshold will trigger an alarm.

[0138] It should be explained that by using redundant storage with multiple replicas, a data source can be provided for data recovery in the event of node failure, and asynchronous replication balances performance and consistency requirements.

[0139] Monitor node health status in real time, and mark a node as a Byzantine node when it meets any of the following conditions:

[0140] Three consecutive heartbeats exceeding the timeout threshold (5 seconds);

[0141] On-chain log signature verification failed; check via the verifyLog contract.

[0142] Replica Consistent Hash replica The deviation from the on-chain record exceeds the threshold;

[0143] Additionally, if the synchronization delay of a single instance is greater than 10 seconds, it is considered a minor fault and will not affect the main instance service.

[0144] If the primary replica fails or ≥2 secondary replicas are unavailable, it is considered a severe failure, triggering the fault tolerance mechanism.

[0145] When a severe failure is detected, a PBFT-based Byzantine fault tolerance mechanism is activated to ensure data consistency and service continuity.

[0146] Among them, the new master node elected by the consensus node broadcasts a pre-preparation message to all replicas;

[0147] After the replica node verifies the validity of the message, it broadcasts a preparation message. When it receives ≥2f+1 preparation messages, it enters the commit phase; f is the number of fault-tolerant nodes, and the default f=2, i.e., 5 replicas.

[0148] The replica node broadcasts a commit message. After collecting ≥2f+1 commit messages, it performs data writing and updates its local state.

[0149] When performing data recovery, in the event of a primary replica failure, a new primary replica is elected from the surviving replicas, prioritizing the replica with the least synchronization latency, and the integrity of the replica data is verified by outputting data hashes in the on-chain logs.

[0150] In case of a replica failure, a new replica is automatically started, and all data and incremental logs are synchronized from the primary replica, with the recovery time controlled within 30 seconds.

[0151] In this embodiment of the invention, the consortium blockchain is combined with digital signatures to reduce the tampering rate of data processing logs to 0, meeting the audit and compliance requirements of architectural surveying data. By combining multi-replica asynchronous replication with Byzantine fault tolerance, the service recovery time in the event of node failure can be effectively reduced, avoiding surveying task interruption due to single point of failure. Through PBFT consensus and hash verification mechanism, the data consistency of distributed nodes is ensured, and the replica data deviation rate can be effectively controlled to meet the stringent requirements of architectural surveying for data accuracy. Through dynamic fault detection and hierarchical recovery strategies, various faults ranging from minor network jitter to malicious node attacks can be dealt with, and the robustness in complex industrial environments is significantly better than that of traditional master-slave backup architectures.

[0152] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0153] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0155] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A big data processing system for building surveying and mapping information services, characterized in that, Includes a robust preprocessing module for multi-source heterogeneous data: it collects multi-source heterogeneous data from the field of architectural surveying, identifies and corrects anomalous data through an anomaly detection mechanism that combines spatiotemporal correlation analysis with an improved isolated forest algorithm, and generates standardized data sequences; Anti-interference feature enhancement extraction module: Based on the attention mechanism-optimized CNN-LSTM hybrid model, features are extracted from the standardized data sequence, and a dynamic weight allocation module is introduced to enhance the feature representation ability of noisy data; Distributed computing resource elastic scheduling module: Constructs a load prediction model based on reinforcement learning, monitors node computing power utilization and network latency in real time, and dynamically adjusts task allocation strategies to achieve elastic scaling of resources; Adaptive fault tolerance and data consistency guarantee module: It uses consortium blockchain technology to record the entire data processing log and combines it with a multi-replica asynchronous replication mechanism. When a node failure is detected, data recovery is triggered through the Byzantine fault tolerance algorithm.

2. The big data processing system for building surveying information services according to claim 1, characterized in that, When performing spatiotemporal correlation analysis, a spatiotemporal correlation score matrix S is constructed to quantify the consistency between data points and their surrounding spatiotemporal environment: S = α·S T +(1-α)·S S In the formula, S T S represents the time relevance score. S α represents the spatial relevance score; α is the spatiotemporal weighting coefficient.

3. The big data processing system for building surveying information services according to claim 2, characterized in that, When implementing improved isolated forest anomaly detection, for local micro-anomalies: In the formula, IF(x) is the anomaly score, ranging from [0,1]; h(x) is the path length of data point x in the isolated tree; E[h(x)] is the average path length; and c(n) is the correction function. γ is Euler's constant, and n is the sample size.

4. The big data processing system for building surveying information services according to claim 3, characterized in that, When implementing the fusion anomaly judgment and correction, the spatiotemporal correlation score S and the isolated forest anomaly score IF are fused, and the comprehensive anomaly score D is calculated using the formula: D=ω×(1-S)+(1-ω)×IF; where ω is the fusion weight. When the calculated comprehensive anomaly score D is greater than or equal to the anomaly determination threshold, the corresponding data is marked as anomaly data.

5. The big data processing system for building surveying information services according to claim 4, characterized in that, When the dynamic weight allocation module is working, it dynamically adjusts the fusion weights of spatial and temporal features based on noise intensity to enhance the feature representation of noisy data. When calculating the fusion weights, the formula is as follows: In the formula, F fusion β is the fusion feature; β is the spatial feature weight, with a value range of [0,1], and the temporal feature weight is 1-β; β=σ′(γ′×∈+θ); where σ′ is the sigmoid function, γ′ is the sensitivity coefficient, and θ is the bias term; The fused feature F is achieved through a fully connected layer. fusion Dimensionality reduction to the target dimension K: F final =ReLU(W fc F fusion +b fc ); L2 normalization is applied to the output features to ensure consistent feature scale across different samples:

6. The big data processing system for building surveying information services according to claim 5, characterized in that, Construct a load prediction model, including a state space S and a reward function R; The state space S is: S = [U cpu U mem D net Q task ,T time In the formula, U cpu U represents the current node's CPU utilization. mem For memory utilization; D net For network latency; Q task Task queue length; T time It is a time-related feature; The reward function R is: R = a·(1-|U target -U pred |)-b·D pred -c·C scale In the formula, U target Target resource utilization rate; U pred To predict utilization rate; D pred To predict network latency; C scale The cost of resource scaling; a, b, and c are all weighting coefficients.

7. The big data processing system for building surveying information services according to claim 6, characterized in that, Based on the task allocation results and the resource demand increment ΔR, perform elastic resource adjustments: When the predicted resource demand ΔR>R threshold When the duration is greater than 30 seconds, expansion is triggered; where R threshold =0.8×R current That is, 80% of the current resources; When expanding capacity, idle nodes will be used first; otherwise, cloud server instances will be created automatically. When the average node utilization rate And if the duration is greater than 5 minutes, shrinkage is triggered; When scaling down, a migration-then-shutdown process is adopted, migrating tasks to nodes with lower loads and shutting down idle nodes after ensuring no task interruption.

8. The big data processing system for building surveying information services according to claim 7, characterized in that, When processing end-to-end logs, log collection modules are embedded in key nodes of data preprocessing, feature extraction, and distributed computing to achieve automated on-chain processing.

9. A big data processing system for building surveying information services according to claim 8, characterized in that, When a severe failure is detected, a PBFT-based Byzantine fault tolerance mechanism is activated to ensure data consistency and service continuity.

10. A big data processing system for building surveying information services according to claim 9, characterized in that, When performing data recovery, in the event of a primary replica failure, a new primary replica is elected from the surviving replicas, prioritizing the replica with the least synchronization latency, and the integrity of the replica data is verified by outputting data hashes in the on-chain logs. In case of a replica failure, a new replica is automatically started to synchronize all data and incremental logs from the primary replica.

Images