Big Data-Based Surveying and Mapping Quality Assessment and Optimization System

Abstract

This invention relates to the field of surveying and mapping quality assessment and optimization technology, specifically a big data-based surveying and mapping quality assessment and optimization system. The system operates as follows: First, it acquires data using multi-source surveying and mapping equipment to improve spatial coverage and measurement accuracy. Second, it uses an adaptive weighted clustering denoising method, dynamically weighting data based on source reliability, measurement accuracy, and spatial distribution characteristics to remove outliers and fill in missing data using linear interpolation, standardizing data from different equipment to improve fusion performance. Then, it dynamically adjusts the weighted average method based on data trust level to improve data consistency and accuracy, and generates a loss prevention verification sequence at the edge device. Finally, it achieves dual verification through modular arithmetic and hash verification, and employs a comprehensive index system of integrity, consistency, accuracy, and reliability to provide a systematic surveying and mapping data assessment framework. This invention improves the quality and security of surveying and mapping data and provides a scientific basis for subsequent optimization.

Description

Technical Field

[0001] This invention relates to the field of surveying and mapping quality assessment and optimization technology, specifically to a surveying and mapping quality assessment and optimization system based on big data. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, the surveying and mapping industry faces many challenges such as a surge in data volume, fusion of multi-source data, and inconsistent quality. Traditional surveying and mapping methods often encounter problems such as noise interference, data loss, and inconsistencies in data standards across devices when collecting, processing, and analyzing multi-dimensional heterogeneous data. As a result, the surveying and mapping results are difficult to meet the needs of modern refined management and intelligent decision-making in terms of accuracy, consistency, and reliability.

[0003] Chinese invention patent application CN119377867A discloses an AI-based surveying and mapping data processing system. The system includes a data source weight allocation module, a data fusion module, a real-time terrain monitoring module, and a geographic anomaly analysis module. By introducing timeliness and accuracy assessments of different data sources in the data source weight allocation, it can assign weights to multi-source data. This allows for appropriate adjustments based on the reliability of data from different sources during synthesis, avoiding data redundancy and optimizing the impact of high-quality data. The system monitors environmental changes in real time based on the terrain change rate and accurately assesses the degree of anomaly based on update frequency and terrain type, identifying potential terrain data problems. By analyzing abnormal terrain changes and combining them with an impact assessment of human activity areas, the system can effectively determine the urgency of abnormal changes, providing timely and effective support for decision-making.

[0004] In recent years, the widespread application of advanced equipment such as drones, satellites, laser scanners, and ground sensors has greatly enriched the means of acquiring surveying and mapping data. However, it has also made data preprocessing, noise removal, missing value imputation, and cross-device data fusion more complex. Current surveying and mapping data processing methods mainly rely on fixed threshold filtering, simple clustering denoising, or rule-based fusion methods, which lack adaptability and intelligence and are difficult to meet the high-precision requirements of multi-source surveying and mapping data. Furthermore, due to differences in the accuracy of surveying and mapping equipment, interference from environmental factors, and errors in the data transmission process, surveying and mapping data often suffers from noise, missing values, and inconsistencies, which seriously affect the accuracy and reliability of surveying and mapping results. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background technology by proposing a surveying and mapping quality assessment and optimization system based on big data.

[0006] The technical solution of the present invention is a surveying and mapping quality assessment and optimization system based on big data, comprising: a surveying and mapping data acquisition module, a surveying and mapping data processing module, a quality assessment module, an optimization scheme generation module, a result feedback and adjustment module, and a visualization platform;

[0007] The surveying data acquisition module is responsible for acquiring surveying data from surveying equipment in real time.

[0008] The surveying data processing module is used to preprocess the surveying data and generate a loss prevention verification sequence for the processed surveying data.

[0009] The quality assessment module receives the loss prevention verification sequence, verifies the mapping data, defines four major indicators: integrity, consistency, accuracy, and reliability, and calculates a comprehensive quality score Q based on weighted coefficients. The higher the Q value, the better the data quality. The quality level is divided into four levels: excellent, good, qualified, and unqualified, based on the score. Finally, the quality assessment result is output, and the relevant data is transmitted to the optimization scheme generation module.

[0010] The optimization scheme generation module is used to set the target quality score, calculate the difference between the current quality score and the target score, determine the necessity of optimization, and if optimization is required, formulate an optimization strategy based on four indicators: completeness, consistency, accuracy, and reliability, and automatically execute the optimization strategy.

[0011] The results feedback and adjustment module provides feedback on the improvement in surveying and mapping quality after the optimization strategy is implemented, and makes real-time adjustments accordingly.

[0012] A visualization platform is used to provide users with a visual interface.

[0013] Preferably, the specific implementation steps for preprocessing surveying and mapping data are as follows:

[0014] S21. A multi-level denoising algorithm based on adaptive weighted clustering denoises the collected survey data by comprehensively considering factors such as spatial distribution of data, measurement accuracy, and equipment reliability.

[0015] S22. Introduce a missing value imputation method based on linear interpolation to fill in missing data. The missing value imputation formula based on linear interpolation is as follows:

[0016] ;

[0017] In the formula, y(t) represents the predicted value at time point t; y(t0) and y(t1) represent the known values ​​at time points t0 and t1, respectively; t represents the time point where the missing value is located; t0 and t1 represent the nearest time points before and after the missing value, respectively.

[0018] S23. Standardize the survey data. The standardization formula is as follows:

[0019] ;

[0020] In the formula, x' represents the standardized data value; x represents the original data value; μ' represents the data mean; and σ represents the data standard deviation.

[0021] The preferred implementation process of the multi-level denoising algorithm based on adaptive weighted clustering is as follows:

[0022] S31. Using a spatial distribution-based partitioning method, the dataset is divided into different levels. The data partitioning formula is as follows:

[0023] ;

[0024] In the formula, x j D represents a data point. i and D i+1 Indicates the distance range for dividing the hierarchy; c i The layer represents the center of the i-th layer; Layer i Let x represent the dataset of the i-th layer, containing all data points x that satisfy the partitioning criteria. j ;

[0025] S32. In each data layer, an adaptive weighted clustering algorithm is used to dynamically adjust the weights of data points based on the signal quality and error magnitude to improve the clustering effect, as detailed below:

[0026] S3201, Data Point Weight Calculation: Each data point is assigned a different weight based on its precision, source device, and spatial location. Points with higher weights are given higher priority. The weight calculation formula is as follows:

[0027] ;

[0028] In the formula, w j Represents data point x j weights; accuracy(x) j The value () represents the measurement accuracy, calculated based on equipment type and environmental factors; reliability(x) j The distance(x) indicates the reliability of the data source; j (μ): Data point x j With data point x j The distance μ from the cluster center of the layer; α, β, and the coefficients of different factors in the adjustment weight calculation;

[0029] S3202. Through a weighted clustering process, using an improved weighted K-means clustering algorithm, the data points are divided into multiple clusters. The weighted K-means clustering formula is:

[0030] ;

[0031] In the formula, J represents the weighted error summation; n i μ represents the number of data points in the i-th class; i w represents the cluster center of the i-th class; j Represents data point x j The weights; k represents the total number of categories;

[0032] Based on this, by minimizing the weighted error, the clustering algorithm dynamically adjusts the cluster centers and allocates clusters;

[0033] After clustering is completed, the density of each cluster is analyzed. If the density of a cluster is lower than the set threshold θ1, it is a sparse region. The data points in this region are marked as potential noise and removed.

[0034] S3203. Introduce a dynamic adjustment mechanism, that is, during the clustering process, dynamically adjust the weights according to the clustering results and denoising effect of the data points: increase the weights of points with low initial weights.

[0035] S33. Through a collaborative mechanism of local and global denoising, local and global noise are eliminated, specifically as follows:

[0036] S3301, Local Denoising: Fine-grained processing is performed on each cluster. For each cluster, a neighborhood-based density assessment is used to treat data points with low density within the cluster as noise points. The local density assessment formula is as follows:

[0037] ;

[0038] In the formula, ρ(x) j ) represents data point x j Local density; N(x) j ) represents data point x j The neighborhood; w i Represents the data point x in the neighborhood. i The weights;

[0039] Therefore, points with a local density lower than the set threshold θ2 will be considered noise points and eliminated.

[0040] S3302, Global Denoising: This process handles noise across the entire dataset, integrates the denoising results from the hierarchical data divisions, filters out noise points globally, and further cleans the data through global consistency analysis. The global consistency evaluation formula is:

[0041] ;

[0042] where C' represents the global consistency metric; N represents the total number of data points; w i and w j respectively represent the weights of data points x i and x j ; ||x i -x j || represents the distance between data points x i and x j ;

[0043] S34. By calculating the global consistency, further detect global abnormal data and perform denoising.

[0044] Preferably, the process of generating the loss prevention check sequence is as follows:

[0045] S41. Select a random number r ∈ {0, 1} k , and calculate the auxiliary sequence generation parameter APg = H(r||data);

[0046] where data is the binary character sequence of the summarized surveying and mapping data; k is a predefined security parameter; H is a predefined hash function, H: {0, 1} k → ; represents the integer ring modulo a; a = p × q; p and q are predefined prime numbers; || represents the string concatenation operation;

[0047] S42. Calculate the sequence generation parameter Pg = (APg) b mod N;

[0048] where b is a predefined sequence encoding parameter, satisfying eb = 1 mod [(p - 1) × (q - 1)]; e is a predefined prime number, satisfying that e is relatively prime to a, and satisfying 1 < e < (p - 1) × (q - 1) and gcd(e, (p - 1) × (q - 1)) = 1; gcd() is the greatest common divisor function;

[0049] S43. Generate the loss prevention check sequence Sc = (r, Pg).

[0050] Preferably, the verification process for the surveying and mapping data is as follows:

[0051] S51. Convert the received summarized surveying and mapping data into a binary string sequence data';

[0052] S52. Calculate the first-order verification parameter Pc1 = (Pg) e ;

[0053] Where e represents a predefined prime number;

[0054] S53. Calculate the second-order parity parameter Pc2=H(r||data');

[0055] Where r is the random number contained in the loss prevention check sequence Sc; || represents the string concatenation operation; H is a predefined hash function;

[0056] S54. If Pc1=Pc2, then the received and summarized survey data has passed the verification.

[0057] Preferably, the calculation process for the overall quality score Q is as follows:

[0058] S32. Define data quality metrics: completeness, consistency, accuracy, and reliability;

[0059] S32. Define the formula for comprehensive quality scoring:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] In the formula, Q represents the overall quality score; w C w I w A and w R w represents the weight of each quality indicator. C +w I +w A +w R =1; C, I, A, and R represent the scores for integrity, consistency, accuracy, and reliability, respectively, all normalized to [0,1]; N valid N represents the number of valid points; total m' represents the total number of data points; m' represents the number of matched data points; x i and This represents measurements from different data sources at the same point in time; y i This represents the i-th measurement value; The i-th reference value is represented by δ; n represents the number of data points; δ represents the allowable range of measurement error; σ represents the standard deviation of the data. max This indicates the maximum permissible standard deviation.

[0066] Preferably, the surveying data processing module is a designated edge device.

[0067] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects:

[0068] This invention designs a surveying and mapping quality assessment and optimization system based on big data. Through key technologies such as adaptive weighted clustering denoising, multi-source data fusion, edge computing verification, and intelligent quality assessment, it significantly improves the quality, security, and reliability of surveying and mapping data. Simultaneously, it reduces manual intervention and enhances the intelligence and automation level of data processing, providing strong technical support for high-precision, high-reliability surveying and mapping applications.

[0069] (1) Improve the quality of surveying and mapping data: By using the adaptive weighted clustering denoising method, outliers and noise in the surveying and mapping data can be effectively removed, thereby improving the accuracy and stability of the data; and by adopting a local + global denoising collaborative mechanism, the data can be refined in the local area while maintaining the consistency of the global data, thereby improving the integrity and reliability of the surveying and mapping results.

[0070] (2) Enhance the accuracy of surveying and mapping data fusion: Combine the weighted average method and the dynamic adjustment mechanism of equipment trust to fuse surveying and mapping data from different sources, improve data consistency, reduce the deviation between multi-source surveying and mapping data, and optimize data interpolation in time and space dimensions through spatiotemporal feature analysis to improve the accuracy of fused data and provide higher quality input for the further application of surveying and mapping data;

[0071] (3) Improve data security and transmission integrity: Introduce loss prevention verification sequence at the edge computing device, and combine hash verification and modular operation dual verification mechanism to effectively prevent data loss, tampering and errors during data transmission, improve the security of surveying and mapping data, and reduce dependence on cloud computing resources through edge data verification, improve the real-time performance of surveying and mapping data, and reduce data transmission bandwidth costs.

[0072] (4) Construct a systematic surveying and mapping data quality assessment system: Use comprehensive indicators such as integrity (C), consistency (I), accuracy (A) and reliability (R) to quantitatively assess the quality of surveying and mapping data, provide a scientific basis for the optimization of surveying and mapping data, and combine machine learning algorithms to dynamically optimize the quality assessment standards, making the analysis of surveying and mapping data quality more intelligent and adaptive, and improving the accuracy of the assessment results;

[0073] (5) Improve the intelligence and automation level of surveying and mapping data processing: Automatically adjust the data denoising parameters through adaptive weighted clustering method, reduce manual intervention, improve data cleaning efficiency and intelligence level, and combine deep learning and spatiotemporal analysis technology to optimize data fusion and evaluation strategies, improve the automation level of surveying and mapping data processing, reduce manual intervention, and improve the overall operating efficiency of surveying and mapping system. Attached Figure Description

[0074] Figure 1 This is a system architecture diagram of a big data-based surveying and mapping quality assessment and optimization system proposed in this invention. Detailed Implementation

[0075] Example 1, as Figure 1 As shown, the present invention proposes a surveying and mapping quality assessment and optimization system based on big data, comprising: a surveying and mapping data acquisition module, a surveying and mapping data processing module, a quality assessment module, an optimization scheme generation module, a result feedback and adjustment module, and a visualization platform.

[0076] The surveying data acquisition module is responsible for acquiring surveying data in real time from surveying equipment (including but not limited to drones, satellites, and laser scanners);

[0077] The surveying data processing module is a set edge device that preprocesses surveying data based on big data technology, including but not limited to data denoising, data supplementation, and missing value handling.

[0078] The quality assessment module uses machine learning and artificial intelligence algorithms to evaluate the quality of data, including but not limited to measurement accuracy, completeness, and consistency.

[0079] The optimization solution generation module automatically generates targeted optimization solutions based on the quality assessment results, including but not limited to improvements in data acquisition processes, equipment configurations, and data processing strategies.

[0080] The results feedback and adjustment module provides feedback on the improvement in surveying quality after the optimization plan is implemented, and makes real-time adjustments accordingly.

[0081] The visualization platform provides users with a visual interface.

[0082] Example 2: The present invention proposes a big data-based surveying and mapping quality assessment and optimization method, which is applied to the big data-based surveying and mapping quality assessment and optimization system proposed in Example 1. The specific implementation steps are as follows:

[0083] S1. The surveying data acquisition module acquires surveying data from various surveying devices, including but not limited to:

[0084] Unmanned Aerial Vehicles (UAVs): Using drones equipped with high-resolution cameras and laser scanners to conduct aerial photography and lidar mapping, they can accurately capture three-dimensional information of ground objects;

[0085] Satellites: Satellite imagery provides large-scale geographic data, suitable for mapping vast areas;

[0086] LiDAR (LiDAR): Obtains high-precision 3D point cloud data through laser scanning, suitable for complex terrain or large-scale detailed mapping;

[0087] Ground sensors: acquire environmental information (including but not limited to temperature, humidity, and ground movement) through ground equipment to supplement spatial mapping data and form a multi-dimensional dataset;

[0088] S2, the surveying data processing module, is a designated edge device that aggregates surveying data and preprocesses it. The specific implementation process is as follows:

[0089] S21. A multi-level denoising algorithm based on adaptive weighted clustering achieves more accurate denoising by comprehensively considering factors such as the spatial distribution of data, measurement accuracy, and equipment reliability. The implementation process is as follows:

[0090] S2101. In actual surveying and mapping, data often exhibits uneven spatial density. Point cloud data in some areas may be too dense, while measurement data in other areas may be sparse. To better handle this data, a spatial distribution-based partitioning method is first adopted to divide the dataset into different levels. The data partitioning formula is as follows:

[0091] ;

[0092] In the formula, x j D represents a data point. i and D i+1 Indicates the distance range for dividing the hierarchy; c i The layer represents the center of the i-th layer; Layer i Let x represent the dataset of the i-th layer, containing all data points x that satisfy the partitioning criteria. j ;

[0093] S2102. In each data layer, an adaptive weighted clustering algorithm is used to dynamically adjust the weights of data points based on the signal quality and error magnitude to improve the clustering effect, as detailed below:

[0094] (1) Data point weight calculation: Each data point is assigned a different weight based on its accuracy, source device, and spatial location. Points with higher weights are given higher priority. The weight calculation formula is as follows:

[0095] ;

[0096] In the formula, w j Represents data point x j weights; accuracy(x) j The value () represents the measurement accuracy, calculated based on equipment type and environmental factors; reliability(x) j The distance(x) indicates the reliability of the data source, including but not limited to the historical performance of the device; j (μ): Data point x j With data point x j The distance μ from the cluster center of the layer; α, β, and the coefficients of different factors in the adjustment weight calculation;

[0097] (2) Through a weighted clustering process, the improved weighted K-means clustering algorithm is used to divide the data points into multiple clusters. The weighted K-means clustering formula is as follows:

[0098] ;

[0099] In the formula, J represents the weighted error summation; n i μ represents the number of data points in the i-th class; i w represents the cluster center of the i-th class; j Represents data point x j The weights; k represents the total number of categories;

[0100] Based on this, by minimizing the weighted error, the clustering algorithm dynamically adjusts the cluster centers to achieve optimal cluster allocation;

[0101] After clustering is completed, the density of each cluster is analyzed. If the density of a cluster is lower than the set threshold θ1 (i.e., a sparse region), the data points in that region are marked as potential noise and removed.

[0102] (3) In order to further improve the clustering effect, a dynamic adjustment mechanism is introduced. That is, during the clustering process, the weights are dynamically adjusted according to the clustering results and denoising effect of the data points, especially for points with low initial weights, to increase their influence in subsequent clustering.

[0103] S2103. Through a collaborative mechanism of local and global denoising, both local and global noise are effectively eliminated. Specifically:

[0104] (1) Local denoising focuses on fine-grained processing of each cluster. For each cluster, a neighborhood-based density assessment is used to treat data points with lower density within the cluster as noise points. The local density assessment formula is as follows:

[0105] ;

[0106] In the formula, ρ(x) j ) represents data point x j Local density; N(x) j ) represents data point x j The neighborhood; w i Represents the data point x in the neighborhood. i The weights;

[0107] Therefore, points with a local density lower than the set threshold θ2 will be considered noise points and eliminated.

[0108] (2) Global denoising focuses on processing noise in the entire dataset, combining the denoising results from multiple levels, filtering out noise points globally, and further cleaning the data through global consistency analysis. The global consistency evaluation formula is:

[0109] ;

[0110] In the formula, C' represents the global consistency measure; N represents the total number of data points; w i and w j They represent data points x respectively i and x j The weights; ||x i -x j || represents data point x i and x j The distance between;

[0111] Based on this, global consistency is calculated to further detect and denoise global abnormal data;

[0112] S22. During the surveying process, missing values ​​may sometimes occur due to equipment failure or data transmission interruption. To ensure data integrity, a missing value filling method based on linear interpolation is introduced to fill in the missing data. The missing value filling formula based on linear interpolation is as follows:

[0113] ;

[0114] In the formula, y(t) represents the predicted value at time point t; y(t0) and y(t1) represent the known values ​​at time points t0 and t1, respectively; t represents the time point where the missing value is located; t0 and t1 represent the nearest time points before and after the missing value, respectively.

[0115] S23. Since the data may come from different types of surveying equipment and have inconsistent scales (including but not limited to different coordinate systems and measurement units of different equipment), standardization is performed. The standardization formula is:

[0116] ;

[0117] Where \(x'\) represents the standardized data value; \(x\) represents the original data value; \(\mu'\) represents the data mean; \(\sigma\) represents the data standard deviation.

[0118] S24. Integrate surveying and mapping data from different devices and in different formats. The surveying and mapping data comes from different sources, including but not limited to drones, satellites, and ground sensors, and varies in many aspects (including but not limited to spatial resolution, time granularity, and measurement accuracy). These data are effectively integrated into a unified multi-dimensional dataset. The data fusion formula based on the weighted average method is:

[0119] ;

[0120] Where represents the fused data value; represents the weight of the \(i\)-th measurement result; \(x\) i represents the value of the \(i\)-th measurement result; \(m\) represents the number of data sources participating in the fusion; [[ID=二十]]

[0121] S25. Since the surveying and mapping data processing module is an edge device set, to ensure that the aggregated surveying and mapping data will not be lost during data transmission, the surveying and mapping data processing module generates a loss prevention check sequence for the surveying and mapping data. The generation process is as follows:

[0122] S2501. Select a random number \(r\in\{0,1\}\) k , and calculate the auxiliary sequence generation parameter \(APg = H(r||data)\);

[0123] where \(data\) is the binary character sequence of the aggregated surveying and mapping data; \(k\) is a predefined security parameter; \(H\) is a predefined hash function, \(H:\{0,1\}\) k \(\to\) ; represents the integer ring modulo \(a\); \(a = p\times q\); \(p\) and \(q\) are predefined prime numbers; \(\|\) represents the string concatenation operation;

[0124] S2502. Calculate the sequence generation parameter \(Pg=(APg)\) b \(\bmod N\);

[0125] where \(b\) is a predefined sequence encoding parameter that satisfies \(eb = 1\bmod[(p - 1)\times(q - 1)]\); \(e\) is a predefined prime number that is relatively prime to \(a\) and satisfies \(1\lt e\lt(p - 1)\times(q - 1)\) and \(\gcd(e,(p - 1)\times(q - 1)) = 1\); \(\gcd()\) is the greatest common divisor function;

[0126] S2503. Generate the loss prevention check sequence \(Sc=(r, Pg)\);

[0127] S26. Summarize the mapping data values ​​of {loss prevention check sequence Sc=(r, Pg)}. Transmitted to the quality assessment module;

[0128] S3, the quality assessment module, conducts a comprehensive analysis of the completeness, consistency, accuracy, and reliability of the surveying area data to ensure that the data meets the requirements of subsequent surveying tasks. Its specific implementation process is as follows:

[0129] S31. Receive {loss prevention check sequence Sc=(r, Pg), summarizing data values} The received aggregated data is validated. The validation process is as follows:

[0130] S3101. Convert the received and summarized survey data into a binary string sequence data';

[0131] S3102. Calculate the first-order parity parameter Pc1=(Pg) e ;

[0132] Where e represents a predefined prime number;

[0133] S3103, Calculate the second-order parity parameter Pc2=H(r||data');

[0134] Where r is the random number contained in the loss prevention check sequence Sc; || represents the string concatenation operation; H is a predefined hash function;

[0135] S3104. If Pc1=Pc2, then the received aggregated survey data has passed the verification.

[0136] S32. Define data quality metrics:

[0137] Completeness (C): The extent to which the data has sufficient coverage;

[0138] Consistency (I): Whether different data sources maintain spatial, temporal, and attribute consistency;

[0139] Accuracy (A): Whether the error in the data is within an acceptable range;

[0140] Reliability (R): Whether the data is stable and can be repeatedly measured to obtain similar results;

[0141] S32. Based on the above indicators, define the comprehensive quality scoring formula:

[0142] ;

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] In the formula, Q represents the overall quality score, ranging from [0,1], with the closer to 1 indicating higher quality; w C w I w A and w R Each represents the weight of a quality indicator, set according to the specific needs of the surveying task. C +w I +w A +w R =1; C, I, A, and R represent the scores for integrity, consistency, accuracy, and reliability, respectively, all normalized to [0,1]; N valid N represents the number of valid points; total m' represents the total number of data points; m' represents the number of matched data points; x i and This represents measurements from different data sources at the same point in time; y i This represents the i-th measurement value; The i-th reference value is represented by δ; n represents the number of data points; δ represents the allowable range of measurement error; σ represents the standard deviation of the data. max This indicates the maximum permissible standard deviation.

[0148] S33. Based on the comprehensive quality score Q and according to the surveying and mapping quality grade classification rule table, output the quality grade to evaluate the data quality. The surveying and mapping quality grade classification rule table is shown in Table 1:

[0149]

[0150] S34. Output the quality level (evaluation result) and transfer the quality level, comprehensive quality score Q, integrity C, consistency I, accuracy A and reliability R scores to the optimization scheme generation module;

[0151] S4. The optimization scheme generation module generates optimization schemes based on the evaluation results and implements them to improve data quality. The goal of the optimization scheme is to improve the integrity, consistency, accuracy, and reliability of the data so that it meets the requirements of the surveying and mapping task. The specific implementation process is as follows:

[0152] S41. Set the target quality score Q target ;

[0153] S42. Calculate the gap between the current quality score and the target quality score to determine the urgency and amount of optimization: ;

[0154] like A value greater than 0 indicates that optimization is needed; conversely, the data quality has met the target.

[0155] S43. Based on quality gap Based on different quality assessment indicators (completeness, consistency, accuracy, reliability), corresponding optimization strategies are formulated, including the following:

[0156] Integrity optimization: If the data integrity is low (C<0.7), increase the frequency of data collection or use completion algorithms (including but not limited to interpolation or data expansion methods).

[0157] ;

[0158] In the formula, C new α' represents the optimized completeness value; α' represents the scaling factor for supplementary data, with a value range of [0,1], determined according to the severity of missing data.

[0159] Consistency optimization: For data with poor consistency (I<0.7), optimization can be performed through data calibration (including but not limited to spatial registration) or temporal alignment methods (including but not limited to interpolation and filtering);

[0160] ;

[0161] In the formula, I new β' represents the optimized consistency value; β' represents the consistency calibration factor, which uses machine learning algorithms (including but not limited to support vector machine regression) to correct the data.

[0162] Accuracy optimization: When the accuracy is low (A<0.7), the data is optimized through accuracy improvement algorithms (including but not limited to error correction models);

[0163] ;

[0164] In the formula, A new This represents the optimized accuracy value; γ' represents the accuracy improvement coefficient, which is adjusted based on the degree of accuracy error measured from the actual data.

[0165] Reliability optimization: If the reliability of the data is low (R<0.7), its reliability can be improved by averaging the data from multiple measurements;

[0166] ;

[0167] In the formula, Rnew δ represents the optimized reliability value; δ' represents the stability enhancement factor, which is improved by increasing the frequency of data collection or the diversity of data sources.

[0168] S44. After the optimization plan is generated, the optimization measures are automatically applied: through the automated scheduling algorithm, the system will select appropriate optimization methods for adjustment;

[0169] For example, if the data integrity is low, the system will automatically schedule drones or satellites to collect additional data; if the accuracy is not up to standard, the system will automatically activate the error correction model to make adjustments.

[0170] S5. The Result Feedback and Adjustment module provides real-time feedback on the optimization effect after the optimization plan is implemented. By conducting another quality assessment, it verifies whether the optimization plan has effectively improved the accuracy, consistency and completeness of the data, and the feedback results will be sent to the visualization platform.

[0171] If the optimization effect is not ideal, the system will automatically identify the cause of the problem and instruct the optimization solution generation module to generate a new adjustment plan. The optimization solution generation module will optimize the implementation plan, adjust the optimization direction, including but not limited to adjusting the data collection timing and improving the data processing strategy, and continue to implement the adjusted plan.

[0172] The results feedback and adjustment module tracks the effects until the data quality reaches the predetermined standard.

[0173] S6. The visualization platform displays the processed surveying data, the generated optimization scheme, and the feedback results on the visualization interface.

[0174] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A surveying and mapping quality assessment and optimization system based on big data, characterized in that, include: The system includes a surveying and mapping data acquisition module, a surveying and mapping data processing module, a quality assessment module, an optimization scheme generation module, a result feedback and adjustment module, and a visualization platform. The surveying data acquisition module is responsible for acquiring surveying data from surveying equipment in real time. The surveying data processing module is used to preprocess the surveying data and generate a loss prevention verification sequence for the processed surveying data. The quality assessment module receives the loss prevention verification sequence, verifies the mapping data, defines four major indicators: integrity, consistency, accuracy, and reliability, and calculates a comprehensive quality score Q based on weighted coefficients. The higher the Q value, the better the data quality. The quality level is divided into four levels: excellent, good, qualified, and unqualified, based on the score. Finally, the quality assessment result is output, and the relevant data is transmitted to the optimization scheme generation module. The optimization scheme generation module is used to set the target quality score and calculate the difference between the current quality score and the target score. If optimization is required, it formulates an optimization strategy based on four indicators: completeness, consistency, accuracy, and reliability, and automatically executes the optimization strategy. Specifically, this includes: A1. Set the target quality score Q target ; A2. Calculate the gap between the current quality score and the target quality score to determine the urgency and scope of optimization: ; like A value greater than 0 indicates that optimization is needed; conversely, the data quality has met the target. A3. Based on quality gap Based on the completeness, consistency, accuracy, and reliability of different quality assessment indicators, corresponding optimization strategies are formulated, including the following: Integrity optimization: If the data integrity score C < 0.7, increase the frequency of data collection and use a completion algorithm; ； In the formula, C new α' represents the optimized integrity value; α' represents the scaling factor for the supplementary data, with a value range of [0,1]. Consistency optimization: For data with a consistency score I < 0.7, optimization is performed through data calibration and time alignment methods; ； In the formula, I new This represents the optimized consistency value; β' represents the consistency calibration factor, which uses machine learning algorithms to correct the data. Precision optimization: When the precision value A of the data is less than 0.7, the data is optimized through a precision enhancement algorithm; ； In the formula, A new This represents the optimized accuracy value; γ' represents the accuracy improvement factor. Reliability optimization: If the reliability score R of the data is less than 0.7, its reliability can be enhanced by averaging several measurements. ； In the formula, R new δ represents the optimized reliability value; δ' represents the stability enhancement factor, which is improved by increasing the frequency of data collection or the diversity of data sources. A4. After the optimization plan is generated, the optimization measures are automatically applied: Through an automated scheduling algorithm, the system selects appropriate optimization methods for adjustment; The results feedback and adjustment module provides feedback on the improvement in surveying and mapping quality after the optimization strategy is implemented, and makes real-time adjustments accordingly. A visualization platform is used to provide users with a visual interface.

2. The surveying and mapping quality assessment and optimization system based on big data according to claim 1, characterized in that, The specific steps for preprocessing surveying data are as follows: S21. A multi-level denoising algorithm based on adaptive weighted clustering denoises the collected survey data by comprehensively considering factors such as spatial distribution of data, measurement accuracy, and equipment reliability. S22. Introduce a missing value imputation method based on linear interpolation to fill in missing data. The missing value imputation formula based on linear interpolation is as follows: ； In the formula, y(t) represents the predicted value at time point t; y(t0) and y(t1) represent the known values ​​at time points t0 and t1, respectively; t represents the time point where the missing value is located; t0 and t1 represent the nearest time points before and after the missing value, respectively. S23. Standardize the survey data. The standardization formula is as follows: ； In the formula, x' represents the standardized data value; x represents the original data value; μ' represents the data mean; and σ represents the data standard deviation.

3. The surveying and mapping quality assessment and optimization system based on big data according to claim 2, characterized in that, The implementation process of the multi-level denoising algorithm based on adaptive weighted clustering is as follows: S31. Using a spatial distribution-based partitioning method, the dataset is divided into different levels. The data partitioning formula is as follows: ； In the formula, x j D represents a data point. i and D i+1 Indicates the distance range for dividing the hierarchy; c i The layer represents the center of the i-th layer; Layer i Let x represent the dataset of layer i, containing all data points x that satisfy the partitioning criteria. j ; S32. In each data layer, an adaptive weighted clustering algorithm is adopted to dynamically adjust the weights of data points according to the signal quality and error magnitude to improve the clustering effect, as follows: S3201, Data Point Weight Calculation: Each data point is assigned a different weight based on its precision, source device, and spatial location. The weight calculation formula is as follows: ; In the formula, w j Represents data point x j Weights; accuracy(x) j The value () represents the measurement accuracy, calculated based on equipment type and environmental factors; reliability(x) j ) indicates the reliability of the data source; distance(x) j (μ): Data point x j With data point x j The distance to the cluster center μ of the layer; α represents the weighting coefficient of measurement accuracy; β represents the weighting coefficient of data source reliability; γ represents the weighting coefficient of distance; S3202. Through a weighted clustering process, using an improved weighted K-means clustering algorithm, the data points are divided into multiple clusters. The weighted K-means clustering formula is: ； In the formula, J represents the weighted error summation; n i μ represents the number of data points in the i-th class; i w represents the cluster center of the i-th class; j Represents data point x j The weights; k represents the total number of categories; Based on this, by minimizing the weighted error, the clustering algorithm dynamically adjusts the cluster centers and assigns the clusters; After clustering, the density of each cluster is analyzed. If the density of a certain cluster is lower than the set threshold θ1, it is a sparse area, and the data points in this area are marked as potential noise and removed; S3203. Introduce a dynamic adjustment mechanism, that is, during the clustering process, the weights are dynamically adjusted according to the clustering results and denoising effects of the data points; S33. Through the collaborative mechanism of local denoising and global denoising, local noise and global noise are removed, specifically as follows: S3301. Local denoising: Perform fine-grained processing on each cluster. For each cluster, further perform density evaluation based on the neighborhood. The local density evaluation formula is: ； In the formula, ρ(x) j ) represents data point x j Local density; N(x) j ) represents data point x j The neighborhood; w i Represents the data point x in the neighborhood. i The weights; Based on this, points with local density lower than the set threshold θ2 will be regarded as noise points and removed; S3302. Global denoising: Process the noise in the entire dataset, comprehensively combine the denoising results of the divided data levels, screen out the noise points within the global range, and further clean the data through global consistency analysis. The global consistency evaluation formula is: ； In the formula, C' represents the global consistency measure; N represents the total number of data points; w i and w j They represent data points x respectively i and x j The weights; ||x i -x j || represents data point x i and x j The distance between; S34. By calculating the global consistency, further detect global abnormal data and perform denoising.

4. The surveying and mapping quality assessment and optimization system based on big data according to claim 1, characterized in that, The process of generating the loss prevention check sequence is as follows: S41. Select a random number r∈{0,1} k Calculate the auxiliary sequence generation parameter APg=H(r||data); Where data is the binary character sequence of the summarized survey data; k is a predefined security parameter; and H is a predefined hash function, where H: {0,1}. k → ; Represents the integer ring modulo a; a = p × q; p and q are predefined prime numbers; || represents the string concatenation operation; S42. Calculate the sequence generation parameter Pg = (APg) b mod a; Where b is a predefined sequence encoding parameter, satisfying eb = 1 mod [(p - 1)×(q - 1)]; e is a predefined prime number, satisfying that e is relatively prime to a, and satisfying 1 < e < (p - 1)×(q - 1) and gcd(e, (p - 1)×(q - 1)) = 1; gcd() is the greatest common divisor function; S43. Generate the loss prevention check sequence Sc = (r, Pg).

5. The surveying and mapping quality assessment and optimization system based on big data according to claim 4, characterized in that, The check process for checking the surveying and mapping data is as follows: S51. Convert the received aggregated surveying and mapping data into a binary string sequence data'; S52. Calculate the first-order parity parameter Pc1=(Pg) e ; Where e represents a predefined prime number; S53. Calculate the second-order check parameter Pc2 = H(r||data'); Where r is the random number included in the loss prevention check sequence Sc; || represents the string concatenation operation; H is a predefined hash function; S54. If Pc1 = Pc2, the received aggregated surveying and mapping data passes the check.

6. The surveying and mapping quality assessment and optimization system based on big data according to claim 1, characterized in that, The calculation process of the comprehensive quality score Q is as follows: S32. Define data quality indicators: integrity, consistency, accuracy, reliability; S32. Define the comprehensive quality score formula: ； ； ； ； ； In the formula, Q represents the overall quality score; w C w I w A and w R w represents the weight of each quality indicator. C +w I +w A +w R =1; C, I, A, and R represent the scores for integrity, consistency, accuracy, and reliability, respectively, all normalized to [0,1]; N valid N represents the number of valid points; total Indicates the total number of data points; m' represents the number of matched data points; x i and This represents measurements from different data sources at the same point in time; y i This represents the i-th measurement value; The i-th reference value is represented by δ; n represents the number of data points; δ represents the allowable range of measurement error; σ represents the standard deviation of the data. max This indicates the maximum permissible standard deviation.

7. The surveying and mapping quality assessment and optimization system based on big data according to claim 1, characterized in that, The surveying and mapping data processing module is the set edge device.

Images