A big data management method for cooperative monitoring of laser radar and total station

Abstract

The application relates to the technical field of big data management, and discloses a big data management method for cooperative monitoring of a laser radar and a total station. A uniform record is generated by uniformly numbering and discretely indexing an access end; a mean value, an uncertainty, and a completeness of a sub-observation are summarized; a latest value is taken as a missing value, and a history is not zeroed; a reverse uncertainty and effectiveness are constructed according to a reference scale, and are aggregated into a quality score with the same weight as the completeness; a high, medium, and low priority area is adaptively divided according to a global mean value and a standard deviation; a full amount is taken in the high area, an equidistant sampling is taken in the medium area, and an interval selection is taken in the low area and is quantified into a database; a partition index and a priority scheduling are generated, and a periodical reevaluation and a copy are kept continuous. The link eliminates index confusion and alignment difficulty, quantifies quality, suppresses extreme values and missing values, reduces storage and retrieval costs, and realizes quality-driven resource value balance.

Description

Technical Field

[0001] This invention relates to the field of big data management technology, specifically to a big data management method for collaborative monitoring of lidar and total station. Background Technology

[0002] With the rapid development of infrastructure monitoring, geological disaster early warning, and urban digitalization, high-precision collaborative monitoring systems based on lidar and total stations have become an important technical means for data acquisition and spatial modeling in complex scenarios. Existing collaborative monitoring systems typically deploy lidar and total stations at multiple points to acquire large volumes of multi-temporal, multi-dimensional measurement data, serving typical engineering applications such as bridge health diagnosis, tunnel deformation monitoring, and power line inspection. In these scenarios, the monitoring data exhibits typical characteristics such as massive volume, heterogeneous structure, uneven acquisition density, and high real-time requirements, posing significant challenges to data management and subsequent analysis and processing.

[0003] In existing technologies, data management methods for lidar and total stations mostly employ centralized raw data storage or simplified sampling methods based on fixed rules, making it difficult to achieve intelligent fusion and dynamic partitioning management of multi-source heterogeneous data. Specifically, common storage solutions primarily rely on "full retention" or "time window discarding," lacking quantitative evaluation of the quality, validity, and representativeness of the collected data, and failing to adaptively hierarchical storage and access scheduling based on actual data quality. Meanwhile, some technologies alleviate storage pressure by introducing distributed databases, file slicing, or data compression algorithms, but these methods typically depend on static configuration, preset templates, or general compression parameters, failing to dynamically adjust storage priorities and access strategies based on the multi-source data characteristics of specific monitoring tasks. Especially in the context of large-scale concurrent acquisition, multi-device collaboration, and real-time data stream updates, existing methods struggle to balance storage efficiency, data quality preservation, and the refined requirements of subsequent analysis. Furthermore, existing multi-source data management technologies generally do not handle data loss, anomalies, and noise interference that occur during the acquisition process meticulously enough. They often rely on simple null value marking or global discarding strategies, failing to differentiate the validity of data from different devices, time periods, and spatial locations. This leads to information loss or misjudgment in subsequent analysis and intelligent scheduling. For multi-source heterogeneous monitoring environments, there is a lack of a quantitative comprehensive evaluation mechanism that combines multiple factors such as data quality, completeness, uncertainty, and device characteristics to guide partitioned storage and dynamic priority access.

[0004] Therefore, this case aims to propose a big data management method for collaborative monitoring of lidar and total station. Through unified coding and time-series indexing, multi-source heterogeneous data is integrated into a single structured record. Then, through sub-observation decoupling and parallel statistics of uncertainty and completeness, a quality assessment system is constructed. Finally, by combining dynamic threshold division and partitioned storage strategy, differentiated processing of high, medium and low data types is achieved, supplemented by indexing, priority sorting and periodic reassessment to ensure that data storage and access take into account timeliness, accuracy and efficiency. Summary of the Invention

[0005] This invention provides a big data management method for collaborative monitoring of lidar and total station, which helps to solve the problems mentioned in the background art.

[0006] This invention provides the following technical solution: a big data management method for collaborative monitoring of lidar and total station, comprising:

[0007] Construct a unified numbering and discrete-time index to record lidar distance and reflection intensity, total station elevation and viewing angle, and generate a unified structure record;

[0008] Summarize the current period's sub-observations with the same number and classify them by equipment. If sub-observations exist, calculate the mean and variance uncertainty and statistically analyze the completeness. If measurements are missing, fill in the meter dimension component with the most recent historical value. If there is no historical value, set it to zero.

[0009] Set a measurement reference scale, convert the uncertainty into an inverse uncertainty term, retain the integrity term, perform missing value processing, upper limit pruning and normalization on the reflection intensity, and perform missing value processing and range pruning on the viewing angle;

[0010] The reverse uncertainty, completeness, and validity are weighted equally and aggregated into a comprehensive quality score, which is then arranged into a score sequence by number.

[0011] Calculate the global average score and standard deviation. Scores above the average plus one standard deviation are classified as high priority zones, scores within plus or minus one standard deviation are classified as medium priority zones, and scores below the average minus one standard deviation are classified as low priority zones.

[0012] The high priority area is fully stored, the medium priority area is sampled at equal intervals in ascending order of number, and the low priority area is selected at intervals in ascending order and the meter-dimensional components are quantized and rounded according to resolution before being stored.

[0013] Generate an index for each partition containing an index number, time, and partition label; establish priority values ​​and schedule queries in descending order.

[0014] Reassess missing data, sub-observations, statistics, and scores at fixed intervals, update partitions and priorities, retain copies of the previous period when the priority drops from high or medium to low, and add full records for the current period when the priority rises from low to medium or high, thus maintaining sequence continuity.

[0015] Optionally, the construction of a unified numbering and discrete-time index, recording the lidar distance and reflection intensity, total station elevation and viewing angle, and generating a unified structure record specifically includes:

[0016] Configure the lidar sampling unit and the total station sampling unit, and establish their respective equipment number lists according to consecutive natural numbers;

[0017] Establish a time index sequence composed of discrete reception times, and assign a batch number to each batch of data;

[0018] For each discrete moment, the lidar records two data points: the representative value of the distance and the reflection intensity, and arranges them in chronological order to form the observation vector for that moment.

[0019] The total station records two data points at each discrete moment: the representative elevation value and the viewing angle, and then assembles them into the observation vector for that moment in chronological order.

[0020] Map the numbers of the two types of equipment to a unified numbering system so that any unified number corresponds to a complete observation record at any time.

[0021] Optionally, the summary of current period sub-observations with the same number and classified by equipment, when sub-observations exist, calculates the mean and variance uncertainty and statistically analyzes the completeness; when measurements are missing, the meter-dimensional component is filled in with the most recent historical value, and when there is no historical value, it is set to zero. Specifically, this includes:

[0022] For any given unified number, all sub-observation entries in the batch are summarized at any discrete time to form the sub-observation set of the batch;

[0023] The items in the set are classified according to the type of equipment: lidar items are classified as distance measurement items, and total station items are classified as elevation measurement items.

[0024] When there are entries in the set, calculate the arithmetic mean of the sub-observations in that batch, and use it as the representative value at that moment;

[0025] When there are entries in the set, calculate the variance-type uncertainty of the batch of sub-observations, and use it as the uncertainty index at that moment;

[0026] The completeness of sub-observations in this batch is calculated as a percentage based on the ratio of the number of sub-observations to the maximum expected number of observations in the system. The completeness is recorded as zero when the set is empty.

[0027] When the set is empty, deterministic backfilling is performed on components with meter dimensions: if a historical record exists, the most recently defined historical value is taken; if no historical record exists, it is set to zero. This rule applies to both lidar distance components and total station elevation components.

[0028] Optionally, the setting of the measurement reference scale, converting the uncertainty into an inverse uncertainty term, retaining the integrity term, performing missing value processing, upper limit pruning and normalization on the reflection intensity, and performing missing value processing and range pruning on the viewing angle, specifically includes:

[0029] Set a measurement reference scale for each unique number;

[0030] The uncertainty is transformed by a monotone bounded transformation to obtain the inverse uncertainty term, which is denoted as zero when the set is empty.

[0031] The completeness level will be used as a completeness item in subsequent calculations;

[0032] The lidar reflection intensity is processed for missing values ​​and pruned to the upper limit, and then normalized to the upper limit to obtain the validity term at that moment;

[0033] The total station's field of view is processed for missing values ​​and its range is clipped to keep the values ​​within the range of 0 to 180 degrees.

[0034] Optionally, the step of weighting and aggregating inverse uncertainty, completeness, and validity into a comprehensive quality score, and forming a score sequence by number, specifically includes:

[0035] The reverse uncertainty term, completeness term, and validity term are aggregated with equal weights to form a comprehensive quality score;

[0036] At the same time, the comprehensive quality scores of all units are combined into a score sequence according to a unified numbering order.

[0037] Optionally, the calculation of the global average score and standard deviation, with scores above the average plus one standard deviation assigned to the high-priority zone, scores within plus or minus one standard deviation assigned to the medium-priority zone, and scores below the average minus one standard deviation assigned to the low-priority zone, specifically includes:

[0038] Calculate the global average quality score at this moment;

[0039] Calculate the standard deviation of the quality score at this moment;

[0040] A unit score is marked as a high priority zone when it is more than one standard deviation above the global average; a unit score is marked as a medium priority zone when it is within one standard deviation above or below the global average; and a unit score is marked as a low priority zone when it is less than one standard deviation below the global average.

[0041] Optionally, the high-priority region is fully stored, the medium-priority region is sampled at equal intervals in ascending order of number, and the low-priority region is selected at ascending intervals and its meter-dimensional components are quantized and rounded according to resolution before being stored in the database. Specifically, this includes:

[0042] Perform a full storage operation on the high priority area;

[0043] After forming a candidate set for the central priority area, a deterministic sequence is established in ascending order of unified numbering. The number of sampling records, which is not less than one, is determined based on the sampling ratio parameter of the central priority area. The sampling record set is obtained by using an equidistant selection method and stored in the database.

[0044] After forming a candidate set for low priority areas, an effective compression ratio parameter is set and the number of selected records is calculated to be at least one. The selected record set is obtained by deterministically selecting records in ascending order of uniform numbering.

[0045] For meter-dimensional components in the selected records, the quantization step size is set according to the device resolution and the nearest integer is quantized. For non-meter-dimensional components, the range is clipped, and compressed records are generated and stored in the database.

[0046] Optionally, generating an index for each partition containing a number, time, and partition label, establishing priority values, and scheduling queries in descending order specifically includes:

[0047] Generate an index table for each partition, with each index entry containing a unique number, time, and partition label;

[0048] Map partitions to access priority values, with high priority partitions corresponding to the highest level, medium priority partitions corresponding to medium level, and low priority partitions corresponding to low level.

[0049] When executing a query task, it is scheduled from high to low priority, first retrieving data blocks in the high priority area, and then retrieving data blocks in the medium priority area and low priority area.

[0050] Optionally, the step of re-evaluating missing data, sub-observations, statistics, and scores at fixed intervals, updating partitions and priorities, retaining copies of the previous period when the priority drops from high or medium to low, and adding full records for the current period when the priority rises from low to medium or high, to maintain sequence continuity, specifically includes:

[0051] At fixed time intervals, the missing data, sub-observation set, sub-observation mean, standard deviation, reverse uncertainty term, completeness term, validity term and comprehensive quality score of each uniform number in two adjacent intervals are recalculated, and the global average quality score and standard deviation are calculated at the same time.

[0052] Update the partitions and access priorities of each unified number based on the reassessment results;

[0053] When any unified number changes from a high-priority or medium-priority zone to a low-priority zone between two adjacent reassessment intervals, a complete copy of the previous interval's record is retained; when it is promoted from a low-priority zone to a medium-priority or high-priority zone, a full record of that interval is added to ensure the continuity and completeness of the time series.

[0054] The present invention has the following beneficial effects:

[0055] During the data access phase, a unified numbering and discrete-time index are constructed, enabling any observation unit to quickly locate its corresponding multi-source observation record at any given time using a unique number. By packaging the distance and reflection intensity of the lidar and the elevation and viewing angle of the total station into a single data structure, not only is the data preprocessing process simplified, but the foundation for subsequent consistency quality assessment and storage partitioning is also laid. This method solves the problems of index chaos and time alignment difficulties during multi-source data fusion, improving data fusion efficiency. Simultaneously, the unified numbering provides reliable correlation support for subsequent statistical analysis. Compared with existing technologies, no additional cross-source mapping rules or dynamic query merging are required, reducing system complexity and computational overhead, ensuring the organic integration of real-time access and unified organization, and meeting the scalability requirements of large-scale deployments.

[0056] This scheme, under each unified number and time point, first categorizes all sub-observation items by equipment type, then calculates the arithmetic mean, variance-type uncertainty, and completeness to quantify the coverage of sub-observation data. For missing data, the most recent valid record is used to fill in the missing data or the data is reset to zero to ensure an uninterrupted continuous sequence. This achieves a refined characterization of data quality; on the one hand, uncertainty reflects the reliability of measurements in the same batch; on the other hand, completeness reflects the proportion of missing data, providing a quantitative indicator for subsequent quality scoring.

[0057] A measurement reference scale is set for each unified number, and the uncertainty is transformed into an aggregateable inverse uncertainty term through bounded transformation, while retaining the integrity term. Simultaneously, missing value processing, trimming to the rated range, and normalization are performed on the lidar reflection intensity, and missing value processing and range trimming are performed on the total station's viewing angle. This approach unifies the validity indicators of different physical quantities into the same dimensional interval, facilitating subsequent equal-weighted aggregation. This innovation not only eliminates the impact of extreme or missing values ​​on quality assessment but also ensures the comparability and interpretability of the assessment indicators.

[0058] This solution proposes to aggregate three major components—inverse uncertainty, completeness, and validity—in an equally weighted manner to form a comprehensive quality score and generate a numbered sorted sequence. This quantitatively reflects the overall quality level of the data at each time-level unit. This innovation, by introducing three-dimensional indicators, achieves a comprehensive assessment of data reliability, completeness, and usability, and provides direct input for subsequent threshold partitioning and storage strategies through the numbered sorted sequence. Compared with existing technologies, it eliminates the need for manually setting complex weights, ensuring fairness among indicators. Furthermore, the comprehensive scoring method more intuitively reflects the quality differences between different data units, providing an operable and transparent quantitative basis for partitioned storage.

[0059] This solution dynamically divides data into high, medium, and low priority zones based on the global average quality score and standard deviation at any given time: data above the average plus one standard deviation is designated as high priority, data within plus or minus one standard deviation is designated as medium priority, and data below the average minus one standard deviation is designated as low priority. This innovation uses adaptive statistical thresholds to respond promptly to changes in data quality distribution, achieving fine-grained storage priority differentiation. Compared to fixed threshold methods, this mechanism avoids over-storage of low-quality data or omission of high-quality data, ensuring that critical data is always retained with the highest priority. Simultaneously, the capacity of the medium and low priority zones can be automatically adjusted according to real-time quality fluctuations, improving storage resource utilization efficiency.

[0060] High-priority areas are stored in full; medium-priority areas are sampled at equal intervals in ascending order of uniform numbering to ensure at least one record is retained; low-priority areas are selected at predetermined intervals based on ascending order, and the meter-dimension components are quantized at device resolution level and rounded to the nearest integer to form compressed records for storage. This strategy innovatively combines quality-driven partitioning and tiered storage, ensuring the integrity of high-quality data while applying differentiated compression and sampling in medium and low-quality areas, thus reducing storage costs. Compared to traditional one-size-fits-all full or fixed-ratio compression methods, this strategy can flexibly allocate storage resources while ensuring the availability of critical data, achieving a balance between storage efficiency and data value.

[0061] This solution generates index tables at each partition level. Index entries include a unique ID, timestamp, and partition label, mapping the partition label to access priority values. This enables queries to be scheduled and retrieved from highest to lowest priority. This innovation achieves dual optimization of data storage and access: on the one hand, it quickly locates target data through the index table; on the other hand, it ensures that high-value data receives priority in query responses through priority scheduling. Unlike existing technologies that use unified retrieval or simple partitioned retrieval, this solution dynamically controls the retrieval order and scope, making business queries both efficient and quality-assured, reducing query latency and improving the availability of critical data.

[0062] To address the time-varying nature of the monitoring environment and data distribution, the proposed solution recalculates the missing data, sub-observation statistics, uncertainty items, completeness items, validity items, and comprehensive scores for each data number at fixed intervals, and updates the partitions and priorities. When a unit is demoted from a high or medium priority zone to a low priority zone, a complete copy of the previous period is retained; when it is promoted from a low priority zone to a medium or high priority zone, the entire record of the current period is supplemented to ensure the continuity and integrity of the time series. This innovation balances storage optimization and historical data integrity, avoiding data discontinuity caused by qualitative changes in partitioning through copy retention and supplementation mechanisms. Compared with the static storage or only addition strategies in existing technologies, this solution's dynamic evaluation and supplementation mechanism achieves consistent maintenance of long-term time series and traceability of key data, meeting the application scenarios with high requirements for historical evolution analysis and time series integrity. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example, refer to Figure 1 A big data management method for collaborative monitoring of lidar and total station includes:

[0066] Construct a unified numbering and discrete-time index to record lidar distance and reflection intensity, total station elevation and viewing angle, and generate a unified structure record;

[0067] Summarize the current period's sub-observations with the same number and classify them by equipment. If sub-observations exist, calculate the mean and variance uncertainty and statistically analyze the completeness. If measurements are missing, fill in the meter dimension component with the most recent historical value. If there is no historical value, set it to zero.

[0068] Set a measurement reference scale, convert the uncertainty into an inverse uncertainty term, retain the integrity term, perform missing value processing, upper limit pruning and normalization on the reflection intensity, and perform missing value processing and range pruning on the viewing angle;

[0069] The reverse uncertainty, completeness, and validity are weighted equally and aggregated into a comprehensive quality score, which is then arranged into a score sequence by number.

[0070] Calculate the global average score and standard deviation. Scores above the average plus one standard deviation are classified as high priority zones, scores within plus or minus one standard deviation are classified as medium priority zones, and scores below the average minus one standard deviation are classified as low priority zones.

[0071] The high priority area is fully stored, the medium priority area is sampled at equal intervals in ascending order of number, and the low priority area is selected at intervals in ascending order and the meter-dimensional components are quantized and rounded according to resolution before being stored.

[0072] Generate an index for each partition containing an index number, time, and partition label; establish priority values ​​and schedule queries in descending order.

[0073] Reassess missing data, sub-observations, statistics, and scores at fixed intervals, update partitions and priorities, retain copies of the previous period when the priority drops from high or medium to low, and add full records for the current period when the priority rises from low to medium or high, thus maintaining sequence continuity.

[0074] By performing a series of eight consecutive steps on the multi-source lidar and total station observation data during the access phase, including unified numbering and discrete-time indexing, sub-observation decomposition and uncertainty statistics, validity parameter construction, comprehensive quality score calculation, dynamic threshold partitioning, partitioned storage strategy, partitioned indexing and priority scheduling, and periodic re-evaluation and consistency maintenance, a series of problems existing in the back-end fusion of traditional heterogeneous sensor data are solved, such as index chaos, difficulty in time alignment, lack of quantitative basis for quality evaluation, storage redundancy and inefficient retrieval, and discontinuous historical sequences. First, unified numbering and time indexing ensure that each observation record has a unique label at the source, avoiding mapping errors during subsequent cross-source merging. Second, sub-observation decomposition and uncertainty statistics not only replace single-value or discard strategies but also quantify measurement reliability and data integrity, ensuring that subsequent storage strategies are justified. Third, the construction of validity parameters normalizes data of different physical quantities to the same dimension, eliminating the interference of extreme and missing values ​​on subsequent aggregation. The comprehensive quality score equally weights the three dimensions, avoiding the subjectivity of traditional manual weighting. Dynamic threshold partitioning adjusts storage priority in real time based on statistical distribution, eliminating the defects of fixed thresholds. Partitioned storage and index scheduling ensure the full preservation of key data while reducing storage costs and retrieval latency through sampling and quantization compression. Finally, periodic reassessment and replication mechanisms effectively maintain the continuity and traceability of long-term time series. Compared with the existing technology's "full data first, then archive" model, which mostly involves single storage or simple sampling, this solution takes data quality as the core driver, takes into account both real-time performance and historical integrity, achieves the optimal balance between storage resources and data value, and improves the efficiency and reliability of data management in a large-scale collaborative monitoring environment.

[0075] The process of constructing a unified numbering and discrete-time index, recording lidar distance and reflection intensity, total station elevation and viewing angle, and generating a unified structure record specifically includes:

[0076] Configure the lidar sampling unit and the total station sampling unit, and establish their respective equipment number lists according to consecutive natural numbers;

[0077] Establish a time index sequence composed of discrete reception times, and assign a batch number to each batch of data;

[0078] For each discrete moment, the lidar records two data points: the representative value of the distance and the reflection intensity, and arranges them in chronological order to form the observation vector for that moment.

[0079] The total station records two data points at each discrete moment: the representative elevation value and the viewing angle, and then assembles them into the observation vector for that moment in chronological order.

[0080] Map the numbers of the two types of equipment to a unified numbering system so that any unified number corresponds to a complete observation record at any time.

[0081] Further specific implementation steps include:

[0082] L lidar sampling units were set up within the collaborative monitoring area, numbered as follows: And set the S total station sampling units numbered as Each unit transmits the observations to the data management center; where L is the total number of lidar sampling units and S is the total number of total station sampling units. s is the index for the lidar unit; s is the index for the total station unit;

[0083] Set a unified time index sequence: ;in, Let T be the set of discrete time points; T is the number of discrete batches. This represents the k-th discrete reception time; k is the discrete time index.

[0084] Building the first lidar Unit in Observation vector at time ;in, For lidar Unit in Distance at any given time represents a specific value; For lidar Unit in The intensity of reflection at any given moment;

[0085] Constructing the s-th unit of the total station in Observation vector at time ;in, For the s-th unit of the total station The elevation value at any given time; For the s-th unit of the total station A perspective of a moment;

[0086] Unified Number:

[0087] Where m is a multi-source unified number, if Then the corresponding lidar unit, if This corresponds to the total station unit; For the m-th uniformly numbered unit at time... The observation vector.

[0088] By configuring lidar and total station sampling units at the monitoring site in advance, assigning them sequential numbers, and establishing a discrete-time index sequence, the data from each batch of observations are uniformly collected into numbered observation records at the source. This solves the problems of data fusion difficulties and real-time synchronization issues caused by the independent numbering of different sensors and the lack of clear time labels. It tightly connects the data access layer with the subsequent statistical analysis layer: all observation units have unique numbers and can be directly shared with other modules without the need for additional mapping tables or post-processing alignment; at the same time, the discrete-time index takes into account the actual situation of asynchronous sampling, ensuring that data can be accurately assigned to the corresponding time regardless of when it arrives.

[0089] The process involves summarizing current period sub-observations with the same number and classifying them by equipment. When sub-observations exist, the mean and variance-type uncertainty are calculated, and the completeness is statistically analyzed. For missing measurements, the meter-dimensional component is filled in using the most recent historical value; if no historical value exists, it is set to zero. Specifically, this includes:

[0090] For any given unified number, all sub-observation entries in the batch are summarized at any discrete time to form the sub-observation set of the batch;

[0091] The items in the set are classified according to the type of equipment: lidar items are classified as distance measurement items, and total station items are classified as elevation measurement items.

[0092] When there are entries in the set, calculate the arithmetic mean of the sub-observations in that batch, and use it as the representative value at that moment;

[0093] When there are entries in the set, calculate the variance-type uncertainty of the batch of sub-observations, and use it as the uncertainty index at that moment;

[0094] The completeness of sub-observations in this batch is calculated as a percentage based on the ratio of the number of sub-observations to the maximum expected number of observations in the system. The completeness is recorded as zero when the set is empty.

[0095] When the set is empty, deterministic backfilling is performed on components with meter dimensions: if a historical record exists, the most recently defined historical value is taken; if no historical record exists, it is set to zero. This rule applies to both lidar distance components and total station elevation components.

[0096] Further specific implementation steps include:

[0097] For each multi-source unit m in Establish a set of all sub-observations at time 1. :

[0098] ;in, For the m-th unit at time... The j-th sub-observation scalar; j is the sub-observation index within the cell; This represents the number of sub-observations in the m-th unit within a single batch;

[0099] If m≤L, it is stipulated that This is the j-th distance measurement within this batch;

[0100] If m > L, it is stipulated that This is the j-th elevation measurement within this batch;

[0101] when At that time, calculate the mean of the sub-observations: ;in, For the m-th unit at time... The arithmetic mean of the sub-observations;

[0102] when When calculating the uncertainty:

[0103] ;in, For the m-th unit at time... Uncertainty;

[0104] Calculate observation completeness: ;in, For the m-th unit at time... Completeness; This represents the system's expected maximum number of sub-observations.

[0105] like If so, no calculation is performed on this unit at this moment. , and order ;in, For the m-th unit at time... The inverse uncertainty term;

[0106] Perform steps S201 to S202, for The dimensional components of the meter are backfilled definitively:

[0107] S201. If m≤L, define:

[0108] ;

[0109] in, For lidar Unit in The distance representation value after time-based backfilling; Less than The most recent moment satisfies the existing definition of this component;

[0110] S202. If m > L, define:

[0111] ;in, For the s-th unit of the total station Representative elevation value after backfilling;

[0112] in, If the set is empty, then It is false.

[0113] By decomposing and statistically analyzing the sub-observation data—first dividing all original sampled values ​​of the same number and time into two categories, distance and elevation, according to equipment type—and then calculating the arithmetic mean as the representative value and the variance-type uncertainty as the reliability index, the system simultaneously calculates the number of sub-observations and the expected maximum number of sub-observations to form a completeness index. This approach overcomes the shortcomings of traditional methods that only take the first observation or discard multiple measurements after simple noise reduction. It preserves the internal differences and degree of missing data, fully reflecting the reliability and usability of the data through quantitative methods, and avoiding the crude processing of blindly adding zeros or discarding data. In particular, the missing data backfilling strategy prioritizes the use of the most recent valid historical records, ensuring the continuity of the time series rather than simply setting it to zero, thus improving the smoothness and accuracy of the time series analysis.

[0114] The established measurement reference scale converts uncertainty into an inverse uncertainty term, retains the integrity term, performs missing value processing, upper limit pruning, and normalization on reflection intensity, and performs missing value processing and range pruning on viewing angle, specifically including:

[0115] Set a measurement reference scale for each unique number;

[0116] The uncertainty is transformed by a monotone bounded transformation to obtain the inverse uncertainty term, which is denoted as zero when the set is empty.

[0117] The completeness level will be used as a completeness item in subsequent calculations;

[0118] The lidar reflection intensity is processed for missing values ​​and pruned to the upper limit, and then normalized to the upper limit to obtain the validity term at that moment;

[0119] The total station's field of view is processed for missing values ​​and its range is clipped to keep the values ​​within the range of 0 to 180 degrees.

[0120] Further specific implementation steps include:

[0121] Define a measurement reference scale for each unit. ;in, is the reference scale for the m-th unit;

[0122] when At that time, construct the inverse uncertainty term. ;

[0123] when season ;

[0124] Set the completeness item to ;in, For the m-th unit at time... Completeness item;

[0125] Perform steps S301 to S302 to construct the validity items:

[0126] S301. For lidar (m≤L), pruning including missing data handling:

[0127] ;in, This is the rated upper limit of the reflection intensity; For lidar Unit in The reflection intensity after clipping at any moment;

[0128] Take again ;in, For the m-th unit in Validity of time;

[0129] S302. For total stations (m>L), clipping and displacement including missing data handling:

[0130] ;in, For the s-th unit of the total station The perspective after being cropped in real time.

[0131] By setting a measurement reference scale for each observation unit and converting the uncertainty index calculated in the previous step into an inverse uncertainty term while retaining the integrity term, and by performing missing value processing on the lidar reflection intensity, pruning it to the rated upper limit and normalizing it, and performing missing value processing on the total station perspective, pruning it to a reasonable range, a unified aggregateable validity term is generated. This solves the problems of inconsistent dimensions between different physical quantities and the impact of extreme and missing values ​​on aggregation. A controllable pruning and normalization strategy is adopted to map multivariate indicators to a unified interval, ensuring that all indicators are comparable during subsequent equal-weighted aggregation and that no single dimension is overly dominant or ignored.

[0132] The process of weighting and aggregating inverse uncertainty, completeness, and validity into a comprehensive quality score, and forming a score sequence by number, specifically includes:

[0133] The reverse uncertainty term, completeness term, and validity term are aggregated with equal weights to form a comprehensive quality score;

[0134] At the same time, the comprehensive quality scores of all units are combined into a score sequence according to a unified numbering order.

[0135] Further specific implementation steps include:

[0136] Construct a comprehensive quality score function: ;in, For the m-th unit in The overall quality score at any given moment;

[0137] All units Composition of quality score vector:

[0138] ;in, Let be the quality score vector of all units at time k.

[0139] By aggregating three indicators—inverse uncertainty, completeness, and normalized validity—into a comprehensive quality score with equal weights, and generating a time-level score sequence with a unified numbering order, this approach solves the problems of subjectivity and difficulty in unifying weight settings when fusing multiple indicators. It achieves a quantitative assessment of overall data quality while ensuring that all indicators are equally important, enabling the system to intuitively compare quality differences between different units or time points without additional manual intervention or complex machine learning algorithms.

[0140] The calculation of the global average score and standard deviation is as follows: scores above the average plus one standard deviation are classified as high-priority zones; scores within plus or minus one standard deviation are classified as medium-priority zones; and scores below the average minus one standard deviation are classified as low-priority zones. Specifically, this includes:

[0141] Calculate the global average quality score at this moment;

[0142] Calculate the standard deviation of the quality score at this moment;

[0143] A unit score is marked as a high priority zone when it is more than one standard deviation above the global average; a unit score is marked as a medium priority zone when it is within one standard deviation above or below the global average; and a unit score is marked as a low priority zone when it is less than one standard deviation below the global average.

[0144] Further specific implementation steps include:

[0145] Calculate the global average quality score: ;in, The global average quality score at time k;

[0146] Calculate the standard deviation of the quality score: ;in, Let $\frac{k}{k}$ be the standard deviation of the global quality score at time $k$.

[0147] Establish partitioning rules:

[0148] ;

[0149] in, For the m-th unit in The partition category label for each moment.

[0150] By calculating the average quality score and standard deviation of all units at the current moment, and dynamically dividing the data into high, medium, and low priority zones with the average plus or minus one standard deviation as the boundary, the problem of static threshold partitioning being unable to adapt to fluctuations in data distribution is solved. The dynamic threshold adaptive mechanism can automatically adjust the partition boundaries based on real-time data distribution, so that the storage priority division can be updated synchronously when the overall data quality improves or declines, avoiding the underestimation of critical data or the overestimation of low-quality data due to the invalidation of historical experience thresholds.

[0151] The high-priority region is fully stored, the medium-priority region is sampled at equal intervals in ascending order of number, and the low-priority region is selected at ascending intervals and its meter-dimensional components are quantized and rounded according to resolution before being stored. Specifically, this includes:

[0152] Perform a full storage operation on the high priority area;

[0153] After forming a candidate set for the central priority area, a deterministic sequence is established in ascending order of unified numbering. The number of sampling records, which is not less than one, is determined based on the sampling ratio parameter of the central priority area. The sampling record set is obtained by using an equidistant selection method and stored in the database.

[0154] After forming a candidate set for low priority areas, an effective compression ratio parameter is set and the number of selected records is calculated to be at least one. The selected record set is obtained by deterministically selecting records in ascending order of uniform numbering.

[0155] For meter-dimensional components in the selected records, the quantization step size is set according to the device resolution and the nearest integer is quantized. For non-meter-dimensional components, the range is clipped, and compressed records are generated and stored in the database.

[0156] Further specific implementation steps include:

[0157] make Construct two projection operators: , ;in, For the m-th unit in A time-recording tuple; For numbered projection, enter records. Output its number m; For vector projection, input records Output its observation vector ;

[0158] S601, Perform full storage on high priority areas:

[0159] ;in, This is the set of high-priority records at time k.

[0160] S602. For the central priority area, perform proportional sampling storage:

[0161] make: ,

[0162] ;in, Let be the set of middle-priority candidate records at time k; For set The cardinality;

[0163] Set deterministic sort: by Sort in ascending order to obtain the sequence ,in Strictly increasing; among them, For the sorted number One element; For sequence indexing;

[0164] Constructing the sampling ratio: ;in, Sampling proportion in the priority area;

[0165] And set the number of samples:

[0166] ;in, The number of samples taken from the middle priority area;

[0167] Construct a deterministic interval sampling function:

[0168] when season ;

[0169] in, Given a candidate set and a ratio as input, output the set of selected records.

[0170] when First, obtain the index of the equally spaced selection sequence. :

[0171] , ,make

[0172] ;

[0173] Based on this, the final set of stored records for the middle priority zone is obtained. :

[0174] ;

[0175] S603. For low-priority regions, perform deterministic interval selection and quantization compression:

[0176] make ,

[0177] ;in, This is the set of low-priority candidate records; For set The cardinality;

[0178] Set the low-priority raw compression rate to And set the low priority effective compression rate to :

[0179] ;

[0180] when When selecting the number of items

[0181] ;

[0182] according to Sort in ascending order to obtain the sequence ;in, For the sorted number One element; For sequence indexing;

[0183] Constructing a deterministic interval selection:

[0184] ;

[0185] in, Select a deterministic interval for the low-priority region and output the record set; The index of the selected input;

[0186] For any scalar component Z, construct a deterministic quantization function:

[0187] , ;in, For deterministic quantization functions, the input scalar Z and the compression ratio parameter are... Output the quantized scalar; The resolution of the device to which component Z belongs; To round to the nearest integer; Δ is the quantization step size;

[0188] Constructing composite functions:

[0189] ;in, As a composite function, take the record and compression ratio as input, apply a deterministic quantization function to the meter-dimensional components according to the device type, take the clipping value for the non-meter-dimensional components, and output a new record with a number.

[0190] Therefore, the composite compression function is constructed as follows:

[0191] ;

[0192] in, For composite compression functions, a composite operation of first selecting and then quantizing is performed. The input is a low-priority candidate set and an effective compression ratio, and the output is a set of compressed records.

[0193] Construct the final set of records for the low-priority region:

[0194] .

[0195] By employing differentiated storage strategies for different priority zones—high-priority zones perform full storage and establish secondary indexes; medium-priority zones are stored by uniformly numbered ascending order with equidistant sampling, ensuring at least one record is retained; and low-priority zones, based on ascending order, first select records by interval, then quantize and trim distance and elevation data according to device resolution before storage—this resolves the contradiction between ensuring the integrity of high-value data and the storage cost of medium- and low-value data within the same system. High-value data is fully retained to meet the needs of precise analysis; medium-value data is sampled to ensure time-series representativeness while reducing storage volume; and low-value data retains its basic trend information through resolution-level compression, saving storage space.

[0196] The process of generating an index for each partition containing an index number, time, and partition label, establishing priority values, and scheduling queries in descending order includes:

[0197] Generate an index table for each partition, with each index entry containing a unique number, time, and partition label;

[0198] Map partitions to access priority values, with high priority partitions corresponding to the highest level, medium priority partitions corresponding to medium level, and low priority partitions corresponding to low level.

[0199] When executing a query task, it is scheduled from high to low priority, first retrieving data blocks in the high priority area, and then retrieving data blocks in the medium priority area and low priority area.

[0200] Further specific implementation steps include:

[0201] Create an index table for each partition: ;in, A table containing records of triples;

[0202] Construct the access priority function:

[0203] ;in, This is an access priority function; a larger value indicates higher priority.

[0204] Press when querying tasks Scheduling from high to low priority enables priority retrieval of high-priority data blocks.

[0205] By generating an index table containing a unique number, timestamp, and partition label for each partition, and mapping the partition label to an access priority value, the system schedules retrievals from high to low priority during queries. This solves the problems of high retrieval complexity after storage partitioning and the inability to prioritize access to high-value data. It not only supports quickly locating the storage position of each block but also automatically tilts queries towards high-priority data, enabling the system to return the most critical data first, thus improving retrieval efficiency and business real-time performance.

[0206] The process of reassessing missing data, sub-observations, statistics, and scores at fixed intervals, updating partitions and priorities, retaining copies of the previous period when the priority drops from high or medium to low, and adding full records for the current period when the priority rises from low to medium or high, thus maintaining sequence continuity, specifically includes:

[0207] At fixed time intervals, the missing data, sub-observation set, sub-observation mean, standard deviation, reverse uncertainty term, completeness term, validity term and comprehensive quality score of each uniform number in two adjacent intervals are recalculated, and the global average quality score and standard deviation are calculated at the same time.

[0208] Update the partitions and access priorities of each unified number based on the reassessment results;

[0209] When any unified number changes from a high-priority or medium-priority zone to a low-priority zone between two adjacent reassessment intervals, a complete copy of the previous interval's record is retained; when it is promoted from a low-priority zone to a medium-priority or high-priority zone, a full record of that interval is added to ensure the continuity and completeness of the time series.

[0210] Further specific implementation steps include:

[0211] The missing data, sub-observation set, sub-observation mean, standard deviation, reverse uncertainty term, completeness term, validity term, comprehensive quality score and global statistics of each unit are recalculated at fixed intervals, and the partition and access priority are updated accordingly.

[0212] The global statistics include the global average quality score and the standard deviation of the global quality score;

[0213] When any unit changes from a high-priority or medium-priority zone to a low-priority zone between two adjacent reassessment periods, a complete copy of the previous period's record is retained; when it moves from a low-priority zone to a medium- or high-priority zone, a full record of that period is added to ensure the continuity and integrity of the time series.

[0214] This forms a quality-driven partitioned storage and partitioned access structure.

[0215] By periodically reassessing the data loss, sub-observation sets, statistics, uncertainty terms, completeness terms, validity terms, and overall quality scores of all units, along with their global statistics, and updating partitions and access priorities accordingly, the system maintains the continuity and integrity of the time series. When a partition is downgraded from high or medium to low, a complete copy of the previous period's records is retained; when it is upgraded from low to medium or high, the entire record of the current period is supplemented, resolving the problem of discontinuous historical data or missing important data caused by static storage strategies in long-term monitoring. Under a dynamic partitioning strategy, it simultaneously considers storage optimization and time series consistency, ensuring that the integrity of any data is not compromised during state changes through bidirectional replication and supplementation mechanisms.

[0216] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0217] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A big data management method for laser radar and total station cooperative monitoring, characterized in that, The method comprises the following steps: A unified number and discrete time index are constructed, and laser radar distance and reflection intensity, total station elevation and viewing angle are recorded to generate a unified structure record; The same number of current sub-observations is summarized and classified according to the equipment. When the sub-observation exists, the mean value and variance type uncertainty are calculated and the completeness is counted. When there is a missing value, the meter dimension component is backfilled according to the latest historical value. When there is no historical value, it is set to zero. The measurement reference scale is set, the uncertainty is converted into a reverse uncertainty term, the completeness term is retained, the reflection intensity is processed for missing, upper limit clipping and normalization, and the viewing angle is processed for missing and range clipping; The reverse uncertainty, completeness, validity and the like are aggregated into a comprehensive quality score, and a score sequence is formed according to the number; The global average score and standard deviation are calculated, and the value higher than the average value plus one standard deviation is classified into a high priority area, the value within one standard deviation is classified into a medium priority area, and the value lower than the average value minus one standard deviation is classified into a low priority area; The high priority area is stored in full, the medium priority area is sampled in ascending order of number at equal intervals, and the low priority area is selected at intervals in ascending order and quantized to an integer after the resolution of the meter dimension component before being stored in the database; An index containing the number, time and partition label is generated for each partition, and a priority value is established and scheduled in the order from high to low for query; The missing value, sub-observation, statistical quantity and score are re-evaluated at fixed intervals, and the partition and priority are updated. When the value from high or medium to low is retained, the last copy of the previous period is retained. When the value from low to medium or high is added, the full record of the current fixed interval is added, and the sequence is kept continuous; The method comprises the following steps: The laser radar sampling unit and the total station sampling unit are configured, and the device number list of each is established according to the continuous natural number respectively; A time index sequence composed of discrete receiving time is established, and a batch number is allocated to each batch of data; The laser radar records distance representative value and reflection intensity at each discrete time, and the observation vector of the time is formed in the order; The total station records elevation representative value and viewing angle at each discrete time, and the observation vector of the time is formed in the order; The numbers of the two types of equipment are mapped into a unified numbering system, so that any unified number corresponds to a complete observation record at any time. 2.The big data management method for the cooperation of laser radar and total station monitoring according to claim 1, wherein, The method comprises the following steps: All sub-observation items of the batch are summarized for any unified number at any discrete time to form a sub-observation set of the batch; The items in the set are classified according to the equipment type, and the laser radar item is taken as the distance measurement item, and the total station item is taken as the elevation measurement item; When there are items in the set, the arithmetic mean value of the batch of sub-observations is calculated as the representative value at the time; When there are items in the set, the variance type uncertainty of the batch of sub-observations is calculated as the uncertainty index at the time; The completeness of the sub-observation of the batch is counted, and a percentage is given according to the proportion of the number of sub-observations to the maximum number of system expectations. When the set is empty, the completeness is zero. When the set is empty, perform deterministic backfilling on components with meter dimension: take the last defined historical value when there is historical record, or set to zero when there is no historical record; the deterministic backfilling rules apply to the laser radar distance component and the total station height component respectively. 3.The big data management method for the cooperation of the laser radar and the total station according to claim 2, wherein, The setting measurement reference scale, converting uncertainty into reverse uncertainty term, keeping integrity term, performing missing processing, upper limit clipping and normalization on reflection intensity, performing missing processing and range clipping on view angle, specifically includes: Set measurement reference scale for each uniform number; Convert the uncertainty into reverse uncertainty term through monotonic bounded transformation, and set the term to zero when the set is empty; The integrity is taken as the integrity term to participate in subsequent calculation; Perform missing processing and rated upper limit clipping on laser radar reflection intensity, and then normalize according to the rated upper limit to obtain the effectiveness term at this moment; Perform missing processing and range clipping on total station view angle to keep the value within the range of zero to one hundred and eighty degrees.

4. The big data management method for the cooperation of laser radar and total station monitoring according to claim 3, characterized in that, The reverse uncertainty, integrity, effectiveness and other weights are aggregated into comprehensive quality score, and the score sequence is formed according to the number, specifically including: Aggregate the reverse uncertainty term, integrity term and effectiveness term into comprehensive quality score according to equal weight; The comprehensive quality scores of all units are grouped into a score sequence in order of uniform number at the same time.

5. The big data management method for the cooperation of laser radar and total station monitoring according to claim 4, characterized in that, The global average score and standard deviation are calculated, and the high priority area is divided into more than one standard deviation above the average value, the medium priority area is within one standard deviation of the average value, and the low priority area is below one standard deviation of the average value, specifically including: Calculate the global average quality score at this moment; Calculate the standard deviation of the quality score at this moment; When the unit score is higher than one standard deviation above the global average, it is marked as high priority area, when the unit score is within one standard deviation of the global average, it is marked as medium priority area, and when the unit score is lower than one standard deviation below the global average, it is marked as low priority area.

6. The big data management method for the cooperation of laser radar and total station monitoring according to claim 5, characterized in that, The high priority area is stored in full, the medium priority area is sampled in ascending order of number at least one, and the low priority area is selected in ascending order of interval and quantized to integer after resolution of the meter dimension component and then stored, specifically including: Perform full storage on the high priority area; After forming the candidate set, a deterministic sequence is established in ascending order of uniform number, the number of sampling records is determined according to the sampling ratio parameter of the medium priority area, the sampling record set is obtained by equidistant selection and stored; After forming the candidate set, set the effective compression rate parameter and calculate the selected number of at least one, and select the selected record set in ascending order of uniform number by deterministic interval selection; The meter dimension component in the selected record is quantized by setting the quantization step according to the device resolution, and the non-meter dimension component is range clipped to generate the compressed record and store it.

7. The big data management method for the cooperation of laser radar and total station monitoring according to claim 6, characterized in that, Generate an index containing number, time and partition label for each partition, establish priority value and schedule query in high to low order, specifically including: Generate an index table for each partition, and the index item contains uniform number, time and partition label; Map the partition to access priority value, the high priority area corresponds to the highest level, the medium priority area corresponds to the medium level, and the low priority area corresponds to the low level; The data blocks in the high priority area are retrieved first, and then the data blocks in the medium priority area and the low priority area are retrieved in order of priority from high to low when the query task is executed. 8.The method according to claim 7, wherein, The missing data, sub-observations, statistics, and scores are re-evaluated at fixed intervals, and the partition and priority are updated. When the priority is reduced from high or medium to low, the previous copy is retained. When the priority is increased from low to medium or high, the full records of the current fixed interval are added, and the sequence is kept continuous. Specifically, the method comprises the following steps: The missing data, sub-observation set, sub-observation mean, standard deviation, inverse uncertainty term, completeness term, effectiveness term, and comprehensive quality score of each uniform number within the adjacent two intervals are recalculated at fixed time intervals. The global average quality score and standard deviation are also calculated. The partition and access priority of each uniform number are updated according to the re-evaluation results. When any uniform number is transferred from the high priority area or the medium priority area to the low priority area between the adjacent two re-evaluation intervals, the complete record copy of the previous interval is retained. When it is upgraded from the low priority area to the medium priority area or the high priority area, the full record of the interval is added to ensure the continuity and integrity of the time series.

Images