A statistical method and system for large satellite remote sensing data sets

Abstract

The present invention discloses a satellite remote sensing large data set statistical method and system, which relates to the field of remote sensing science and technology. The method comprises the following steps: obtaining observation data points of a satellite remote sensing large data set; calculating the lunar phase angle of the observation data points, matching and optimizing reflectivity reference values ​​from the lunar phase correction table of the International Space Survey Research System (ILRS) based on the lunar phase angle; using the optimized reflectivity reference values ​​as normalization coefficients to perform normalization conversion on the reflectivity values ​​of the observation data points to generate a benchmark unified data set; performing time matching on the benchmark unified data set and the NASA SWPC event library, marking successfully matched observation data points as interstellar interference data, and integrating unmatched observation data points into a weighted data set; performing full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, fusing the full-precision and fast statistical results, and generating a statistical report. The present invention eliminates statistical bias caused by lunar phase changes by intelligently optimizing the reflectivity reference value calibration process.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing science and technology, and in particular to a statistical method and system for a large satellite remote sensing data set. Background Art

[0002] With the rapid development of earth observation technology, satellite remote sensing has become an important means of obtaining information about the earth's surface. Various satellite sensors around the world continue to acquire data, showing explosive growth in multiple dimensions, forming a large satellite remote sensing data set with huge volume, diverse structure and high update frequency. Therefore, statistics on large satellite remote sensing data sets have become increasingly important. The conventional method is usually to use the available computing resources and sharding strategies of the distributed computing framework to perform local statistical calculations on the maximum value, minimum value, sum, number of pixels, median variance and histogram distribution of the massive remote sensing image files to be counted. After the calculation is completed, the statistical results are submitted to the summary center, and the summary center performs a global summary operation to obtain the global distribution.

[0003] However, conventional statistical methods still face challenges in their implementation. Conventional methods rely on loading the data as a whole into memory for calculation, and require writing multi-step processing flow scripts. The execution process consumes huge resources, is time-consuming, and has complex processes. In addition, they lack the ability to perform targeted and differentiated processing on the intrinsic characteristics of the data and external interference, which reduces the efficiency and accuracy of statistics. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a statistical method for large satellite remote sensing data sets to solve the problem of lack of targeted differentiated processing capabilities for intrinsic characteristics of data and external interference.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for statistically analyzing a large satellite remote sensing data set, which includes obtaining observation data points of a large satellite remote sensing data set; calculating the lunar phase angle of the observation data point, and matching and optimizing the reflectivity reference value from the lunar phase correction table of the ILRS based on the lunar phase angle; using the optimized reflectivity reference value as a normalization coefficient, performing normalization conversion on the reflectivity value of the observation data point, and generating a benchmark unified data set; performing time matching on the benchmark unified data set and the NASA SWPC event library, marking the observation data points that have been successfully matched as interstellar interference data, and integrating the unmatched observation data points into a weighted data set; performing full-precision statistics on the weighted data set and performing fast statistics on the interstellar interference data, fusing the full-precision statistics and fast statistics results, and generating a statistical report with an interstellar interference mark.

[0008] As a preferred embodiment of the satellite remote sensing large data set statistical method of the present invention, the steps of calculating the lunar phase angle of the observation data point and matching and optimizing the reflectivity reference value from the ILRS lunar phase correction table based on the lunar phase angle are as follows:

[0009] The moon phase angle of the observed data points is calculated using the Meeus astronomical algorithm;

[0010] Extract the initial reflectance reference value corresponding to the lunar phase angle from the ILRS lunar phase correction table;

[0011] The initial reflectivity benchmark value is input into the pre-trained convolutional neural network, and the optimized reflectivity benchmark value is generated after cross-channel feature interaction and residual correction.

[0012] As a preferred solution of the satellite remote sensing large data set statistical method of the present invention, wherein: the optimized reflectivity reference value is used as a normalization coefficient to perform normalization conversion on the reflectivity values ​​of the observation data points to generate a benchmark unified data set, the specific steps are:

[0013] Convert the optimized reflectance benchmark value into a semantic format;

[0014] Query the association information between the observed data points and the optimized reflectivity benchmark values ​​through the semantic network to generate semantic relationships;

[0015] Dynamically partition observation data points based on semantic relationships and transfer the partitioned observation data points to computing nodes;

[0016] In the calculation node, the optimized reflectivity reference value is used as the normalization coefficient to perform normalization transformation on the reflectivity value of each partition observation data point to generate a normalized data point set;

[0017] After a distributed consistency check is performed on the normalized data point set, it is aggregated into a benchmark unified dataset by timestamp and geographic coordinates.

[0018] As a preferred solution of the satellite remote sensing large data set statistical method of the present invention, the benchmark unified data set and the NASA SWPC event library are time-matched, the observation data points with successful matching are marked as interstellar interference data, and the unmatched observation data points are integrated into a weighted data set. The specific steps are:

[0019] Convert event records in the NASA SWPC event library into semantic event nodes and integrate them into a distributed graph database;

[0020] Execute distributed parallel queries. If the timestamp of the observed data point meets the time matching criteria in the distributed graph database, it is marked as interstellar interference data and associated with the corresponding semantic event node.

[0021] Extract observation data points that are not matched to any semantic event nodes and assign them a baseline weight value;

[0022] The observation data points assigned with benchmark weight values ​​are distributed and partitioned according to the preset spatiotemporal dimension partitioning rules and integrated into a weighted data set.

[0023] As a preferred solution of the satellite remote sensing large data set statistical method of the present invention, the time matching judgment condition is to calculate the time difference between the timestamp of the observation data point and the event time range in the NASA SWPC event library, compare the time difference with the preset matching threshold, and determine whether the match is successful;

[0024] The preset spatiotemporal dimension partitioning rule refers to a rule for partitioning observation data points assigned with reference weight values ​​according to timestamps and geographic coordinates.

[0025] As a preferred solution of the satellite remote sensing large data set statistics method of the present invention, wherein: performing full-precision statistics on the weighted data set refers to performing multi-dimensional full-precision statistics on the weighted data set using a distributed computing framework;

[0026] The performing fast statistics on the interstellar interference data refers to applying parallel sampling to perform fast statistics on the interstellar interference data;

[0027] The fusing of full-precision statistics and fast statistics results to generate a statistics report with an interstellar interference mark refers to generating a statistics report with an interstellar interference mark by weighted fusing of full-precision statistics results and fast statistics results.

[0028] As a preferred solution of the satellite remote sensing large data set statistics method of the present invention, wherein: the application of parallel sampling to quickly count interstellar interference data is specifically carried out as follows:

[0029] Dynamically assign sampling weight coefficients based on the semantic event node types associated with interstellar interference data;

[0030] Using GPU parallel architecture, the interstellar interference data is partitioned into space-time grids, and a weight-aware Latin hypercube sampling algorithm is used to extract subsamples of the interstellar interference data.

[0031] Perform multi-scale wavelet transform on the sub-samples to extract the time-space frequency domain eigenvalues ​​of the sub-samples;

[0032] Aggregate spatiotemporal and frequency domain eigenvalues ​​into fast statistical results.

[0033] In a second aspect, the present invention provides a satellite remote sensing large data set statistics system, comprising:

[0034] The acquisition module is used to obtain observation data points of large satellite remote sensing data sets;

[0035] Correction module, used to calculate the lunar phase angle of the observation data point, match and optimize the reflectivity reference value from the ILRS lunar phase correction table based on the lunar phase angle;

[0036] The normalization module is used to use the optimized reflectivity reference value as the normalization coefficient to perform normalization transformation on the reflectivity value of the observation data point to generate a benchmark unified data set;

[0037] The classification module is used to time-match the benchmark unified dataset and the NASA SWPC event library, marking the observation data points that are successfully matched as interstellar interference data, and integrating the unmatched observation data points into a weighted dataset;

[0038] The statistics module is used to perform full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, and to generate a statistical report with an interstellar interference mark by fusing the full-precision statistics and fast statistics results.

[0039] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the satellite remote sensing large data set statistical method as described in the first aspect of the present invention is implemented.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the satellite remote sensing large data set statistical method as described in the first aspect of the present invention.

[0041] The beneficial effects of the present invention are as follows: the present invention eliminates statistical deviations caused by lunar phase changes and reflectivity fluctuations caused by lunar phase cycles through intelligent optimization of the reflectivity reference value calibration process, making observation data points in different times and regions directly comparable. The optimized emissivity reference value reduces background noise, significantly improves the signal-to-noise ratio of weak signals in interstellar particle events, ensures the long-term reliability of monitoring, avoids manual correction errors, and greatly reduces calculation pressure and statistical errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 Flowchart of the statistical method for large satellite remote sensing datasets.

[0044] Figure 2 Schematic diagram of the statistical system for large satellite remote sensing data sets.

[0045] Figure 3 Flowchart for the optimized reflectivity benchmark value.

[0046] Figure 4 Flowchart for benchmarking unified dataset. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0050] Reference Figures 1 to 4 , is an embodiment of the present invention, which provides a method for statistically analyzing a large satellite remote sensing data set, comprising the following steps:

[0051] S1. Obtain observation data points of satellite remote sensing large data sets.

[0052] Read observation data points of large satellite remote sensing datasets from the distributed file HDFS, where the data scale of observation data points ranges from tens of millions to hundreds of billions;

[0053] Each observation data point contains the observation time (UTC format, accurate to milliseconds), geographic coordinates (longitude and latitude, accurate to 0.01 degrees, for example, longitude 120.45°E, latitude 30.25°N), sensor band (range 400 to 2500 nm, for example, 550 nm), and land surface type (classified as ocean, land, desert, and forest);

[0054] Use the Apache Spark distributed computing framework to read observation data points by parallel tasks in slices based on observation time. For example, slices are divided into hourly slices, with each slice containing approximately 10 million observation data points.

[0055] The preprocessing of observation data points includes integrity check, standardization and denoising;

[0056] In Apache Spark, we perform integrity checks on the observation data points in parallel, remove missing and invalid observation data points, and assign a unique observation ID to each valid observation data point, as follows:

[0057] Check whether the observation time is in UTC format and not missing. For example, if the observation time is null or not in UTC format, it is invalid.

[0058] Verify that the geographic coordinates are within a reasonable range, which is longitude -180° to 180° and latitude -90° to 90°;

[0059] Make sure the sensor band is within the range of 400 to 2500 nm. For example, 550 nm is valid but 300 nm is not.

[0060] Ensure that land surface types fall into predefined categories, namely ocean, land, desert, and forest;

[0061] The corrected observation data points are formatted and standardized as follows:

[0062] Standardize observation times to ISO 8601 format;

[0063] Convert geographic coordinates to WGS84 standard, retaining two decimal places, for example, 120.45°E, 30.25°N;

[0064] Round the sensor band to the nearest whole nanometer, for example, 550.7 nm is normalized to 551 nm;

[0065] Encode land surface types as uniform numerical identifiers, for example, ocean = 1, land = 2, desert = 3, forest = 4, with land coded as 2;

[0066] It should be noted that during the standardization process, the consistency of the observation data points was checked to ensure that there were no duplicate observation IDs;

[0067] The standardized observation data points are denoised as follows:

[0068] Use anomalous observations detected based on land surface type, for example, the land radiance threshold is 10 to 100 W / m² / sr, 50.5 W / m² / sr is normal, and 500 W / m² / sr is abnormal;

[0069] Eliminate abnormal data points. For example, if the radiance is 500W / m² / sr, mark it as an abnormal data point and remove it, and record it in the elimination log.

[0070] Apply median filtering to smooth non-abnormal observation data points. For example, take the median of five adjacent time points to smooth the observation data points to ensure the stability of the observation data points.

[0071] S2. Calculate the lunar phase angle of the observed data point, and match and optimize the reflectivity reference value from the ILRS lunar phase correction table based on the lunar phase angle.

[0072] The pre-processed observation data points are extracted from the satellite remote sensing large data set, and the Meeus astronomical algorithm is used to query the JPL DE430 ephemeris to determine the relative positions of the moon, the earth, and the sun. The angle of the moon relative to the earth and the sun is calculated to generate the lunar phase angle of the observation data point. The formula is:

[0073] ;

[0074] in, Indicates the moon phase angle (unit: degree), represents the vector from the Earth to the Moon (unit: kilometers), represents the vector from the moon to the sun (unit: kilometers), represents the vector length from the Earth to the Moon (unit: kilometers), represents the vector length from the moon to the sun (unit: kilometers), Represents the arccosine function, which converts the cosine value into the moon phase angle (unit: degree). An index variable representing the Earth, The index variable representing the moon, An index variable representing the sun;

[0075] In Apache Spark, the system shards the observation time and checks whether the moon phase angle is within the valid range (valid range: 0° to 180°) to obtain the valid moon phase angle.

[0076] We use the Apache Cassandra distributed database to query the ILRS lunar phase correction table, locate candidate partitions based on the effective lunar phase angle, and extract the initial reflectance reference value corresponding to the lunar phase angle. Specifically:

[0077] Use a distributed query tool to find the initial reflectivity reference value corresponding to the lunar phase angle from the lunar phase correction table provided by the International Laser Ranging Service (ILRS);

[0078] The ILRS lunar phase correction table is a predefined database that stores atmospheric reflectivity correction coefficients at different lunar phase angles. The unit is dimensionless reflectivity coefficient.

[0079] Among them, the initial reflectivity reference value of the missing moon phase angle is obtained by linear interpolation method. For example, if the moon phase angle is 45.3°, but there are only records of 45° and 46° in the moon phase correction table, the corresponding initial reflectivity reference value is calculated by interpolation;

[0080] The initial reflectivity benchmark value is input into the pre-trained convolutional neural network. After cross-channel feature interaction and residual correction, the optimized reflectivity benchmark value is generated. The specific steps are as follows:

[0081] The convolutional neural network consists of an input layer, an interactive convolution layer, a physical constraint convolution layer, a residual correction layer, and an output layer;

[0082] The input layer receives the initial reflectance reference value and the corresponding moon phase angle, and generates a reflectance feature vector set after normalization;

[0083] The initial reflectance reference value and the corresponding lunar phase angle are normalized by linear transformation to the range of 0 to 1 to reduce the impact of numerical scale differences on subsequent levels and generate a reflectance feature vector.

[0084] The interactive convolution layer captures the cross-channel feature interaction between the initial reflectivity reference value and the moon phase angle value through convolution operation to generate a reflectance feature map;

[0085] The interactive convolution layer uses multiple convolution kernels (such as 3×3 filters) to convolve the reflectance feature vector. Each convolution kernel focuses on capturing the cross-channel feature interaction between different reflectance feature vectors, such as the nonlinear correlation between the change in the moon phase angle and the reflectance baseline value.

[0086] Convolution processing refers to scanning each area of ​​the reflection feature vector through a sliding window mechanism to generate a reflection feature map that reflects the characteristic interaction relationship between the initial reflectivity reference value and the lunar phase angle value at different scales;

[0087] To enhance the capture effect of the reflection feature map, a maximum pooling operation is applied to reduce the spatial dimension of the reflection feature map. After pooling, the ReLU activation function is used to set negative values ​​to zero to increase the nonlinear expression ability and reduce the computational complexity while retaining the main interaction relationships.

[0088] The physical constraint convolution layer further optimizes the reflection feature map using the convolution kernel and predefined physical rules to obtain the optimized reflection feature map;

[0089] The physical constraint convolution layer receives the reflectance feature map output by the interactive convolution layer and incorporates physical constraints into the feature extraction process to ensure that the generated reflectance feature map conforms to the physical laws of satellite remote sensing data;

[0090] Physical constraints refer to further processing the feature map through specially designed convolution kernels combined with predefined physical rules, such as the theoretical relationship between the moon phase angle and atmospheric reflectivity;

[0091] Physical constraints are implemented through weight initialization. For example, the weights of the convolution kernels are preset according to the theoretical distribution of the ILRS lunar phase correction table, giving priority to enhancing reflection feature vectors that are consistent with physical laws, such as the changing trend of the reflectivity baseline value when the lunar phase angle is close to 0° and 180°.

[0092] To avoid overfitting, regularization mechanisms (such as L2 regularization) are used to limit excessive weights and ensure the generalization ability of convolutional neural networks under physical constraints;

[0093] The residual correction layer corrects the deviation in the optimized reflection feature map through the residual learning mechanism to generate a corrected reflection feature map;

[0094] The residual correction layer performs residual connection between the reflection feature map and the reflection feature vector to obtain the residual;

[0095] The residual is processed by convolution operation, and a small convolution kernel (e.g. 1×1) is used to fine-tune the reflection feature map to capture the details missed in the reflection feature map;

[0096] The fine-tuned residual is fused with the reflection feature map through an addition operation to generate a corrected reflection feature map;

[0097] The output layer compresses the spatial dimension of the corrected reflection feature map into a single reflection feature vector through a global average pooling operation, extracting the global information of each reflection feature map;

[0098] Based on the global information of the reflection feature map, the reflection feature vectors are linearly combined to generate an optimized reflectivity reference value;

[0099] To ensure that the reflectivity reference value meets the actual requirements, an inverse normalization operation is applied to map the optimized reflectivity reference value from the range of 0 to 1 back to the range of the original reflectivity reference value.

[0100] S3. Using the optimized reflectivity reference value as a normalization coefficient, normalize the reflectivity values ​​of the observed data points to generate a reference unified data set.

[0101] Convert the optimized reflectance benchmark value into a semantic format;

[0102] The optimized reflectance reference values ​​are divided into semantic labels using predefined mapping rules. For example, the reflectance reference values ​​are divided into three intervals: low (e.g., 0 to 0.3), medium (e.g., 0.3 to 0.7), and high (e.g., 0.7 to 1.0), which correspond to "low reflectance correction coefficient," "medium reflectance correction coefficient," and "high reflectance correction coefficient," respectively.

[0103] A hash lookup table is used for semantic format conversion. For example, the optimized reflectance baseline value of 0.6 is mapped to the "medium reflectance correction coefficient";

[0104] At the same time, the original values ​​are retained for subsequent calculations;

[0105] Query the association information between the observed data points and the optimized reflectivity benchmark values ​​through the semantic network to generate semantic relationships;

[0106] Input the semantically formatted reflectance benchmark values ​​into the semantic network, use the distributed graph database to query the association information between the reflectance benchmark values ​​and the observed data points, match the semantic reflectance benchmark values ​​of each observed data point, and form a semantic relationship, such as "the medium reflectance correction coefficient has high credibility in ocean areas and nighttime observation conditions";

[0107] Association information refers to the domain knowledge rules pre-stored in the distributed graph database, which is used to bind the reflectivity reference value with the spatiotemporal and physical properties of the observed data points;

[0108] The semantic network stores the association rules between reflectance benchmark values ​​and observation data points, such as surface type and observation time;

[0109] Dynamically partition observation data points based on semantic relationships and transfer the partitioned observation data points to computing nodes;

[0110] In the distributed computing framework, the K-means clustering algorithm is used to group the semantic feature vectors of the semantic relationship (such as surface type and time period), and each group forms a partition;

[0111] After partitioning, the observation data points of each partition are assigned to different computing nodes to ensure load balancing;

[0112] The observation data points of each partition include the original reflectance value, timestamp, geographic coordinates and semantic reflectance benchmark value;

[0113] Using the computing nodes of a distributed computing framework (such as Apache Spark), the optimized reflectance benchmark value is used as the normalization coefficient to normalize the original reflectance value of each observation data point. The formula is:

[0114] Normalized reflectance value = original reflectance value ÷ optimized reflectance reference value;

[0115] Normalization is performed in parallel on each computing node, with each node processing observation data points of a partition, ensuring efficient processing of large-scale data sets;

[0116] After the conversion is completed, a normalized data set is generated;

[0117] The generated normalized dataset is passed to the distributed computing framework for execution consistency check;

[0118] The consistency check verifies whether the normalized reflectance value is within a reasonable range (e.g., 0 to 10, considering that normalization may result in a value greater than 1), whether the timestamp is continuous (e.g., no missing or duplicated values), and whether the geographic coordinates are valid (e.g., longitude -180° to 180°, latitude -90° to 90°);

[0119] A hash-based check algorithm determines the distributed consistency protocol to ensure that the normalized data sets generated by different computing nodes are consistent within the scope and eliminate outliers (for example, observation data points with normalized reflectance values ​​outside the reasonable range);

[0120] After passing the check, the normalized data points are aggregated according to the ascending order of timestamp and geographic coordinates to generate a benchmark unified dataset.

[0121] S4. Time-match the benchmark unified dataset and the NASA SWPC event library, mark the observation data points that are successfully matched as interstellar interference data, and integrate the unmatched observation data points into a weighted dataset.

[0122] Extract timestamp information from the benchmark unified dataset;

[0123] Based on the timestamp information, query the NASA Space Weather Prediction Center SWPC event database to obtain event records;

[0124] Event records include the time range, event type, and event intensity of solar activity and geomagnetic disturbance events;

[0125] Use Drools to convert each event record into a semantic event node;

[0126] The converted semantic event nodes are stored in a distributed graph database (such as Neo4j). The association relationship between semantic event nodes is established based on event type and time range. For example, flare events of the same type are connected as subgraphs.

[0127] Perform distributed parallel queries in the distributed graph database, calculate the difference between the timestamp of each observation data point and the time range of the semantic event node, and determine whether the time matching criteria are met;

[0128] The judgment condition is as follows: calculate the time difference between the timestamp of the observation data point and the time range of the event in the NASA SWPC event library, compare the time difference with the preset matching threshold, and determine whether the match is successful;

[0129] For example, when the time difference is less than a preset matching threshold (e.g., 300 seconds), that is, if the time difference between the timestamp of the observed data point within the event time range and the boundary is less than 300 seconds, the match is determined to be successful;

[0130] Mark the successfully matched observation data points as interstellar interference data and bind them to the semantic event nodes corresponding to the interstellar interference data, such as "strong flare, high intensity";

[0131] Interstellar interference data refers to observation data points marked as interstellar interference, including normalized reflectivity values, timestamps, geographic coordinates, and associated semantic event nodes;

[0132] It should be noted that when executing queries in parallel in a distributed computing framework, multiple computing nodes simultaneously process observation data points from different time periods, improving efficiency;

[0133] If the time difference is greater than the preset matching threshold (e.g. 300 seconds), it is determined that the match is unsuccessful;

[0134] Extract observation data points that do not match any semantic event nodes to form unmatched observation data points;

[0135] Unmatched observation data points include normalized reflectance values, timestamps, and geographic coordinates;

[0136] For each unmatched observation data point, a baseline weight value (e.g., 1.0) is assigned to represent the weight without the influence of interstellar interference. For example, all unmatched observation data points are uniformly assigned a value of 1.0;

[0137] After the assignment is completed, a set of unmatched observation data points with weights is generated;

[0138] The observation data point set assigned with the benchmark weight value is distributed and partitioned according to the preset time and space dimension partitioning rules and integrated into a weighted data set;

[0139] Distribute and partition the observation data points assigned with benchmark weight values ​​according to the preset time and space dimension partitioning rules;

[0140] The space-time dimension partitioning rule refers to the two-dimensional partitioning based on timestamps and geographic coordinates;

[0141] The partitioning process is performed in a distributed computing framework (such as Apache Spark), using a hash partitioning algorithm to distribute observation data points to different computing nodes, ensuring that the data is evenly distributed and that each partition contains observation data points with similar spatiotemporal characteristics.

[0142] After the partitioning is completed, the unmatched observation data points in all partitions are integrated into a weighted dataset.

[0143] S5. Perform full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, integrate the full-precision statistics and fast statistics results, and generate a statistical report with an interstellar interference mark.

[0144] Perform multi-dimensional, full-precision statistics on weighted datasets in distributed computing frameworks such as Apache Spark;

[0145] Statistical dimensions include time dimension (e.g., aggregation by day), geographic dimension (e.g., aggregation by 10°×10° longitude and latitude grid), and reflectance dimension (e.g., aggregation by normalized reflectance value intervals of 0 to 1, 1 to 2, etc.);

[0146] Calculate statistical indicators for each statistical dimension, such as mean (the average value of normalized reflectance values ​​within a certain spatiotemporal grid), variance (reflecting the degree of data dispersion), median, and weighted average (based on a baseline weight value of 1.0);

[0147] The statistical process is executed in parallel on multiple computing nodes, and after the statistics are completed, full-precision statistical results are generated;

[0148] Performing fast statistics on interstellar interference data refers to applying parallel sampling to perform fast statistics on interstellar interference data;

[0149] According to the semantic event node type associated with the interstellar interference data, the sampling weight coefficient is dynamically allocated as follows:

[0150] Extract observation data points labeled as interstellar interference and corresponding semantic event nodes (e.g., “strong flare, high intensity” and “weak geomagnetic storm, low intensity”) from the interstellar interference dataset;

[0151] Semantic event nodes include semantic event type, semantic event intensity and duration;

[0152] The formula for the sampling weight coefficient is:

[0153] ;

[0154] in, represents the sampling weight coefficient, Represents the semantic event type coefficient, represents the semantic event intensity coefficient, Indicates duration, Indicates the contribution ratio of semantic event type to the sampling weight coefficient, Indicates the contribution ratio of event intensity to the sampling weight coefficient, Indicates the contribution ratio of duration to the sampling weight coefficient;

[0155] For example The default value is 0.5. The default value is 0.3. The default value is 0.2, and the specific weight is adjusted according to the actual situation;

[0156] For example, a strong solar flare event is assigned a high sampling weight coefficient (e.g., 2.0), while a weak geomagnetic storm event is assigned a low sampling weight coefficient (e.g., 0.5);

[0157] After the allocation is completed, an interstellar interference dataset with sampling weight coefficients is generated;

[0158] Using the GPU parallel architecture, the interstellar interference dataset is partitioned according to the space-time grid, and the weight-aware Latin hypercube sampling algorithm is used to extract subsamples of the interstellar interference data as follows:

[0159] Partitioning interstellar interference data into space-time grids on GPU parallel architecture;

[0160] The spatiotemporal grid is defined as a temporal dimension (e.g., grouped by hours) and a geographic dimension (e.g., grouped by a 10° × 10° longitude and latitude grid);

[0161] After partitioning, a weight-aware Latin hypercube sampling algorithm is applied to extract subsamples from each spatiotemporal grid according to the sampling weight coefficient (e.g., 2.0 or 0.5), and stratified sampling is used to ensure the representativeness of the subsamples in terms of time and geography.

[0162] After the sampling is completed, a subsample of interstellar interference data is generated;

[0163] The Daubechies wavelet basis function is used to perform multi-scale wavelet transform on the interstellar interference data subsamples, decompose the normalized reflectivity values ​​of the subsamples into frequency domain components of different scales, and extract the spatiotemporal frequency domain eigenvalues ​​of the subsamples.

[0164] The spatiotemporal frequency domain characteristic values ​​refer to the low-frequency trends (reflecting long-term changes, such as diurnal cycles) and high-frequency fluctuations (reflecting short-term disturbances, such as the transient impact of flares) in the time dimension, as well as the spatial characteristics in the geographic dimension (such as regional reflectivity changes);

[0165] Aggregate the spatiotemporal and frequency domain eigenvalues ​​into fast statistical results;

[0166] Extract the spatiotemporal frequency domain feature vector of each subsample from the spatiotemporal frequency domain feature values, and perform aggregation operations in a distributed computing framework to generate fast statistical results;

[0167] The aggregation process calculates statistical indicators of spatiotemporal frequency domain eigenvalues, such as mean, standard deviation, and frequency domain energy distribution;

[0168] Use the weighted average method to fuse the full-precision statistical results and the fast statistical results to generate a statistical report with interstellar interference marks;

[0169] The fusion process assigns weights based on the importance of the data type. For example, the full-precision statistical results of the weighted dataset are given a high weight (e.g., 0.7), while the rapid statistical results of the interstellar interference data are given a low weight (e.g., 0.3), in order to balance the comprehensiveness and the specificity of the interference event.

[0170] The statistical report includes full-precision statistical results, fast statistical results, and interstellar interference data markers, such as strong flare correlation data.

[0171] This embodiment also provides a satellite remote sensing large data set statistics system, including:

[0172] The acquisition module is used to obtain observation data points of large satellite remote sensing data sets;

[0173] Correction module, used to calculate the lunar phase angle of the observation data point, match and optimize the reflectivity reference value from the ILRS lunar phase correction table based on the lunar phase angle;

[0174] The normalization module is used to use the optimized reflectivity reference value as the normalization coefficient to perform normalization conversion on the reflectivity value of the observation data point to generate a benchmark unified data set;

[0175] The classification module is used to time-match the benchmark unified dataset and the NASA SWPC event library, marking the observation data points that are successfully matched as interstellar interference data, and integrating the unmatched observation data points into a weighted dataset;

[0176] The statistics module is used to perform full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, and to generate a statistical report with an interstellar interference mark by fusing the full-precision statistics and fast statistics results.

[0177] This embodiment also provides a computer device suitable for the case of a satellite remote sensing large data set statistical method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the satellite remote sensing large data set statistical method proposed in the above embodiment.

[0178] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0179] This embodiment also provides a storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements the method for implementing satellite remote sensing large data set statistics as proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0180] In summary, the present invention eliminates statistical deviations caused by lunar phase changes and reflectivity fluctuations caused by the lunar phase cycle through intelligent optimization of the reflectivity reference value calibration process, making observation data points at different times and regions directly comparable. The optimized emissivity reference value reduces background noise, significantly improves the signal-to-noise ratio of weak signals in interstellar particle events, ensures the long-term reliability of monitoring, avoids manual correction errors, and greatly reduces calculation pressure and statistical errors.

[0181] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A statistical method for large satellite remote sensing data sets, characterized by: include, Obtain observation data points from large satellite remote sensing datasets; Calculate the lunar phase angle of the observed data point, and match and optimize the reflectance reference value from the ILRS lunar phase correction table based on the lunar phase angle; The optimized reflectivity benchmark value is used as the normalization coefficient to perform normalization transformation on the reflectivity values ​​of the observed data points to generate a benchmark unified data set; The benchmark unified dataset and the NASA SWPC event library are time-matched, and the observation data points that are successfully matched are marked as interstellar interference data. The unmatched observation data points are integrated into a weighted dataset. Perform full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, fuse the full-precision statistics and fast statistics results, and generate a statistical report with interstellar interference tags.

2. The satellite remote sensing large data set statistical method according to claim 1, wherein: The calculation of the lunar phase angle of the observation data point and the matching and optimization of the reflectivity reference value from the lunar phase correction table of the ILRS based on the lunar phase angle are specifically carried out as follows: The moon phase angle of the observed data points is calculated using the Meeus astronomical algorithm; Extract the initial reflectance reference value corresponding to the lunar phase angle from the ILRS lunar phase correction table; The initial reflectivity benchmark value is input into the pre-trained convolutional neural network, and the optimized reflectivity benchmark value is generated after cross-channel feature interaction and residual correction.

3. The satellite remote sensing large data set statistical method according to claim 1, wherein: The optimized reflectivity reference value is used as a normalization coefficient to perform normalization conversion on the reflectivity value of the observation data point to generate a benchmark unified data set. The specific steps are: Convert the optimized reflectance benchmark value into a semantic format; Query the association information between the observed data points and the optimized reflectivity benchmark values ​​through the semantic network to generate semantic relationships; Dynamically partition observation data points based on semantic relationships and transfer the partitioned observation data points to computing nodes; In the calculation node, the optimized reflectivity reference value is used as the normalization coefficient to perform normalization transformation on the reflectivity value of each partition observation data point to generate a normalized data point set; After a distributed consistency check is performed on the normalized data point set, it is aggregated into a benchmark unified dataset by timestamp and geographic coordinates.

4. The satellite remote sensing large data set statistical method according to claim 1, wherein: The benchmark unified data set and the NASA SWPC event library are time-matched, and the observation data points that are successfully matched are marked as interstellar interference data. The unmatched observation data points are integrated into a weighted data set. The specific steps are: Convert event records in the NASA SWPC event library into semantic event nodes and integrate them into a distributed graph database; Execute distributed parallel queries. If the timestamp of the observed data point meets the time matching criteria in the distributed graph database, it is marked as interstellar interference data and associated with the corresponding semantic event node. Extract observation data points that are not matched to any semantic event nodes and assign them a baseline weight value; The observation data points assigned with benchmark weight values ​​are distributed and partitioned according to the preset spatiotemporal dimension partitioning rules and integrated into a weighted data set.

5. The satellite remote sensing large data set statistical method according to claim 4, wherein: The time matching judgment condition is to calculate the time difference between the timestamp of the observation data point and the time range of the event in the NASA SWPC event library, compare the time difference with the preset matching threshold, and determine whether the match is successful; The preset spatiotemporal dimension partitioning rule refers to a rule for partitioning observation data points assigned with reference weight values ​​according to timestamps and geographic coordinates.

6. The satellite remote sensing large data set statistical method according to claim 1, wherein: The performing full-precision statistics on the weighted data set refers to performing multi-dimensional full-precision statistics on the weighted data set using a distributed computing framework; The performing fast statistics on the interstellar interference data refers to applying parallel sampling to perform fast statistics on the interstellar interference data; The fusing of full-precision statistics and fast statistics results to generate a statistics report with an interstellar interference mark refers to generating a statistics report with an interstellar interference mark by weighted fusing of full-precision statistics results and fast statistics results.

7. The method for statistically analyzing a large satellite remote sensing data set according to claim 6, wherein: The application of parallel sampling to quickly count interstellar interference data includes the following specific steps: Dynamically assign sampling weight coefficients based on the semantic event node types associated with interstellar interference data; Using GPU parallel architecture, the interstellar interference data is partitioned into space-time grids, and a weight-aware Latin hypercube sampling algorithm is used to extract subsamples of the interstellar interference data. Perform multi-scale wavelet transform on the sub-samples to extract the time-space frequency domain eigenvalues ​​of the sub-samples; Aggregate spatiotemporal and frequency domain eigenvalues ​​into fast statistical results.

8. A satellite remote sensing large data set statistics system, based on the satellite remote sensing large data set statistics method according to any one of claims 1 to 7, characterized in that: include, The acquisition module is used to obtain observation data points of large satellite remote sensing data sets; Correction module, used to calculate the lunar phase angle of the observation data point, match and optimize the reflectivity reference value from the ILRS lunar phase correction table based on the lunar phase angle; The normalization module is used to use the optimized reflectivity reference value as the normalization coefficient to perform normalization transformation on the reflectivity value of the observation data point to generate a benchmark unified data set; The classification module is used to time-match the benchmark unified dataset and the NASA SWPC event library, marking the observation data points that are successfully matched as interstellar interference data, and integrating the unmatched observation data points into a weighted dataset; The statistics module is used to perform full-precision statistics on the weighted data set and fast statistics on the interstellar interference data, and to generate a statistical report with an interstellar interference mark by fusing the full-precision statistics and fast statistics results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the satellite remote sensing large data set statistics method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the satellite remote sensing large data set statistics method according to any one of claims 1 to 7 are implemented.

Images