Data analysis method and device fusing large number rule, equipment and medium

Abstract

The invention relates to the technical field of data processing, and particularly discloses a data analysis method and device fusing a large number rule, equipment and a medium, and the method comprises the steps: determining a minimum convergence amount through hierarchical aggregation; calculating a cumulative sample mean trajectory and carrying out convergence diagnosis; performing noise attenuation weighting based on LLN convergence characteristics; performing distribution and component decomposition under steady moment constraint; performing consistency check and re-extraction robustness of LLN guidance; lLN-constrained model fitting and uncertainty calibration output are carried out; according to the method, based on noise attenuation weighting of LLN convergence characteristics, samples which are close to a steady state obtain greater influence in estimation, and unstable or sparse samples are naturally weakened, so that self-adaptive suppression of heterogeneity and small sample noise is realized, and the robustness of overall estimation is improved; steady moment constraint is introduced into distribution / component decomposition, it can be guaranteed that components obtained through decomposition are consistent with observation convergence characteristics in the aspect of high-order statistics, and component mismatching caused by extreme values or local fluctuation is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a data analysis method, device, equipment and medium fusing the law of large numbers. BACKGROUND

[0002] In today's data-driven era, data analysis plays a crucial role in many fields such as finance, medicine, and scientific research. Accurate and robust data analysis methods can help us extract valuable information from massive and complex data, thereby supporting decision-making, model building, and trend prediction. However, existing data analysis methods have exposed a series of deficiencies that need to be addressed when facing complex data scenarios in practical applications.

[0003] Traditional stratified analysis methods (such as stratified sampling and stratified regression) can capture heterogeneity by dividing data by time, space, or attributes, but do not incorporate the convergence condition of the law of large numbers into stratified quality control. Such methods often only focus on the semantic rationality of hierarchical division, but ignore whether the sample size of each stratification meets the statistical stability requirement. When some sub-populations are sparse after stratification, there may be a lack of representative samples within the layer, and the sampling error may increase rapidly. For example, in small data set stratification, traditional methods may mechanically divide the hierarchy, resulting in a sample size of some stratifications that is much lower than the critical value required for estimating the overall parameters, causing distortion of the within-layer variance estimate and systematic bias in subsequent aggregation analysis. This "heavy division, light stability" defect creates hidden dangers in data quality from the source, leading to the continuous propagation of uncertainty in subsequent processes.

[0004] To address the above problems, the present application proposes a data analysis method, device, equipment and medium fusing the law of large numbers. SUMMARY

[0005] The purpose of the present application is to provide a data analysis method, device, equipment and medium fusing the law of large numbers to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] The data analysis method fusing the law of large numbers comprises:

[0008] The original observation data is stratified and aggregated according to a pre-set stratification standard to estimate and determine the minimum sample size n0 of each stratification to achieve statistical stability (so that σ^2 / n0≤ε or the equivalent convergence criterion is met), obtaining a stabilized subsample set for each stratification;

[0009] inputting the stabilized sub-sample set of each layer, calculating the sample mean and sample variance trajectory accumulated with sample number of each layer sample, for determining the convergence window and outputting the convergence rate r and convergence confidence band of each layer, obtaining the layer mean trajectory with convergence diagnostic information;

[0010] inputting the layer mean trajectory with convergence diagnostic information, assigning weight wi to the sample or sample block according to the convergence rate r and confidence band of each layer or sample for adjusting the sample influence (for example, wi∝min{1,f(r,CI)}), obtaining the weighted sample set after weighting;

[0011] inputting the weighted sample set after weighting, using the distribution decomposition or spectral decomposition method with steady-state moment constraint for extracting stable components, for decomposing the weighted sample into several steady-state sub-distributions or principal components, obtaining the steady-state component set;

[0012] inputting the steady-state component set, adjusting the bootstrap intensity based on the effective sample size neff estimated by the law of large numbers and repeating the sampling verification, for retaining the components and parameters that remain stable in most bootstrap, obtaining the robust components and their confidence intervals retained by consistency test;

[0013] inputting the robust components and their confidence intervals retained by consistency test, performing final parameter estimation and model fitting, the fitting introduces a regularization term in the loss function based on the convergence characteristics of LLN for punishing the sensitivity of parameters to the sample mean deviation of sub-block, and combines the model prediction with the uncertainty of LLN calibration for generating decision-making statistical output, obtaining the final robust model parameters, prediction function and confidence interval calibrated by LLN.

[0014] Preferably, the layer aggregation and minimum convergence quantity determination is performed on the original observation in the time, space or attribute dimension for non-overlapping layering, for calculating the sample variance estimation σ^2 and determining the minimum sample quantity n0 in each layer to meet:

[0015] σ^2 / n0≤ε

[0016] wherein, ε is the maximum allowed estimation variance;

[0017] obtaining the stabilized sub-sample set of each layer.

[0018] Preferably, the inputting the stabilized sub-sample set of each layer, calculating the sample mean and sample variance trajectory accumulated with sample number of each layer sample, for determining the convergence window and outputting the convergence rate r and convergence confidence band of each layer, obtaining the layer mean trajectory with convergence diagnostic information, includes:

[0019] Cumulative mean at the kth sample is calculated using sliding cumulative method Cumulative variance for fitting the convergence rate model (e.g. ) and based on the fitted convergence rate r = a and convergence confidence band, to obtain the stratified mean trajectory with convergence diagnostic information.

[0020] Preferably, the stratified mean trajectory with convergence diagnostic information is inputted, and based on the convergence rate r and confidence band of each stratum or sample, a weight wi is assigned to the sample or sample block for adjusting the sample influence (e.g. wi ∝ min{1, f(r, CI)}), to obtain the weighted sample set after weighting, including:

[0021] Based on the noise attenuation weighting of LLN convergence characteristics, the weight function wi adopts the mapping form of convergence rate and confidence band wi = φ(ri, CIi), and satisfies the monotonicity: the higher the convergence rate or the narrower the confidence band, the greater the weight, to obtain the weighted sample set after weighting.

[0022] Preferably, the weighted sample set after weighting is inputted, and a distribution decomposition or spectral decomposition method with steady-state moment constraint is used to extract stable components, to decompose the weighted sample into several steady-state sub-distributions or principal components, to obtain the steady-state component set, including:

[0023] The following optional method is used for decomposition: expectation maximization (EM) algorithm with robust constraint, robust principal component analysis (PCA) or mixed distribution fitting with moment condition constraint, to make the high-order moment of the decomposition component consistent with the steady-state moment determined, to obtain the steady-state component set.

[0024] Preferably, the consistency check guided by LLN and the robustness of resampling, based on the minimum sample size and the convergence rate, the effective sample size neff = g(n0, r) is calculated, to adjust the sampling proportion and the number of repetitions in the self-help resampling process, so as to identify and retain stable components in the resampling process, to obtain the robust components and their confidence intervals retained by the consistency check.

[0025] Preferably, the regularization term has the form or the equivalent penalty term based on the block mean shift, to suppress the sensitivity of the parameter to the mean shift of the stratified sample, and based on neff, the prediction confidence interval is scaled and calibrated, to obtain the final robust model parameter, the prediction function and the LLN calibrated confidence interval.

[0026] The present application also provides a data analysis device integrating the law of large numbers, including:

[0027] The hierarchical aggregation determination minimum convergence quantity module: the original observation data are aggregated according to a preset hierarchical standard, and are used for estimating and determining the minimum sample quantity n0 of each hierarchical layer to reach statistical stability (so that σ^2 / n0≤ε or the equivalent convergence criterion is met), so that the stabilized sub-sample set of each hierarchical layer is obtained;

[0028] The cumulative sample mean trajectory calculation and convergence diagnosis module: taking the stabilized sub-sample set of each hierarchical layer as input, the sample mean and sample variance trajectories of each hierarchical sample accumulated with the sample number are calculated, which are used for determining the convergence window and outputting the convergence rate r and convergence confidence band of each hierarchical layer, so that the hierarchical mean trajectory with convergence diagnosis information is obtained;

[0029] The noise attenuation weighting module based on the LLN convergence characteristics: taking the hierarchical mean trajectory with convergence diagnosis information as input, the convergence rate r and confidence band of each hierarchical layer or sample are used to assign weights wi to the sample or sample block for adjusting the sample influence (for example, wi∝min{1,f(r,CI)}), so that the weighted sample set after weighting is obtained;

[0030] The distribution and component decomposition module under the steady-state moment constraint: taking the weighted sample set after weighting as input, the distribution decomposition or spectral decomposition method with the steady-state moment constraint is used to extract stable components, which is used for decomposing the weighted sample into several steady-state sub-distributions or principal components, so that the steady-state component set is obtained;

[0031] The LLN-guided consistency test and robustness verification module: taking the steady-state component set as input, the bootstrap strength is adjusted based on the effective sample size neff estimated by the law of large numbers, and the sample is repeatedly drawn for verification, which is used for retaining the components and parameters that remain stable in most bootstrap, so that the robust components and confidence intervals retained by the consistency test are obtained;

[0032] The LLN-constrained model fitting and uncertainty calibration output module: taking the robust components and confidence intervals retained by the consistency test as input, the final parameter estimation and model fitting are performed, the regularization term based on the LLN convergence characteristics is introduced into the loss function for punishing the sensitivity of the parameters to the sample mean deviation of the block, and the model prediction and the uncertainty calibrated by the LLN are combined for generating the decision-making statistical output, so that the final robust model parameters, the prediction function and the confidence interval calibrated by the LLN are obtained.

[0033] The application also provides an electronic device, which comprises:

[0034] at least one processor; and a memory connected with the at least one processor in communication; wherein:

[0035] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the above.

[0036] The application also provides a computer readable storage medium, which is a computer readable storage medium, and a computer program is stored on the medium, and the computer program is executed by a processor to implement the method of any one of the above.

[0037] Compared with the prior art, the application has the following beneficial effects:

[0038] (1) The noise attenuation weight of the application is based on the convergence characteristics of LLN, so that samples close to steady state have greater influence in estimation, and unstable or sparse samples are naturally weakened, thereby realizing adaptive suppression of heterogeneity and small sample noise and improving the robustness of overall estimation; the introduction of steady state moment constraints in distribution / component decomposition can ensure that the components obtained by decomposition are consistent with the observed convergence characteristics in high-order statistics, reduce component mismatch caused by extreme values or local fluctuations, and improve the physical / semantic interpretability of components.

[0039] (2) The application uses the law of large numbers for hierarchical aggregation and determines the minimum convergence amount, which can guarantee the basic requirement of statistical stability for each hierarchy at the source, thereby reducing the estimation bias and uncertainty propagation caused by insufficient sample size; convergence diagnosis is performed on the cumulative sample mean trajectory, which can clearly determine the convergence behavior and rate of the sample mean of each hierarchy, provide reliable convergence evidence for subsequent processing, and avoid using non-converged samples as stable information. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The fusion law of large numbers data analysis method of the application is shown in the flowchart. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0042] Embodiment one:

[0043] Please refer to Figure 1 The fusion law of large numbers data analysis method, as shown in the figure, includes the following steps:

[0044] S1, stratified aggregation is performed on the original observation data according to a preset stratification standard, used for estimating and determining the minimum sample size n0 of each stratification to reach statistical stability (so that σ^2 / n0≤ε or equivalent convergence criterion is met), and a stabilized subsample set of each stratification is obtained;

[0045] Stratified aggregation and determination of minimum convergence, non-overlapping stratification of original observation according to time, space or attribute dimension, used for calculating sample variance estimation σ^2 in each stratification and determining the minimum sample size n0 to meet:

[0046] σ^2 / n0≤ε

[0047] Wherein, ε is the maximum allowable estimation variance;

[0048] A stabilized subsample set of each stratification is obtained;

[0049] In step S1, scientific stratification and sample size control are used to establish a statistically stable basic data set for subsequent analysis.

[0050] In each stratification, the sample variance estimation value σ 2 is calculated to determine the minimum sample size n0 that meets the statistical stability requirement, and the determination criterion is: σ 2 / n0≤ε, wherein ε represents the maximum allowable estimation variance. The essence of this formula is to control the proportional relationship between sample size and variance to ensure that the estimation error is within an acceptable range. In actual application, the setting of ε value needs to balance the analysis accuracy requirement and the data acquisition cost. When the value of ε is small, a larger sample size n0 is needed to ensure the accuracy, but the data collection cost will increase; when the value of ε is large, the sample size can be reduced, but the estimation accuracy will decrease.

[0051] When the sample size is large enough, the sample statistic will converge to the population parameter. By ensuring that the sample size of each layer reaches n0, the basic statistical stability of the data in each stratification is ensured, providing a reliable data basis for subsequent analysis.

[0052] S2, taking the stabilized subsample set of each stratification as input, calculating the sample mean and sample variance trajectory of each stratification sample accumulated with sample size, used for determining the convergence window and outputting the convergence rate and convergence confidence band of each stratification, obtaining the stratified mean trajectory with convergence diagnostic information;

[0053] The cumulative mean at the kth sample is calculated by sliding accumulation method And the cumulative variance Used to fit the convergence rate model (for example ) and based on the fitted convergence rate r=α and convergence confidence band, the stratified mean trajectory with convergence diagnostic information is obtained;

[0054] In step S2, the statistical convergence characteristics of each layer of data are quantified by monitoring the convergence process of the sample mean and variance. Based on the cumulative trajectory data, a convergence rate model (e.g., r = a, where a is a model parameter) is fitted, and the convergence rate r and the convergence confidence band are determined by statistical methods. The convergence rate r quantifies the speed at which the mean converges to a stable value, with a higher value indicating faster convergence. The convergence confidence band reflects the uncertainty range of the estimated value, with a narrower band indicating more accurate estimation.

[0055] In step S3, the layered mean trajectory with convergence diagnostic information is input, and based on the convergence rate r and the confidence band of each layer or sample, a weight wi is assigned to the sample or sample block to adjust the influence of the sample (e.g., wi ∝ min{1, f(r, CI)}), obtaining a weighted sample set after weighting.

[0056] Based on the noise attenuation weighting of the LLN convergence characteristics, the weight function wi adopts a mapping form wi = φ(ri, CIi) of the convergence rate and the confidence band, and satisfies the monotonicity: the higher the convergence rate or the narrower the confidence band, the greater the weight, obtaining a weighted sample set after weighting.

[0057] In step S3, the influence of high-quality data is enhanced and the interference of low-quality data on the analysis result is reduced through dynamic weight allocation. Based on the convergence diagnostic information obtained in step two, a weight wi is assigned to each sample or sample block, and the weight function adopts a mapping form w i = φ(ri, CIi) of the convergence rate and the confidence band. This weight function has clear monotonicity: the higher the convergence rate ri (the faster the convergence) or the narrower the confidence band CIi (the more accurate the estimation), the greater the weight obtained.

[0058] In step S4, the weighted sample set after weighting is input, and a distribution decomposition or spectral decomposition method with steady-state moment constraints is used to extract stable components, for decomposing the weighted samples into several steady-state sub-distributions or principal components, obtaining a steady-state component set.

[0059] The following optional method is used for decomposition: robust expectation maximization (EM) algorithm with robust constraints, robust principal component analysis (PCA), or hybrid distribution fitting with moment condition constraints, for making the high-order moments of the decomposition components consistent with the steady-state moments determined, obtaining a steady-state component set.

[0060] In step S4, the core components with statistical stability are separated from the weighted sample set, and the noise and abnormal fluctuations are removed. The weighted sample set is processed by using a distribution decomposition or spectral decomposition method with a steady-state moment constraint. The optional methods include a robust expectation maximization (EM) algorithm, a robust principal component analysis (PCA), or a hybrid distribution fitting with a moment condition constraint. These methods have the common feature of imposing a steady-state moment constraint during decomposition to ensure that the extracted components meet the preset statistical stability condition.

[0061] In step S5, the set of steady-state components is input, the bootstrap intensity is adjusted based on the effective sample size neff estimated by the law of large numbers, and the resampling verification is repeated to retain the components and parameters that remain stable in the majority of resampling, thereby obtaining the robust components and their confidence intervals that pass the consistency test;

[0062] The consistency test guided by the law of large numbers and the resampling robustness are based on the minimum sample size and the convergence rate to calculate the effective sample size neff=g(n0,r), which is used to adjust the sampling proportion and the number of repetitions during bootstrap, so as to identify and retain stable components in the resampling process;

[0063] In step S5, the extracted steady-state components are ensured to have universal stability rather than accidental phenomena through statistical verification. Based on the law of large numbers, the effective sample size n e ff=g(n0,r) is calculated, which comprehensively considers the minimum sample size n0 determined in step one and the convergence rate r obtained in step two, and more truly reflects the effective information content in the data. The n e ff is used to adjust the sampling proportion and the number of repetitions during bootstrap. Generally, the larger the effective sample size, the fewer the required number of resampling.

[0064] In step S6, the robust components and their confidence intervals that pass the consistency test are input, and the final parameter estimation and model fitting are performed. The fitting introduces a regularization term based on the convergence characteristics of the law of large numbers into the loss function to punish the sensitivity of the parameters to the mean shift of the block sample, and combines the model prediction with the uncertainty calibrated by the law of large numbers to generate a decision-making statistical output, thereby obtaining the final robust model parameters, the prediction function, and the confidence interval calibrated by the law of large numbers;

[0065] The regularization term has the form or an equivalent penalty term based on the mean shift of the block, which is used to suppress the sensitivity of the parameters to the mean shift of the hierarchical sample, and scale and calibrate the prediction confidence interval based on neff;

[0066] In step S6, a robust model insensitive to data fluctuation is constructed, and a statistically calibrated prediction result is provided. The final parameter estimation and model fitting are performed by taking the robust components and their confidence intervals as inputs, and a regularization term based on the LLN convergence characteristics is introduced into the loss function, such as a penalty term based on block mean shift. This regularization term is different from the traditional L2 regularization, and the purpose is to penalize the sensitivity of the parameters to the stratified sample mean shift, rather than simply limiting the parameter size.

[0067] As can be seen from the above, the noise attenuation weighting based on the LLN convergence characteristics enables the samples close to steady state to have greater influence in estimation, and the unstable or sparse samples are naturally weakened, thereby achieving adaptive suppression of heterogeneity and small sample noise, and improving the robustness of overall estimation; the steady state moment constraint is introduced in the distribution / component decomposition, which can ensure that the components obtained by decomposition are consistent with the observation convergence characteristics in high-order statistics, reduces the component mismatch caused by extreme values or local fluctuations, and improves the physical / semantic interpretability of the components.

[0068] Embodiment two:

[0069] Urban air quality long-term monitoring - PM2.5 concentration analysis:

[0070] Scenario and data acquisition: 50 fixed monitoring points are deployed in a city, and PM2.5 hourly average concentration data are collected, with a sampling period of 1 hour and a data duration of 1 year (365 days). The amount of original observation data = 50 x 24 x 365 = 438000 records. Each record format includes: timestamp, site ID, PM2.5 (μg / m 3 ), temperature, humidity, wind speed, and other auxiliary variables. The data is exported as CSV through the sensor log, and the preprocessing steps are only missing value interpolation (linear interpolation, missing rate <1%) and simple outlier truncation (values exceeding the 99.9th percentile of the site history are recorded as extreme values and retained for subsequent detection).

[0071] Step 1: stratified aggregation and determination of minimum convergence amount, used to calculate the stabilized minimum sample size for the original observation by time-site stratification, and obtain the stabilized subsample set of each stratification.

[0072] Stratification rule: divided by "site x month", a total of 50 x 12 = 600 stratifications.

[0073] Calculate the sample variance estimate σ^2 (use unbiased variance estimate) in each stratification. For example, the sample variance of a certain site in a certain month is measured as σ^2 = 400 (μg 2 , standard deviation ≈ 20 μg / m 3 ).

[0074] Let the convergence precision threshold ε = 1 (target: standard error ≤ 1 μg / m 3, which is equivalent to σ^2 / n0≤1.

[0075] The minimum convergence quantity n0 is obtained from σ^2 / n0≤ε, which is n0=σ^2 / ε. As in the above example, n0=400 / 1=400.

[0076] For each of the 600 layers, the first n0 records are taken as the stabilized subsample of the layer (if the total sample of the layer is less than n0, the layer is marked as "sample sparse" and enters the subsequent weight dilution process).

[0077] Output: 600 layers of their respective stabilized subsample sets and their σ^2, n0 values.

[0078] Step 2: Cumulative sample mean trajectory calculation and convergence diagnosis, used to calculate the cumulative mean trajectory for each layer in the stabilized subsample set of each layer and fit a convergence rate model, obtaining the layer mean trajectory with convergence diagnosis information.

[0079] For each layer's time series, calculate the cumulative mean at the kth record and the cumulative variance

[0080] k=1,…,n.

[0081] Construct the convergence difference sequence (or if the true value μ is estimated by a long-term reference, then ) can be used. Linear fitting: fit the model logdk=logc-αlogk+εk, and obtain the parameters α (convergence rate) and c by least squares estimation.

[0082] Calculate the confidence band at k: use the cumulative variance estimate to obtain the 95% half-width CIk=1.96·Sk / k.

[0083] Numerical value (based on the sampling results of this data set): the average α of the 600 layers is about 0.48 (median 0.50), indicating that the sample mean converges approximately at k^-0.48 as the sample size grows; the typical convergence confidence band is CI≈0.9μg / m 3 .

[0084] Output: Each layer's (convergence diagnosis set).

[0085] Step 3: Noise attenuation weighting based on LLN convergence characteristics, used to assign weights to records based on the convergence rate and confidence band of the layer mean trajectory with convergence diagnosis information, obtaining the weighted sample set after weighting.

[0086] The weight function is defined as:

[0087]

[0088] where ri = ai (hierarchical convergence rate), rref = 0.5, β = 1.2, CIi is the 95% half-width of the current hierarchy, CIref = 0.9 (take the median value of the dataset as reference).

[0089] Note: when the hierarchical convergence rate is high and the confidence band is narrow, wi is close to 1; for sparse / unstable hierarchies, wi will be significantly less than 1, thus weakening their impact in subsequent statistical estimation.

[0090] Calculation: if a hierarchy a = 0.50, CI = 0.8, then w ~ 1; if a hierarchy a = 0.25, CI = 2.4, then w ~ min(1, (0.25 / 0.5) 1.2 (0.9 / 2.4)) ~ 0.11.

[0091] Output: Weighted sample set with weights (each record is attached with the weight of the hierarchy it belongs to, for weighted estimation and decomposition).

[0092] Step 4: Distribution / component decomposition under steady-state moment constraints, used to perform distribution or principal component decomposition with constraints on the weighted sample set after weighting, to obtain a steady-state component set.

[0093] Take the weighted Gaussian mixture model (GMM) as an example on the weighted sample, use the EM algorithm with moment constraints: in the M-step update, force the high-order moments of the component mean and variance to be consistent with the steady-state moments (such as the estimation of mean and variance within the convergence window) in Step 2 (implement moment constraints through the Lagrange multiplier method).

[0094] Process points: initialize K = 3 (background - long-term baseline, periodic term - meteorological correlation, sudden event term - short-term pollution event), in each EM convergence iteration, check if the high-order moments of the component fall within the convergence confidence band given in Step 2, if not, modify the weight or mean of the component for constraint.

[0095] Results: extract 3 steady-state components, component proportions are 72%, 20%, and 8% respectively, and obtain the mean and variance of each component (consistent with the steady-state moments in Step 2).

[0096] Output steady-state component set: steady-state component set (parameters and proportion of each component) and proof of consistency of moment constraints.

[0097] Step 5: LLN-guided consistency check and robustness of re-drawing, used to perform self-help re-drawing based on the effective sample size estimated by the steady-state component set and LLN, and retain stable components, to obtain robust components and confidence intervals retained by consistency check.

[0098] First, compute effective sample size: define neff = n0·(1+γ·r), take γ=2.0. If n0=400 and r=0.48, then neff=400*(1+2*0.48)=400*1.96=784.

[0099] Bootstrap times are adjusted according to neff: baseline bootstrap times B0=500, adjusted B=B0·neff / n0. In the example, B≈500*1.96=980.

[0100] In each bootstrap, perform the decomposition with constraints, record whether each component is stably identified (e.g. component mean, variance, and baseline deviation are all less than a threshold). Set the retention threshold to be “similar components appear in ≥70% of bootstraps”.

[0101] Output: robust components and parameter confidence intervals retained after consistency check (background component mean 22.5 μg / m 3 , 95% CI [21.8, 23.2]).

[0102] Step 6: LLN-constrained model fitting and uncertainty calibration output, for final fitting with consistency-checked robust components and their confidence intervals as prior constraints, and output calibrated prediction and decision statistics, resulting in final robust model parameters, prediction function, and LLN-calibrated confidence intervals.

[0103] Final model uses least squares / maximum likelihood estimation with LLN regularization, loss function:

[0104]

[0105] where denotes the block mean shift caused by model parameters (computed per stratum in Step 2), regularization coefficient λ=0.1.

[0106] Prediction uncertainty is interval-scaled by neff: original 95% interval is I, calibrated interval is I·n0 / neff (or inversely scaled by neff / n0, principle is: larger effective sample size leads to narrower confidence interval; here Icalibrated=I·n0 / neff).

[0107] Output: final robust model parameters, prediction function, and LLN-calibrated 95% confidence intervals, for decision use.

[0108] Test and comparative example explanation for Example 2 (data, computation process, and results):

[0109] Comparison: The conventional method (control) is the traditional weighted or ordinary GMM decomposition without LLN guidance, using a fixed bootstrap B0=500 without n_eff adjustment and without LLN regularization.

[0110] Test sample: 20% (87600) of the full year data is randomly selected as the validation set, and the rest is used for training. Comparison index: parameter estimation mean square error (MSE), average width of 95% prediction interval, interval coverage rate (target 95%), steady-state component retention rate (bootstrap retention rate threshold ≥ 70%), false alarm rate (misjudgment rate for sudden pollution events) and relative computational overhead (1.0 for the control scheme).

[0111] Results:

[0112] Parameter estimation MSE (the smaller the better): 16.0 for the control scheme; 12.0 (25% reduction) for the current scheme.

[0113] Average width of 95% prediction interval (μg / m 3 ): 6.0 for the control; 4.8 (20% narrower) for the current scheme.

[0114] Interval coverage rate (target 95%): 90.0% for the control; 94.6% for the current scheme.

[0115] Steady-state component retention rate (representing the proportion of background components detected in bootstrap): 68% for the control; 86% for the current scheme.

[0116] False alarm rate for sudden pollution (according to post-hoc manual labeling): 14% for the control; 9% for the current scheme.

[0117] Relative computational overhead: 1.0 for the control; 1.35 (an increase of about 35% in computational load from more bootstrap resampling and constrained EM steps) for the current scheme.

[0118] The simulation and rationalized estimation from small-scale real operation of the parameters (ε=1, rref=0.5, β=1.2, γ=2.0, B0=500, λ=0.1) of this embodiment reflect the actual improvement of the method in robustness and uncertainty control.

[0119] Example Three:

[0120] Abnormal detection and robust feature modeling of Internet payment transactions - transaction amount analysis:

[0121] Scenario and data acquisition: 6 months of transaction logs from a payment platform, totaling 2,000,000 transaction records. Each record contains: timestamp, merchant category, transaction amount, device type, geographic region, etc. The data is used for research after being exported from the platform log and desensitized. Data preprocessing: log-transform the amount to reduce skew; impute missing auxiliary fields by category median (missing rate < 0.5%).

[0122] Step 1: Hierarchical aggregation and determination of minimum convergence quantity, used to stratify the original transactions by "merchant category x week" and determine the minimum convergence quantity n0 for each layer.

[0123] Stratification rule: The platform has a total of 200 merchant categories, 6 months are about 26 weeks, and the number of stratification is 200 x 26 = 5200 stratifications.

[0124] In a certain stratification (for example, "online ticketing - 12th week"), the sample variance estimate of the transaction amount log is σ^2 = 2500.

[0125] Let ε = 4 (target: σ^2 / n0 ≤ 4), then n0 = 2500 / 4 = 625.

[0126] Calculate n0 for all 5,200 stratifications, and identify the sparse stratifications with insufficient samples. Output: the stabilized subsample and n0 for each stratification.

[0127] Step 2: Cumulative sample mean trajectory calculation and convergence diagnosis, used to fit the convergence model and obtain the stratification convergence rate r.

[0128] Calculate the cumulative mean of each stratification variance Fit logdk = logc - αlogk to obtain α.

[0129] Results: The average α of the 5200 stratifications is about 0.55 (faster convergence than air quality data due to large data volume and relatively stable heterogeneity within categories); the median of the typical CI is about 2.2 (in log amount units).

[0130] Output: Set of stratification convergence diagnoses.

[0131] Step 3: Noise attenuation weighting based on LLN convergence characteristics, used to assign weights to each record according to r, CI in the stratification convergence diagnosis set, and obtain the weighted sample set after weighting.

[0132] Still use the weight function: wi = min(1, (ri / rref β · (CIref / CIi)), where rref = 0.5, β = 1.2, CIref = 2.2.

[0133] If certain layer α = 0.6, CI = 1.8, then w ≈ 1; if α = 0.3, CI = 5.0, then w ≈ (0.3 / 0.5) 1.2 · (2.2 / 5.0) ≈ 0.16.

[0134] Output: Weighted sample set for subsequent decomposition.

[0135] Step 4: Distribution / Component Decomposition under Steady State Moment Constraints, for extracting stable features (e.g. common real transaction distribution, seasonal peaks, anomalous patterns).

[0136] Fit weighted log amount data with mixture distribution with moment constraints (e.g. mixture log-normal distribution with robustness constraints), initialize K = 4 (regular consumption, low-frequency high-amount, large-amount merchant normal, anomalous fraud indication).

[0137] Constrain each component's first and second moments to be consistent with the steady state moments estimated in Step 2 in EM iterations, to avoid being pulled off by a small number of extreme transactions.

[0138] Output: 4 stable components (give mean, variance, and proportion), and ensure consistency with Step 2 convergence diagnostics.

[0139] Step 5: LLN-guided Consistency Check and Resampling Robustness, for verifying component stability through bootstrap resampling and adjusting resampling times.

[0140] Take γ = 1.5, neff = n0(1 + γr). If certain layer n0 = 625, r = 0.55, then neff = 625 * (1 + 1.5 * 0.55) = 625 * 1.825 = 1140.

[0141] Adjust resampling layer convergence diagnostic set B = 500 * neff / n0 ≈ 500 * 1.825 = 913 times.

[0142] Keep robust components that appear in ≥ 70% of resamplings.

[0143] Output: Robust components and parameter confidence intervals (e.g. regular consumption component log mean 3.2, 95% CI [3.15, 3.25]).

[0144] Step 6: LLN-constrained Model Fitting and Uncertainty Calibration Output, for final model training and prediction and output calibrated confidence intervals for anomaly alert threshold setting.

[0145] Final loss function adopts the same regularization form as in Example One, with λ = 0.08.

[0146] Predictive intervals are scaled and calibrated according to neff.

[0147] Output: final robust parameters, prediction function (for transaction probability scoring) with calibrated 95% intervals.

[0148] Example 3 and Comparative Example:

[0149] Control: Conventional mixture model without LLN guidance and fixed stratification convergence diagnostic set B=500 bootstrap. Test set randomly drawn from 6 months data 20% (400,000). Metrics as above (MSE, interval width, coverage, stable component retention, false alarm rate, computational overhead).

[0150] Results:

[0151] Parameter estimation MSE: Control 1.80; Current scheme 1.35 (about 25% reduction).

[0152] Average width of prediction 95% intervals (log amount units): Control 0.48; Current scheme 0.39 (about 18.75% narrowing).

[0153] Interval coverage: Control 91.2%; Current scheme 95.1%.

[0154] Stable component retention: Control 64%; Current scheme 88%.

[0155] False alarm rate for anomalies (fraud): Control 6.5%; Current scheme 4.1%.

[0156] Relative computational overhead: Control 1.0; Current scheme 1.30 (about 30% increase).

[0157] Summary comparison table, Example 1, Example 2 and Control scheme as shown in Table 1 below.

[0158] "Control scheme" in the table represents the conventional method without the LLN guidance steps (LLN constraints and adaptive resampling, LLN regularization term, etc. in steps 2-6) in the present application in the corresponding scenario.

[0159] Table 1

[0160]

[0161]

[0162] The method in the present application applies the law of large numbers (LLN) throughout each layer of the data stream: for determining the minimum convergence quantity, for convergence diagnostics, mapping to sample / stratum weights, as a constraint on the matrix of decomposition, guiding the intensity of bootstrap resampling and adjusting the effective sample size, and as a basis for regularization of the final fit, thereby achieving end-to-end consistency control in data aggregation, component extraction, parameter estimation and uncertainty calibration.

[0163] From the above, the law of large numbers is used to stratify aggregation and determine the minimum convergence amount, which can guarantee the basic requirement of statistical stability for each stratification at the source, thereby reducing the estimation bias and uncertainty propagation caused by insufficient sample size; the convergence diagnosis of the cumulative sample mean trajectory can clearly show the convergence behavior and rate of the sample mean of each stratification, providing reliable convergence evidence for subsequent processing and avoiding the use of unstable information as stable information.

[0164] The effective sample size adjustment derived from the LLN, bootstrap resampling and consistency test can significantly enhance the stability discrimination ability of the resampling results, thereby screening out the components and parameters that remain robust in most cases and reducing the risk of misjudging pseudo-stable or accidental structures as stable signals. The introduction of regularization and uncertainty calibration based on the LLN in the final fitting can constrain the sensitivity of the model to block mean drift and make the prediction interval truly reflect the sample convergence degree, thereby simultaneously improving the robustness of the model and the reliability of the uncertainty representation.

[0165] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A data analysis method incorporating the law of large numbers, characterized in that, The method comprises the following steps: performing stratified aggregation on original observation data according to a preset stratification standard, to estimate and determine the minimum sample size of each stratum to reach statistical stability, and obtain a stabilized subsample set of each stratum; inputting the stabilized subsample set of each stratum, to calculate the sample mean and sample variance trajectory of each stratum sample accumulated with the sample size, to determine the convergence window and output the convergence rate and convergence confidence band of each stratum, and obtain a stratified mean trajectory with convergence diagnostic information; inputting the stratified mean trajectory with convergence diagnostic information, to assign weights to samples or sample blocks according to the convergence rate and confidence band of each stratum or sample, to adjust the sample influence, and obtain a weighted sample set after weighting; inputting the weighted sample set after weighting, to extract stable components by using a distribution decomposition or spectral decomposition method with steady-state moment constraints, to decompose the weighted sample into several steady-state sub-distributions or principal components, and obtain a steady-state component set; inputting the steady-state component set, to adjust the bootstrap intensity based on the effective sample size estimated by the law of large numbers and to verify repeated sampling, to retain components and parameters that remain stable in most bootstraps, and obtain robust components and confidence intervals retained by consistency test; inputting the robust components and confidence intervals retained by consistency test, to perform final parameter estimation and model fitting, the fitting introduces a regularization term in the loss function based on the LLN convergence characteristics to punish the sensitivity of parameters to the deviation of sample mean from the block, and combines the model prediction with the uncertainty of LLN calibration to generate decision-making statistical output, to obtain the final robust model parameters, prediction function and confidence interval calibrated by LLN.

2. The data analysis method of claim 1, wherein, The stratified aggregation and determination of the minimum convergence amount, non-overlapping stratification of original observations according to time, space or attribute dimensions, is used to calculate the sample variance estimate σ^2 and determine the minimum sample size n0 in each stratum to satisfy: σ^2 / n0≤ε where ε is the maximum allowed estimation variance; to obtain a stabilized subsample set of each stratum.

3. The data analysis method of claim 1, wherein, Inputting the stabilized subsample set of each stratum, to calculate the sample mean and sample variance trajectory of each stratum sample accumulated with the sample size, to determine the convergence window and output the convergence rate r and convergence confidence band of each stratum, to obtain a stratified mean trajectory with convergence diagnostic information, including: The cumulative mean at the kth sample is calculated using the sliding cumulative method The cumulative variance The convergence rate model is fitted and based on the fitted convergence rate r = a and the convergence confidence band, the stratified mean trajectory with convergence diagnostic information is obtained.

4. The data analysis method that fuses the law of large numbers according to claim 1, wherein, Inputting the stratified mean trajectory with convergence diagnostic information, to assign weights wi to samples or sample blocks according to the convergence rate r and confidence band of each stratum or sample, to adjust the sample influence, and obtain a weighted sample set after weighting, including: Noise attenuation weighting based on the LLN convergence characteristics, the weight function wi adopts a mapping form wi=φ(ri,CIi) of the convergence rate and confidence band, and satisfies monotonicity: the higher the convergence rate or the narrower the confidence band, the greater the weight, to obtain a weighted sample set after weighting.

5. The data analysis method that fuses the law of large numbers according to claim 1, wherein, Inputting the weighted sample set after weighting, to extract stable components by using a distribution decomposition or spectral decomposition method with steady-state moment constraints, to decompose the weighted sample into several steady-state sub-distributions or principal components, and obtain a steady-state component set, including: The following optional method is used for decomposition: expectation maximization EM algorithm with robust constraints, robust principal component analysis PCA, or mixture distribution fitting with matrix condition constraints, to obtain a steady-state component set.

6. The data analysis method that fuses the law of large numbers according to claim 1, wherein, LLN-guided consistency test and resampling robustness, based on minimum sample size and convergence rate calculation effective sample size neff=g(n0,r), is used to adjust the sampling proportion and the number of repetitions when self-help resampling, so as to identify and retain stable components in the resampling process, and obtain the robust components and their confidence intervals retained by the consistency test.

7. The data analysis method that fuses the law of large numbers according to claim 1, wherein, The regularization term is of the form or an equivalent piecewise mean shift based penalty term, to suppress the sensitivity of the parameters to the mean shift of stratified samples, and to scale the calibration of the prediction confidence intervals based on neff, resulting in final robust model parameters, prediction function, and LLN calibrated confidence intervals.

8. A data analysis apparatus incorporating the law of large numbers, characterized by, It comprises: Hierarchical aggregation determines the minimum convergence amount module: the original observation data is aggregated according to the preset hierarchical standard, which is used to estimate and determine the minimum sample size of each hierarchical statistical stability, and obtain the stable sub-sample set of each hierarchical layer; Cumulative sample mean trajectory calculation and convergence diagnosis module: taking the stable sub-sample set of each hierarchical layer as input, the sample mean and sample variance trajectory of each hierarchical sample accumulated with the sample size is calculated, which is used to determine the convergence window and output the convergence rate r and convergence confidence band of each hierarchical layer, and obtain the hierarchical mean trajectory with convergence diagnosis information; Noise attenuation weighting module based on LLN convergence characteristics: taking the hierarchical mean trajectory with convergence diagnosis information as input, according to the convergence rate r and confidence band of each hierarchical layer or sample, the sample or sample block is assigned a weight wi for adjusting the sample influence, and the weighted sample set after weighting is obtained; Distribution and component decomposition module under steady-state moment constraint: taking the weighted sample set after weighting as input, using distribution decomposition or spectral decomposition method with steady-state moment constraint for extracting stable components, for decomposing the weighted sample into several steady-state sub-distributions or principal components, to obtain a steady-state component set; LLN-guided consistency test and resampling robustness module: taking the steady-state component set as input, adjusting the self-help resampling intensity based on the effective sample size neff estimated by the law of large numbers and repeating the sampling verification, for retaining the components and parameters that remain stable in most resamplings, to obtain the robust components and their confidence intervals retained by the consistency test; LLN-constrained model fitting and uncertainty calibration output module: taking the robust components and their confidence intervals retained by the consistency test as input, performing final parameter estimation and model fitting, the fitting introduces a regularization term based on the LLN convergence characteristics in the loss function to punish the sensitivity of the parameters to the block sample mean deviation, and combines the model prediction with the LLN calibrated uncertainty to generate decision-making statistical output, to obtain the final robust model parameters, prediction function and LLN calibrated confidence interval.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein: The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7. The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein: The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The medium is a computer readable storage medium, and the medium stores a computer program. The computer program is executed by a processor to implement the method in any one of claims 1-7.

Images