A high-performance file hash calculation method based on double-engine adaptive switching

Abstract

The application discloses a high-performance file hash calculation method based on double-engine adaptive switching, relates to the technical field of data security, and aims to solve the problems that the existing scheme is mostly heuristic scheduling based on static threshold values or empirical rules, lacks residual time estimation and interval control based on online statistical learning, and lacks real-time detection of residual drift and adaptive scheduling based on evidence; the method comprises the following steps: constructing a high-dimensional vector by mapping observation characteristics, adopting joint modeling of recursive least squares with ridge regularization and quantile regression, providing point estimation and upper and lower bounds of single fragmentation and residual time consumption; fusing the extreme values of residual variance and quantile interval to obtain a robust interval, and using Page-Hinkley / CUSUM for mutation detection to trigger adaptive adjustment of fragmentation size, concurrency and engine type; and before engine switching, serializing internal working vectors, processed bytes and intermediate summaries and performing mirror checking to ensure consistency of cross-engine results.

Description

Technical Field

[0001] This invention relates to the field of data security technology, and more specifically, to a high-performance file hash calculation method based on dual-engine adaptive switching. Background Technology

[0002] File hashing is widely used in browsers, mobile devices, and servers for integrity verification, deduplication, and content addressing. Common implementations often employ fixed fragment sizes and concurrency levels, using a single execution engine, such as JS / WebCrypto or WASM, to perform incremental hashing or tree hash reduction serially or in simple parallel. However, front-end and edge environments exhibit significant non-stationarity: I / O latency, garbage collection frequency, main thread blocking, CPU / temperature constraints, and queue congestion all fluctuate over time; the relative advantages of different engines under the same load change dynamically. Existing solutions often rely on static thresholds or empirical rules for heuristic scheduling, lacking residual time estimation and interval control based on online statistical learning, real-time detection of residual drift, and evidence-based adaptive scheduling. Furthermore, when switching between engines, they often cannot serialize and seamlessly migrate intermediate hash states, leading to recalculation or consistency risks.

[0003] The existing technology has the following shortcomings:

[0004] In the process of sharded hashing, facing non-stable loads and engine performance drift, how to build a closed-loop method and system that can learn the time consumption online, provide a controllable ETA range, trigger mutation detection and adaptively adjust sharding, concurrency and engine selection accordingly, and ensure the serializable migration of hash state and consistency of results during switching.

[0005] To address the above problems, this invention proposes a solution. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a high-performance file hash calculation method based on dual-engine adaptive switching to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A high-performance file hash calculation method based on dual-engine adaptive switching includes the following steps;

[0009] The observation samples generated by the sharded execution are obtained and the observation features are mapped to construct a high-dimensional feature vector φ. The observation samples contain fields such as shard size, read latency, garbage collection information, concurrency, engine type, queue length, main thread blocking and processor usage.

[0010] Based on the high-dimensional feature vector φ, the online master model is updated using recursive least squares with ridge regularization, while maintaining the quantile regression model to obtain upper and lower quantile predictions.

[0011] Based on the online master model and quantile regression model, the remaining time point estimate and interval are calculated, Gaussian approximation and quantile interval fusion are performed, and scheduling parameters such as partition size, concurrency and engine selection are generated.

[0012] The prediction residuals are subjected to mutation detection. When the triggering conditions are met, hash state serialization, inter-engine migration and mirror verification are performed, and the scheduling parameters are written back to the feature construction and scheduling execution of subsequent shards.

[0013] In a preferred embodiment, the mapping process for the observed features includes: numerically normalizing the fragment byte size and read latency, classifying the garbage collection identifier and engine type, combining the processed features with the processor usage and concurrency, and constructing the high-dimensional feature vector φ that includes engine and concurrency interaction terms.

[0014] In a preferred embodiment, obtaining the observation sample includes: collecting the read delay, garbage collection flag and time since the last garbage collection, current queue length and main thread blocking index when the fragmentation is completed, and then filling in the missing or abnormal observations with robust statistics using a sliding window before writing them into the observation sample.

[0015] In a preferred embodiment, recursive least squares includes: initializing the covariance matrix to a proportional form of the identity matrix according to the regularization parameter, calculating the gain vector related to the high-dimensional eigenvector φ, updating the weight vector and the covariance matrix to rank-one, and performing symmetryization and spectral clipping on the covariance matrix when the values ​​are not fit.

[0016] In a preferred embodiment, maintaining the quantile regression model includes: performing subgradient updates based on a check function on the upper and lower quantiles respectively, uniformly using an indicator function to represent the residual direction, and applying a consistency constraint that the upper bound is not lower than the lower bound to the same high-dimensional feature vector φ.

[0017] In a preferred embodiment, calculating the remaining time point estimate and interval includes: calculating the number of remaining fragments based on the remaining bytes and the current fragment size; constructing the high-dimensional feature vector for each future fragment; and recording the algorithm and pattern indication during tree reduction and incorporating them into the high-dimensional feature vector for subsequent estimation.

[0018] In a preferred embodiment, the fusion of Gaussian approximation and quantile intervals includes: forming lower and upper bound candidates by the Gaussian interval obtained by approximation of residual variance and covariance and the upper and lower quantile regression intervals, respectively, and completing the fusion by taking the smaller of the lower bound and the larger of the upper bound, and introducing a correlation inflation coefficient when accumulating variance.

[0019] In a preferred embodiment, mutation detection includes: calculating the Page-Hinkley or cumulative sum statistic for the predicted residual sequence, making a determination using a bias and threshold configuration derived from the robust residual scale, and updating the residual mean using a sliding window before making the determination.

[0020] In a preferred embodiment, hash state serialization and inter-engine migration includes: serializing the current engine's internal working vector, processed bytes, and intermediate digest, and deserializing them in the target engine; performing mirror verification; rolling back to the previous consistent state when inconsistent; and a hysteresis strategy that restricts reverse switching.

[0021] In a preferred embodiment, the scheduling parameter write-back includes: maintaining the regularization parameter and alarm threshold increase and the learning rate decrease during the cooling period after the mutation is triggered, restoring according to the preset annealing curve after the cooling period ends, and updating the fragment size and concurrency with a binary step size and synchronizing to the subsequent construction of the high-dimensional feature vector φ.

[0022] The technical effects and advantages of the high-performance file hash calculation method based on dual-engine adaptive switching of the present invention are as follows:

[0023] This invention provides point estimates and upper and lower bounds for single-shard and remaining time consumption by constructing high-dimensional vectors from observed feature maps and employing joint modeling with ridge regularized recursive least squares and quantile regression. Robust intervals are obtained by fusing residual variance and extreme values ​​of quantile intervals, and mutation detection is performed using Page-Hinkley / CUSUM to trigger adaptive adjustments to shard size, concurrency, and engine type. Before engine switching, internal working vectors, processed bytes, and intermediate summaries are serialized and mirrored to ensure consistency across engines. During the cooling-off period, annealing write-back is performed on regularization, thresholding, and learning rate to suppress parameter jitter. This enables: stable control and controllable coverage of ETA in dynamic environments; rapid scheduling updates and deterministic hash calculations when performance drift occurs; and reduced unnecessary recalculations and switching round trips under multi-platform conditions, improving overall throughput and resource adaptability. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the process of a high-performance file hash calculation method based on dual-engine adaptive switching according to the present invention;

[0025] Figure 2 This is a timing diagram of a high-performance file hash calculation method based on dual-engine adaptive switching according to the present invention.

[0026] Figure 3 This is a data interaction diagram of a high-performance file hash calculation method based on dual-engine adaptive switching according to the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] This invention provides a high-performance file hash calculation method based on dual-engine adaptive switching. This method constructs an end-to-end closed-loop execution chain around the sharding execution process. Through joint modeling of online ridge regression and quantile regression, supplemented by residual variance estimation, mutation detection, and policy adaptation, it achieves online controllable prediction of the time consumed by a single shard and the remaining total time, as well as rapid and stable convergence when performance drifts.

[0029] Please see Figure 1 , Figure 1 This is a schematic diagram of the online self-calibration process provided in an embodiment of the present invention. The process is indivisible and iterates once as each segment is completed.

[0030] like Figure 1 , Figure 2 As shown, the specific steps are as follows:

[0031] S101, Target and Interface Anchoring; this step aims to clarify the input, output, and statistical targets and establish a unified anchor point. The system generates observation samples for each completed slice. The samples include:

[0032] Observational features: These describe the environment and load conditions during sharding execution. Specifically, they include shard byte size and its logarithmic representation, read latency, garbage collection flag and time since last garbage collection, current concurrency, engine type indicator, worker queue length, main thread blocking metrics, and CPU busy / idle ratio. On mobile devices, observational features can be expanded to include additional environmental information such as temperature trends or battery rate of change.

[0033] Actual processing time: The actual time taken for this segment to complete.

[0034] Meta-information includes file identifier, engine type, worker number, timestamp, etc., used for tracking and diagnosis.

[0035] The method output includes: a point estimate and upper and lower bounds of the remaining time, i.e., the ETA and its confidence interval, a mutation flag, and suggested scheduling parameters, such as fragment size, concurrency, engine selection, etc., specifically as follows: Figure 3 As shown. The method objectives include: to achieve controllable prediction of the remaining total time while meeting the preset confidence coverage target and tail latency limit, and to trigger adaptive scheduling to adjust the strategy when a performance mutation is detected.

[0036] Task Definition: The target output is the hash value H(F) of the input file, such as SHA256, SHA512, BLAKE3, etc., calculated on the chunk sequence {chunk_j} according to a predetermined reduction, such as sequential concatenation or tree reduction. The system guarantees that the final hash is consistent with the serial reference implementation bytes under engine switching and concurrency adjustment.

[0037] S102, Observation Acquisition and Feature Mapping; This step involves acquiring the environment and load proxy for each segment and performing feature mapping to construct a high-dimensional feature vector φ. An example mapping is shown below:

[0038] Size effect: characterized by the logarithm and square of the piece size, for example and .

[0039] Engine type encoding: Indicates whether the variable encoding is WebAssembly, denoted as WASM or JavaScript, denoted as JS, for example. and .

[0040] Concurrency and Engine Interaction: Multiply the concurrency level K by the engine instruction to form the interaction characteristics, for example... and This is to demonstrate the synergistic effect between concurrency and the engine.

[0041] Environmental characteristics: read latency Waste recycling sign Time since the last garbage collection Concurrency Queue length Blocking indicators CPU busy / idle ratio . Measured in milliseconds, taken from the start of scheduling to the time when the first byte is readable, and outliers are truncated according to IQR. Measured in seconds, derived from runtime GC timestamp differences. q represents the number of slices to be executed, taken as the instantaneous value of the sample or the middle value within a window. The block is measured using event loop lag in milliseconds (ms) or its P95 value, obtained from main thread inter-frame latency monitoring. These features are directly used as model input.

[0042] Hash Algorithm and Pattern Characteristics: An `algo` field is added, such as SHA256 / BLAKE3, and the `mode` field can be either `sequential` or `tree`. When `mode=tree`, the tree fan-out `f` is recorded, such as 4 / 8 / 16, along with the upper limit of the level depth, used to reflect the parallel reduction stage in the future sharding construction and ETA estimation of S106. The tree fan-out `f` ∈ {4, 8, 16}, with a default of 8; maximum level depth. equal The independent variable is ⌈|F| / b⌉, rounded up, and then rounded up as a whole, which can be denoted as: In ETA estimation, the prediction time of the tree reduction stage is given by the sum of the parallel time within the sharding layer and the inter-layer merging time. The characteristics of the merged node are constructed from the representative values ​​of the layer number, fan-out f, and queue length q. The calculation expression can be: ;in, Predict the time taken for the j-th fragment. The prediction time for the i-th merged node in the ℓ-th layer can be represented by features such as the layer number ℓ, fan-out f, and queue representative value q∗.

[0043] In this application, the feature dimension is uniformly denoted as D, and the concurrency is... CPU busy / idle ratio ρ∈[0,1], slice size b and remaining bytes Measured in bytes. This embodiment uses a fixed-length sliding window W to maintain robust statistics, such as the median, quantile interval, and MAD. Unless otherwise specified, the fixed-length sliding window W is 64, 128, or 256. To avoid ambiguity, this application uses the following definition of robust statistics:

[0044] (1) IQR (Interquartile Range): ,in , These are the 25th and 75th percentiles of the sample, respectively. Outlier truncation based on features such as Tuckey rule is used: Values ​​outside the boundary are truncated.

[0045] (2) MAD (Median Absolute Deviation): When a robust metric comparable to the standard deviation is required, use... As an unbiased estimate of σ.

[0046] S103, Online Master Model Update; This step uses ridge-regularized recursive least squares (RLS) for online updates. Let w be the weight vector, and P be the approximate inverse representation of the covariance matrix.

[0047] During initialization, w is set as the zero vector, and P is the product of the reciprocal of the regularization parameter λ and the identity matrix I, i.e., P = (1 / λ)I.

[0048] The new sample is updated using the high-dimensional feature vector φ and the actual time y as follows:

[0049] 1. Calculate the predicted value and residual: Obtain the predicted value using the current weight w. Then, the residual is obtained by subtracting the predicted value ŷ from the actual value y. .

[0050] 2. Calculate the gain vector: Multiply the current P by φ and correct by a normalization factor to obtain the gain vector. .

[0051] 3. Update weights: Multiply the residual e by the gain vector g and add it to the weight w: w←w+ge.

[0052] 4. Update covariance: Perform rank-one correction on P in Sherman–Morrison form: P←Pgφ^TP.

[0053] To avoid confusion with the allowable deviation parameter k in CUSUM, this application uniformly denotes the RLS gain vector as g. To improve numerical stability, a small diagonal jitter ε can be added to P initialization: ,in For cases of ill-conditioned collinearity, symmetry and spectral clipping can be applied to P. The computational cost of each update step is proportional to the square of the feature dimension, making it suitable for real-time execution on browsers and mobile devices. Ridge regularization is used to suppress overfitting and improve the model's sensitivity to new observations.

[0054] S104, Residual Variance and Uncertainty Estimation; This step uses an exponentially weighted method to estimate the residual variance online. Each new residual e t The variance at the previous time point The weight α and the current squared residual We then weight the variances by (1-α) to obtain the latest variance estimate: The recommended EWMA decay factor α is 0.90–0.99; the larger the α, the less sensitive it is to recent anomalies and the smoother the response. After a sudden change, α can be temporarily lowered to accelerate the variance response, and then returned to its normal value after the cooling period.

[0055] Online prediction variance is approximately derived from residual variance. Multiplying by the quadratic forms of φ and P yields, i.e. This estimate reflects the impact of model parameter uncertainty on the output and provides a basis for subsequent interval estimation and coverage control.

[0056] The system maintains residual variance based on EWMA and standard deviation estimates based on quantile bandwidth in parallel. When constructing intervals, the more conservative one of the two is taken as the uncertainty input for the same high-dimensional feature vector φ to ensure the robustness of tail coverage.

[0057] S105, Online Quantile Regression; this step maintains the linear quantile regression models for the upper and lower quantiles in parallel. Specifically:

[0058] Select two quantiles and For example, 0.2 and 0.8 each maintain their own independent weight vectors. and .

[0059] For each sample, i.e., the high-dimensional feature vector φ and the actual time y, the weights of the low-quantile and high-quantile are updated using a subgradient rule. The update process is based on a check function. Minimization. This application uniformly adopts... This indicates an indicator function, to distinguish it from the identity matrix I.

[0060] For example, for low quantile models: calculate the residuals Then according to Symbol update .

[0061] After the update is completed, for the same feature, ensure that the upper bound predicted value is not lower than the lower bound predicted value, for example... This step is used to maintain semantic consistency across intervals. It captures the asymmetry and long-tail characteristics of the residual distribution, providing a supplementary path for interval estimation.

[0062] To avoid ambiguity in the notation, this application uses a uniform notation. , Predict the upper and lower quantiles for the current partition. , Predict the upper and lower quantiles of future segment j.

[0063] Learning Rate and Annealing: Quantile Regression uses a fixed or time-decreasing learning rate. It can be used according to sample number t. Alternatively, a stepped decay method, such as a 0.5-fold decay every 64–128 samples, can be used to strike a balance between convergence speed and steady-state fluctuations.

[0064] S106, Future Piece Feature Construction, ETA Calculation, and Interval Fusion; this step performs point estimation and interval estimation for the remaining tasks. Specifically, it includes:

[0065] 1. Calculate the number of remaining fragments: based on the number of bytes remaining in the current file. The remaining number of fragments is obtained by comparing the current fragment size b. .

[0066] 2. Constructing future segment features: Construct a feature vector for each future segment j. Size features are fixed as the logarithm and square of the current fragment size. Concurrency and engine type remain unchanged. Environmental features are represented by the median or quantile of the most recent sliding window. Garbage collection scenarios are simulated using historical occurrence rates or high quantile estimates.

[0067] 3. Point estimation: Estimating each future slice using the online master model. Obtain the predicted time for a single fragment And summed up to form a point estimate of the remaining time. In this application, the point estimate of the remaining time is denoted as... .

[0068] 4. Gaussian approximation interval: based on the residual variance and... The quadratic form of P yields the prediction variance for each future segment j. And summed up to the total variance Without explicitly modeling the correlations between future fragments, The implicit assumption is approximate independence; for conservatism, a correlation inflation factor c≥1 can be introduced, and the following approach is adopted. For example, c = 1.1–1.3. Then, based on the normal approximation, multiplying by the constant coefficient z expands the upper and lower bounds. and The coverage target can be set according to needs: 90% / 95% / 97.5% correspond to z=1.645 / 1.96 / 2.24 respectively.

[0069] 5. Quantile Regression Intervals: For each future segment, use both low and high quantile models. Obtain single-segment upper and lower quantile predictions and And sum them up to get the total upper and lower quantile values. and .

[0070] 6. Interval merging: Lower bound Taking the lower bound of the Gaussian approximation and the total lower quantile The smaller of the three. Upper Realm Take the upper bound of the Gaussian approximation The larger of the total high quantile value ETA_quantile_up and the relative interval width. Exceeding a preset threshold indicates that the estimation is unstable, providing a signal for mutation detection and policy adjustment. The aforementioned lower / upper bound min / max fusion is based on experimental measurements in skewed and long-tailed scenarios: under equal coverage targets, compared to Gaussian fusion alone, quantile fusion alone, and linear weighted fusion, min / max fusion can reduce false alarm rate and oscillation rate while maintaining coverage; the difference can be quantified in the experimental comparison section through coverage, relative interval width, and policy oscillation rate.

[0071] S107, Residual mutation detection;

[0072] This step involves online monitoring of the predicted residuals to identify sudden changes in system performance. This can be achieved using the cumulative sum (CUSUM) method or the difference-based Page-Hinkley detection statistical method.

[0073] With residuals Subtract the most recent mean The increment is obtained by subtracting the tolerance offset δ, and the positive and negative statistics and thresholds are maintained respectively.

[0074] For example, for the Page-Hinkley algorithm:

[0075] ;

[0076] ;

[0077] ;

[0078] When the statistic Exceeding the preset threshold A mutation alert is triggered immediately. The detection results, combined with indicators such as garbage collection rate, read latency, and main thread blocking, are used to pinpoint the source of the bottleneck and provide a basis for diagnosis.

[0079] Empirically, the tolerance for offset and the threshold can be adaptively set according to the robust scale of the residuals: The threshold for Page-Hinkley can be set to... CUSUM's drift sensitivity and threshold are acceptable. , The above-mentioned caliber is adjusted according to the window length W and the business risk appetite.

[0080] S108, Policy Adaptation and Writeback;

[0081] This step performs closed-loop adjustments after the mutation is triggered. First, a soft reset is performed on the statistical side, employing a three-stage strategy: First, robust statistics such as the moving median, quantile interval, and GC occurrence rate are retained a priori, and λ and the alarm threshold are temporarily increased; second, a cooling-off period is initiated. Typically, there are 5-10 segments, during which engine switching and parameter oscillation are restricted; thirdly, after the cooldown period, λ, threshold, and learning rate are gradually reduced to normal according to the set annealing curve to avoid oscillation. During the cooldown period, λ, alarm threshold, and learning rate are fixed at the set upper limit or gradually reduced, for example, λ is increased by 1 order and the learning rate is reduced to 1 / 2 of the normal value. After the cooldown period, according to the annealing curve... Within a single step, the system should return to normal, avoiding repeated fluctuations. Then, parameter adjustment mapping should be performed, paying attention to adjusting parameters according to the mutation characteristics, specifically:

[0082] If the garbage collection frequency increases significantly, reduce the fragment size b and the concurrency K to reduce memory pressure and execution tail.

[0083] If read latency or main thread blocking increases sharply, the execution engine will be temporarily switched based on the engine's sensitivity to blocking, such as switching from WASM to JS, or reducing concurrency to alleviate the bottleneck.

[0084] If the device temperature or CPU load rises significantly, it will enter power-saving mode and reduce the fragment size to reduce resource consumption.

[0085] Finally, engine switching and state safety are implemented. When PH / CUSUM is triggered and I / O / main thread blocking increases, the BLAKE3 computation in mode=tree is preferentially scheduled to the WASM engine. If temperature / power consumption increases, it is temporarily downgraded to JS / WebCrypto serial incremental mode. Before switching, export / import and image verification are performed; if the image fails, the original engine is maintained and downgraded to b, K binary search convergence until recovery is achieved, such as... Figure 3 As shown.

[0086] To ensure consistent results when switching between different engines, the system maintains a serializable hash state:

[0087] 1. Serial incremental class, such as SHA256: The state includes the internal working vector and the number of bytes n processed; during switching, export_state(engine_A)→import_state(engine_B) is executed, and updates continue with the same fragment boundary after switching.

[0088] 2. Tree hash classes, such as BLAKE3: The state consists of a list of leaf node summaries and some intermediate nodes; during switching, the deterministic order of merging nodes with the same fan-out f is guaranteed, and the state transition is completed before the next round of reduction.

[0089] 3. Equivalence check: During the switch, for the most recent m shards, such as m=1–4, the default is 2, the dual-engine image update is performed and the intermediate digest is compared; if they are inconsistent, the switch is rolled back to the previous consistent state and the single engine is forced to recalculate.

[0090] 4. Deterministic constraints: Fixed fragmentation order, byte order, necessary padding rules and tree merging order to ensure that H(F) is unique under any concurrent and switching path.

[0091] Counting and Units:

[0092] (1) The mirror window m is the number of the most recent shards for switching consistency checks. m takes the value [1,4] and the default value is 2.

[0093] (2) The number of bytes processed n is a 64-bit unsigned counter, in Byte; when the underlying implementation is limited to 53-bit integers, such as some JS environments, it is stored in segments of high and low bits of dual 32 bits or in the form of BigInt and a fixed endianness is used during serialization. The endianness or little endianness is uniformly configured by the system.

[0094] To avoid jitter, the fragment size b uses a step size of "× / ÷2" or binary division, and the concurrency K is limited by an upper / lower bound of [1, 32]; a hysteresis threshold is set for engine switching. Lag threshold It is recommended to use 2-3 trigger cycles. Reverse switching is prohibited during the cooldown period. Rollback is only permitted after the cooldown period ends and coverage and relative interval width have recovered to the target. Adjusted parameters are immediately fed back into the next round of feature construction and ETA estimation, forming a closed loop from monitoring to decision-making to execution. This ensures the system can respond quickly and adaptively optimize when performance changes occur. Figure 3 As shown.

[0095] S109, Convergence determination and process iteration;

[0096] Steps S101 to S108 are executed cyclically after each partition. Convergence is determined when any of the following conditions are met, and the learning step size is reduced or the current policy is fixed to reduce unnecessary adjustments:

[0097] 1. Interval width Less than the preset threshold.

[0098] 2. Relative interval width Less than the preset threshold.

[0099] 3. The residuals of multiple consecutive segments are all within acceptable ranges and there are no abrupt change signals.

[0100] As an optional example, this application provides initial state parameters and preprocessing refinement terms, which can be used as a supplement to steps S101 and S102.

[0101] 1. Parameter initialization:

[0102] The regularization parameter λ is used to control the model complexity and forgetting intensity, and its value can be obtained by grid search on a logarithmic scale.

[0103] The covariance matrix P is initialized as the product of the reciprocal of the regularization parameter λ and the identity matrix I.

[0104] The quantile set can be the median quantile, for example, τ=0.5, or two confidence quantiles, for example, τ=0.025 and τ=0.975, which can be selected based on the sample size and tail risk preference.

[0105] The detection module initializes the reference mean μ0 and initial cumulative value M0 to zero, sets the historical minimum value m0 to zero, and sets the detection threshold. It is formulated based on the robustness criteria of the residuals.

[0106] 2. Preprocessing strategy:

[0107] The feature vectors are centered or normalized to satisfy the assumptions of linear regression and improve model performance.

[0108] The sliding window retains only the most recent few data entries, such as 64, 128, or 256, and uses incremental updates to control memory usage and computational load.

[0109] To improve the robustness of the model to outliers, quantile truncation or robust standardization using the median and quantile intervals can be employed.

[0110] On the browser side, prioritize using lightweight floating-point format and TypedArray to store data in order to reduce memory usage.

[0111] As an alternative example, in step S107, in addition to the cumulative sum method, a detection method using the difference between the cumulative deviation and the historical minimum value as a statistic can also be used.

[0112] Specifically, with the increment of the residual sequence The cumulative amount is composed of tolerance offset δ. At the same time, maintain the historical minimum value Current cumulative amount Compared with historical minimum The difference between Exceeding the threshold This is considered drift. Threshold The robust scalar value of the residuals can be adaptively set according to the window length. The accumulated amount will be calculated upon triggering. Compared with historical minimum The threshold is reset or decayed, and then restored to the normal threshold after entering the cooling period to avoid frequent false alarms.

[0113] As an alternative example, to adapt to different operating environments, the present invention provides the following variations:

[0114] Additional Example A: Browser-based;

[0115] A fixed-size sliding window and incremental updates are used to avoid invalid loops. The covariance matrix can be approximated with a low rank or stored sparsely to reduce memory and computational overhead. Matrix operations are accelerated primarily through WebAssembly or WebGL, leveraging the browser's built-in high-performance computing capabilities. Computationally intensive logic such as matrix operations and model updates is executed in WebWorker threads, while the main thread is only responsible for event management and result display, avoiding blocking the main thread and improving user experience. Finally, concurrency and update frequency are dynamically adjusted based on the inter-frame idle ratio and device temperature trends to optimize energy consumption and system responsiveness.

[0116] Additional Implementation Example B: Mobile Terminal;

[0117] To adapt to the limited computing and memory resources of mobile devices, feature dimensions are reduced or downsampling techniques are employed to shorten window lengths. The graphics processing units or dedicated AI chips of mobile devices are fully utilized to accelerate matrix operations and model inference. Finally, the model update and detection frequency are dynamically adjusted based on factors such as network conditions, battery level, and device temperature, ensuring both performance and energy efficiency.

[0118] Additional Implementation Example C: Server-side High Concurrency;

[0119] The computational tasks are distributed across multi-threaded or distributed nodes, and high-performance numerical libraries, such as BLAS or GPU parallel computing, are used for acceleration to handle large-scale real-time data streams. Detection thresholds and model update strategies are centrally managed, and the window length is appropriately increased to utilize more historical data to improve prediction stability. Finally, independent statistical contexts and parameter sets are set for different task queues or users to avoid mutual interference and ensure the prediction accuracy and stability of each task.

[0120] As an optional example, this application also provides the following implementation parameter suggestions and complexity budget:

[0121] The regularization parameter λ can be searched on a logarithmic scale, for example, from 1e6 to 1e2. λ is briefly increased after drift to accelerate stabilization, and then empirically falls back in the interval [1e6, 1e2] after a cooling period.

[0122] For the quantile pair τ, it can be 0.2 or 0.8, or 0.025 or 0.975, depending on the sample size and tail risk preference. The quantile regression learning rate η is [1e4, 1e2], decreasing with the sample size by 1 / √t or in a stepwise manner.

[0123] The sliding window length W can be 64, 128, or 256 segments, adjusted according to the data change rate and memory budget.

[0124] For the emergency cooling period It can be divided into five to ten segments for stabilizing the system after drift detection.

[0125] For trigger threshold It is set according to the robustness scale of the residuals, such as the median absolute deviation (MAD), and the tolerance offset (δ) is set according to a small multiple of the robustness scale of the residuals.

[0126] For scheduling parameters, the fragment size ranges from 64 kilobytes to 4 megabytes, the concurrency ranges from 1 to 32, and the engine switching is set with a lag threshold and a backoff hook to avoid frequent jitter.

[0127] When calculating complexity and resource budget, the time complexity is O(D^2) quadratic of the feature dimension D, and the memory complexity is O(D^2) quadratic of the feature dimension D plus the window length W; the feature dimension O(D^2) quadratic of the feature dimension D is used to store the covariance matrix P, and the window length W is used to store historical data.

[0128] The time complexity of a single-step update is linear O(D) of the feature dimension D.

[0129] When using a low-rank approximation, the overhead of accessing and multiplying the covariance matrix decreases linearly with the chosen rank R, for example, O(DR). The recommended rank R is 16–64, depending on D, and should be based on covering ≥90% of the spectral energy.

[0130] Prioritize on-demand triggering and batch vectorization in browsers and mobile devices to avoid hot looping and optimize actual running efficiency.

[0131] This invention also considers various abnormal situations and provides corresponding degradation handling strategies to enhance the robustness of the system:

[0132] When encountering extreme long tails, i.e., when the relative interval width continuously exceeds the threshold and the coverage is insufficient, increase the tail strength of the quantile pairs, for example, by switching from {0.2, 0.8} to {0.025, 0.975}, and extend the cooldown period. This allows for a more conservative estimate of uncertainty and gives the system more time to recover.

[0133] When measurement points are missing, the short-term missing observation data is filled with the most recent robust statistics, such as the median; if there are consecutive missing points, interval fusion is paused, and only point estimates are output with an upper limit of uncertainty marked to avoid distortion of interval estimates due to insufficient data.

[0134] When the clock drifts, time synchronization is performed based on event-driven timestamps; system time jump samples are removed from the sliding window and counted separately to prevent erroneous data from affecting model updates.

[0135] If engine switching fails, and the performance does not improve or the switching fails after attempting to switch engines, the current engine is maintained and the shard size b and concurrency K are downgraded in a binary manner until the coverage and relative interval width return to the threshold, ensuring that the system can gradually recover stability.

[0136] When the state is corrupted or verification fails, if the mirror verification fails or the intermediate digest verification fails, roll back to the previous consistent state, reduce the size of the shard b, and prohibit the engine from switching back until it is consistent for r consecutive times, such as r=2–5, with the default being 3.

[0137] When cross-platform consistency is required, byte order differences between different platforms are resolved through a unified serialization protocol, such as fixed big-endian or little-endian; all intermediate digests are normalized as byte sequences before participating in reduction.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-performance file hash calculation method based on dual-engine adaptive switching, characterized in that, The process includes the following steps: obtaining the observation samples generated by the sliced ​​execution and mapping the observation features to construct a high-dimensional feature vector. The observed samples include fields such as slice size, read latency, garbage collection information, concurrency, engine type, queue length, main thread blocking, and processor usage; based on the high-dimensional feature vector The online master model is updated using recursive least squares with ridge regularization. The update includes: during initialization, taking the weight vector. The approximate inverse of the covariance matrix is ​​the zero vector. Regular expression parameters The reciprocal of the identity matrix The product of, i.e. ; For new samples, use high-dimensional feature vectors Compared to actual time spent The following updates will be made: Calculate the predicted value With residual ; Calculate the gain vector Specifically, this includes: With the current weight vector Obtain the predicted value Then, the residual is obtained by subtracting the predicted value ŷ from the actual value y. ; Multiply the current P by φ and correct it with a normalization factor to obtain the gain vector. ; Update weights ; in Sherman–Morrison form Perform rank-one correction Simultaneously, maintain the quantile regression model to obtain upper and lower quantile predictions; calculate the remaining time point estimates and intervals based on the online master model and quantile regression model, perform Gaussian approximation and quantile interval fusion, and generate scheduling parameters for shard size, concurrency and engine selection; perform mutation detection on the prediction residuals, execute hash state serialization and inter-engine migration and mirror verification when the trigger conditions are met, and write back the scheduling parameters to the feature construction and scheduling execution of subsequent shards.

2. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 1, characterized in that: The mapping process for observed features includes: numerically normalizing the fragment byte size and read latency, classifying the garbage collection identifier and engine type, combining the processed features with the processor usage and concurrency, and constructing the high-dimensional feature vector containing engine and concurrency interaction terms. .

3. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 2, characterized in that: The acquisition of observation samples includes: collecting reading latency, garbage collection flag and time since the last garbage collection, current queue length and main thread blocking index when the fragmentation is completed, and then filling in missing or abnormal observations with robust statistics using a sliding window before writing them into the observation samples.

4. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 1, characterized in that; The recursive least squares method further includes: adjusting the covariance matrix when the numerical timing is ill-fitting. Perform symmetry transformation and spectral clipping.

5. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 4, characterized in that: Maintaining the quantile regression model includes: performing subgradient updates based on a check function on the upper and lower quantiles respectively, uniformly using an indicator function to represent the residual direction, and applying the same high-dimensional feature vector... Apply a consistency constraint where the upper bound is not lower than the lower bound.

6. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 1, characterized in that, The calculation of the remaining time point estimate and interval includes: calculating the number of remaining fragments based on the remaining bytes and the current fragment size; constructing the high-dimensional feature vector for each future fragment; recording the algorithm and mode indication during tree reduction and incorporating them into the high-dimensional feature vector for subsequent estimation.

7. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 6, characterized in that, The fusion of Gaussian approximation and quantile intervals includes: using the Gaussian interval obtained by approximating the residual variance and covariance and the upper and lower quantile regression intervals to form candidate lower and upper bounds respectively, and completing the fusion by taking the smaller of the lower bound and the larger of the upper bound, and introducing a correlation inflation coefficient when accumulating variances.

8. The high-performance file hash calculation method based on dual-engine adaptive switching according to claim 1, characterized in that, Mutation detection includes: calculating the Page-Hinkley or cumulative sum statistic for the predicted residual sequence, making a determination using a offset and threshold configuration derived from the robust residual scale, and updating the residual mean using a sliding window before making the determination.

9. A high-performance file hash calculation method based on dual-engine adaptive switching according to claim 8, characterized in that, Hash state serialization and inter-engine migration include: serializing the current engine's internal working vector, processed bytes, and intermediate digest, and deserializing them in the target engine; performing mirror verification; rolling back to the previous consistent state when inconsistent; and a lag strategy that restricts reverse switching.

10. A high-performance file hash calculation method based on dual-engine adaptive switching according to claim 9, characterized in that, The scheduling parameter write-back includes: after a mutation is triggered, maintaining the regularization parameter and alarm threshold by increasing and the learning rate by decreasing during the cooldown period; after the cooldown period ends, restoring according to the preset annealing curve; and simultaneously updating the partition size and concurrency with a binary step size and synchronizing them to the subsequent high-dimensional feature vectors. The structure.

Images