Big data-based data optimization method, device and electronic equipment

Abstract

The present application relates to the technical field of computer, especially to a data optimization method and device based on big data and electronic equipment, the method comprises the following steps: determining the data health score according to the disorder arrival rate and the null proportion; determining the drift feature based on the sliding skewness and kurtosis of the data source input throughput; determining the data pipeline pressure feature according to the data health score and the drift feature; determining the resource congestion feature according to the CPU flow limiting time and the heap memory usage; determining the system running state level according to the data pipeline pressure index and the resource congestion feature, and then adjusting the data cleaning and deduplication strategy and the data storage and archiving strategy. The present application realizes the hierarchical adaptive optimization of the cleaning strategy and the storage strategy of the stream data processing pipeline through the linkage analysis of the fusion of the data distribution form drift and the resource congestion degree.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data optimization method, apparatus, and electronic device based on big data. Background Technology

[0002] With the widespread adoption of cloud computing and IoT technologies, large data centers and edge computing nodes need to process massive amounts of real-time streaming data. Existing stream processing frameworks mostly rely on static thresholds or rules set by human experience to monitor and optimize data pipelines. However, the characteristic distribution of streaming data will dynamically drift due to business peaks and troughs, network jitter, and resource contention, making it difficult for static thresholds to accurately capture the non-linear degradation trend of data quality.

[0003] Furthermore, traditional resource monitoring metrics (such as CPU utilization and memory usage) are disconnected from the health of the data itself, causing optimization strategies to often lag behind actual failures. When the system faces increasingly disordered data, a rising proportion of null values, or changes in throughput distribution, existing methods lack a means to link statistical changes in data characteristics with the congestion level of underlying computing resources, making it difficult to achieve an optimal balance between computing resources and storage costs while ensuring data integrity. Summary of the Invention

[0004] The purpose of this invention is to provide a data optimization method, apparatus, and electronic device based on big data to solve at least one of the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A data optimization method based on big data includes:

[0007] The data health score is determined based on the out-of-order arrival rate and the proportion of null values.

[0008] The drift characteristics are determined based on the sliding skewness and kurtosis of the data source input throughput.

[0009] Determine the data pipeline pressure characteristics based on data health scores and drift characteristics;

[0010] Determine resource congestion characteristics based on CPU throttling time and heap memory utilization;

[0011] The system's operating status level is determined based on the data pipeline pressure index and resource congestion characteristics, and then the data cleaning and deduplication strategies and data storage and archiving strategies are adjusted accordingly.

[0012] Furthermore, the ratio of the number of out-of-order events whose timestamps lag behind the watermarks to the total number of events within the current real-time period is calculated and denoted as the out-of-order factor Lx.

[0013] Calculate the ratio of the number of null fields to the total number of fields in the current real-time period, denoted as the missing factor Qm;

[0014] The data health score Dh is determined based on the disorder factor and the missing factor:

[0015] Dh=1-[w1×min(1,Lx / LO)+w2×min(1,Qm / Q0)]

[0016] Where L0 is the out-of-order tolerance threshold, Q0 is the null value tolerance threshold, w1 is the out-of-order weight, w2 is the missing weight, and w1+w2=1.

[0017] Furthermore, at the end of each statistical window, the data distribution of the data source input throughput within that window is analyzed, and the sliding skewness and sliding kurtosis are calculated:

[0018] The expression for the slip bias is: ;

[0019] The expression for the sliding kurtosis is:

[0020] ;

[0021] The offset feature Cd is determined based on the sliding skewness Sk and sliding kurtosis Ku of the data source input throughput, and Cd = max(0, (|Sk-Skb| / σSk) + |Ku-Kub| / σKu-γ);

[0022] Where xi is the i-th sampled value of the data source input throughput within the statistical window, xp is the average value of each sampled value of the data source input throughput within the statistical window, n is the number of sampled points of the data source input throughput within the statistical window, Skb is the sliding skewness baseline value during the calibration phase, Kub is the sliding kurtosis baseline value during the calibration phase, γ is the preset offset adjustment coefficient, and σSk and σKu are the sample standard deviations of the skewness sequence and the kurtosis sequence during the calibration phase.

[0023] Furthermore, at the end of each statistical window, the arithmetic mean of the data health scores Dh for all real-time periods within that window is obtained, denoted as Dha. Combined with the drift feature Cd, the data pipeline pressure index Pi is determined.

[0024] Pi=β1×(1-Dha)+β2×lg(5×Cd+1) / lg6;

[0025] Where β1 is the quality degradation weight, β2 is the drift effect weight, and β1+β2=1.

[0026] Furthermore, within each real-time period, the CPU rate limiting time percentage Thr and heap memory utilization Me of the operator instance are read from the container monitoring interface to determine the resource congestion feature Rc, Rc=λ1×max(0,(Thr-Thr0) / (1-Thr0))+λ2×max(0,(Me-Me0) / (1-Me0)).

[0027] Where Thr0 is the security rate limiting threshold, Me0 is the security memory threshold, λ1 is the rate limiting weight, λ2 is the utilization weight, and λ1+λ2=1.

[0028] Furthermore, at the end of each optimization cycle, the 90th percentile of the data pipeline pressure index Pi generated by all statistical windows included in the optimization cycle is calculated and denoted as Piy, and the 90th percentile of the resource congestion feature Rc generated by all 360 real-time cycles in the optimization cycle is denoted as Rcy.

[0029] Determine the system operating status level L based on Piy and Rcy:

[0030] If Piy≤Pth1 and Rcy≤Rth, then L=0;

[0031] If Piy > Pth1 and Rcy ≤ Rth, then L = 1;

[0032] If Rcy > Rth and Piy ≤ Pth1, then L = 2;

[0033] If Piy > Pth1 and Rcy > Rth, then L = 3;

[0034] Wherein, Pth1 is the first preset pipeline pressure, and Rth is the preset congestion characteristic.

[0035] Furthermore, at the end of each optimization cycle, if the system operating status level L is 1 or 3, the data cleaning and deduplication strategy is adjusted according to the current specific value of Piy:

[0036] If Pth1 < Piy ≤ Pth2, perform a minor adjustment;

[0037] If Pth2 < Piy ≤ Pth3, perform a moderate adjustment based on the minor adjustment;

[0038] If Piy > Pth3, perform a heavy adjustment on top of the moderate adjustment;

[0039] If the system operating status level L is 0 or 2, the current data cleaning and deduplication strategy will remain unchanged.

[0040] Wherein, Pth2 is the second preset pipeline pressure index, and Pth3 is the third preset pipeline pressure index.

[0041] Furthermore, at the end of each archiving cycle, the system scans the time-series data of the system operating status level L recorded within that archiving cycle and adjusts the data storage and archiving strategy accordingly.

[0042] If the maximum value of L in the past 24 hours is ≤1: adopt the conventional storage strategy, write the original data to the standard SSD storage layer in the first compression format, and retain it for 30 days;

[0043] If the maximum value of L in the past 24 hours is 2: enable the downgraded write strategy, perform 1% random sampling and discard before writing data, and switch the compression algorithm to the second compression format with a higher compression ratio to reduce storage space occupation and write I / O pressure;

[0044] If the maximum value of L in the past 24 hours is 3: enable the emergency archiving strategy, retain only the average, maximum and minimum values ​​after every 5-minute window, discard the original detailed data, and send a "data integrity degradation" alarm notification to the operation and maintenance monitoring platform.

[0045] According to another aspect of this application, a data optimization device based on big data is provided, comprising:

[0046] The data health assessment unit is used to determine the data health score based on the out-of-order arrival rate and the proportion of null values.

[0047] The distribution drift detection unit is used to determine drift characteristics based on the sliding skewness and kurtosis of the data source input throughput;

[0048] The pipeline pressure calculation unit is used to determine the pipeline pressure characteristics based on the data health score and drift characteristics.

[0049] The resource congestion monitoring unit is used to determine resource congestion characteristics based on CPU throttling time and heap memory utilization.

[0050] The strategy hierarchical execution unit is used to determine the system operating status level based on the data pipeline pressure index and resource congestion characteristics, and then adjust the data cleaning and deduplication strategies and data storage and archiving strategies.

[0051] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0052] One or more processors;

[0053] Storage device for storing one or more programs;

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement the big data-based data optimization method.

[0055] The beneficial effects of this invention are as follows: This solution achieves adaptive optimization of streaming data processing pipelines by constructing a multi-dimensional joint evaluation system that integrates data quality metadata, characteristic distribution statistical indicators, and computing resource operating status. First, the solution introduces sliding skewness and kurtosis as quantitative characteristics of data drift. Compared to traditional monitoring methods that rely solely on mean or variance, this approach can capture subtle changes in data distribution patterns earlier and more sensitively, providing early warnings of pipeline pressure anomalies. Second, the solution establishes a collaborative judgment matrix for data health and resource congestion. Through a hierarchical strategy engine, it dynamically adjusts the cleaning logic and storage archiving granularity, achieving an intelligent trade-off between data integrity and system throughput performance. Finally, this solution requires no intrusive modification to existing stream processing business code; it can be deployed using only framework bypass interfaces and container orchestration systems. It possesses high engineering versatility and computational lightweightness, significantly reducing the operational complexity and total storage cost of large-scale real-time data systems. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the data optimization method based on big data in this embodiment.

[0058] Figure 2 This is a flowchart illustrating the strategy adjustment method in this embodiment.

[0059] Figure 3 This is a schematic diagram of the data optimization device based on big data in this embodiment.

[0060] Figure 4 This is a schematic diagram of the electronic device in this embodiment. Detailed Implementation

[0061] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0062] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] Specifically, this embodiment is applied to real-time streaming data processing systems in large data centers or edge computing nodes.

[0064] Please see Figure 1 As shown, this is a flowchart illustrating the data optimization method based on big data in this embodiment. Before the method is executed, the system synchronously collects metadata and operational status data through monitoring agents deployed at various stages of the data pipeline; the data includes:

[0065] Data quality metadata: Real-time collection of event time latency, out-of-order arrival rate, and proportion of null fields for each data partition through the Metrics interface of stream processing frameworks (such as Apache Flink or Spark Streaming);

[0066] Feature distribution data: A statistical computing unit is deployed in the bypass of the feature engineering node to collect the sliding window skewness and kurtosis of at least one numerical monitoring feature in the streaming data processing pipeline. The numerical monitoring feature refers to pipeline operation status indicators or business aggregation indicators recorded in continuous real value form, including but not limited to: data source input throughput, operator processing latency, etc. In this embodiment, data source input throughput is used as an exemplary monitoring feature for specific explanation, and data source global input throughput is used as an exemplary monitoring feature for specific explanation. The data source global input throughput refers to the average rate obtained by dividing the total number of all partition events received at the data pipeline inlet in a single real-time period by the period duration. Those skilled in the art can replace or add other numerical features according to actual monitoring needs.

[0067] Computational resource data: Obtain the CPU throttling time percentage and heap memory usage of each operator instance from a container orchestration system (such as Kubernetes Metrics Server).

[0068] This embodiment does not specifically limit the communication protocol of the above data; those skilled in the art can freely set it. The method uses a 10-second real-time cycle, a 5-minute statistical window, an 1-hour optimization cycle, and a 24-hour archiving cycle.

[0069] The method includes:

[0070] Step S1: Determine the data health score based on the out-of-order arrival rate and the proportion of null values.

[0071] Specifically, the ratio of the number of out-of-order events whose timestamps lag behind the watermarks to the total number of events within the current real-time period is calculated and denoted as the out-of-order factor Lx.

[0072] Calculate the ratio of the number of null fields to the total number of fields in the current real-time period, denoted as the missing factor Qm;

[0073] The data health score Dh is determined based on the disorder factor and the missing factor:

[0074] Dh=1-[w1×min(1,Lx / LO)+w2×min(1,Qm / Q0)]

[0075] Where L0 is the out-of-order tolerance threshold, Q0 is the null value tolerance threshold, w1 is the out-of-order weight, w2 is the missing weight, and w1+w2=1.

[0076] Preferably, in this embodiment, the out-of-order tolerance threshold is 0.1, the null value tolerance threshold is 0.05, the out-of-order weight is 0.6, and the missing value weight is 0.4.

[0077] Specifically, by quantifying and weighting out-of-order and missing factors, this step can transform discrete metadata alarms into normalized continuous health scores, providing a unified and smooth data quality input benchmark for subsequent pipeline pressure assessment and avoiding strategy jitter caused by frequent exceedances of a single indicator.

[0078] Please continue reading. Figure 1 As shown, the data optimization method based on big data also includes:

[0079] Step S2: Determine the drift characteristics based on the sliding skewness and kurtosis of the data source input throughput.

[0080] Specifically, at the end of each statistical window, the data distribution of the data source input throughput within that window is analyzed, and the moving skewness and moving kurtosis are calculated:

[0081] The expression for the slip bias is:

[0082] ;

[0083] The expression for the sliding kurtosis is:

[0084] ;

[0085] The offset feature Cd is determined based on the sliding skewness Sk and sliding kurtosis Ku of the data source input throughput, and Cd = max(0, (|Sk-Skb| / σSk) + |Ku-Kub| / σKu-γ);

[0086] Where xi is the i-th sampled value of the data source input throughput within the statistical window, xp is the average value of each sampled value of the data source input throughput within the statistical window, n is the number of sampled points of the data source input throughput within the statistical window, Skb is the sliding skewness baseline value during the calibration phase, Kub is the sliding kurtosis baseline value during the calibration phase, γ is the preset offset adjustment coefficient, and σSk and σKu are the sample standard deviations of the skewness sequence and the kurtosis sequence during the calibration phase.

[0087] Preferably, in this embodiment, the calibration phase is defined as the first 24 hours of continuous and stable operation after the system goes online; during this phase, the skewness sequence and kurtosis sequence are calculated in units of 5-minute statistical windows, and a total of 288 skewness samples and 288 kurtosis samples are obtained; Skb and Kub are the arithmetic mean of the 288 samples, σSk and σKu are the sample standard deviations of the 288 samples, and the preset offset adjustment coefficient is 1.5.

[0088] Specifically, using higher-order statistical moments instead of traditional mean monitoring can effectively identify subtle anomalies in the symmetry of throughput distribution and the thickness of the tail in streaming data. Even if the data does not show significant fluctuations at the mean level, changes in skewness and kurtosis can reveal early signs of changes in upstream business logic or queue backlogs within the pipeline, improving the system's sensitivity to hidden faults.

[0089] Please continue reading. Figure 1 As shown, the data optimization method based on big data also includes:

[0090] Step S3: Determine the data pipeline pressure characteristics based on the data health score and drift characteristics.

[0091] Specifically, at the end of each statistical window, the arithmetic mean of the data health scores Dh for all real-time periods within that window is obtained, denoted as Dha. Combined with the drift feature Cd, the data pipeline pressure index Pi is determined.

[0092] Pi=β1×(1-Dha)+β2×lg(5×Cd+1) / lg6;

[0093] Where β1 is the quality degradation weight, β2 is the drift effect weight, and β1+β2=1.

[0094] Preferably, in this embodiment, β1 is 0.7 and β2 is 0.3.

[0095] Specifically, a pressure index characterizing the overall pressure-bearing state of a pipeline is constructed by nonlinearly coupling the degree of micro-level data quality degradation with the degree of macro-level feature distribution drift. This index effectively integrates information from both immediate errors and trend drift, enabling a more accurate depiction of the overall disturbance intensity currently faced by the stream processing system.

[0096] Please continue reading. Figure 1 As shown, the data optimization method based on big data also includes:

[0097] Step S4: Determine resource congestion characteristics based on CPU throttling time and heap memory utilization.

[0098] Specifically, within each real-time period, the CPU rate limiting time percentage Thr and heap memory utilization Me of the operator instance are read from the container monitoring interface, and then the resource congestion feature Rc is determined, Rc=λ1×max(0,(Thr-Thr0) / (1-Thr0))+λ2×max(0,(Me-Me0) / (1-Me0)).

[0099] Where Thr0 is the security rate limiting threshold, Me0 is the security memory threshold, λ1 is the rate limiting weight, λ2 is the utilization weight, and λ1+λ2=1.

[0100] Preferably, in this embodiment, the security rate limiting threshold is 0.1, the security memory threshold is 0.75, the rate limiting weight is 0.5, and the utilization rate weight is 0.5.

[0101] Specifically, by focusing on the actual resource constraints in a containerized runtime environment and selecting CPU throttling time percentage and heap memory utilization as core parameters, this approach can accurately reflect the execution blocking risk of operator instances due to insufficient resource quotas. Compared to simply observing absolute usage, this feature better reflects the actual erosion effect of resource contention on processing performance.

[0102] Please continue reading. Figure 1 As shown, the data optimization method based on big data also includes:

[0103] Step S5: Determine the system operating status level based on the data pipeline pressure index and resource congestion characteristics, and then adjust the data cleaning and deduplication strategy and the data storage and archiving strategy.

[0104] Please see Figure 3 As shown, the strategy adjustment method includes:

[0105] Step S51: Determine the system operating status level based on the data pipeline pressure index and resource congestion characteristics.

[0106] Specifically, at the end of each optimization cycle, the 90th percentile of the data pipeline pressure index Pi generated by all statistical windows included in the optimization cycle is calculated and denoted as Piy, and the 90th percentile of the resource congestion feature Rc generated by all 360 real-time cycles in the optimization cycle is denoted as Rcy.

[0107] Determine the system operating status level L based on Piy and Rcy:

[0108] If Piy≤Pth1 and Rcy≤Rth, then L=0;

[0109] If Piy > Pth1 and Rcy ≤ Rth, then L = 1;

[0110] If Rcy > Rth and Piy ≤ Pth1, then L = 2;

[0111] If Piy > Pth1 and Rcy > Rth, then L = 3;

[0112] Wherein, Pth1 is the first preset pipeline pressure, and Rth is the preset congestion characteristic.

[0113] Preferably, in this embodiment, the first preset pipeline pressure index is 0.6 and the preset congestion characteristic is 0.5.

[0114] Specifically, by comparing quantile statistics with thresholds, continuous pressure and congestion values ​​are mapped to discrete system state levels, clearly distinguishing multiple operating modes such as "data quality degradation-dominated," "resource bottleneck-dominated," and "comprehensive deterioration-dominated." This level classification provides a clear decision-making basis for subsequent implementation of differentiated optimization actions.

[0115] Please continue reading. Figure 2 As shown, the strategy adjustment method includes:

[0116] Step S52: Adjust the data cleaning and deduplication strategy based on the system operating status level.

[0117] Specifically, at the end of each optimization cycle, if the system operating status level L is 1 or 3, the data cleaning and deduplication strategy is adjusted according to the current Piy value:

[0118] If Pth1 < Piy ≤ Pth2, perform a minor adjustment, which is to change the handling strategy for out-of-order events from "waiting for late data" to "discarding late data directly", and shorten the watermark idle timeout from the default 60 seconds to 30 seconds;

[0119] If Pth2 < Piy ≤ Pth3, perform a medium adjustment on the basis of a light adjustment. The medium adjustment is to enable the null value default filling logic and fill the value with the median of the field in the most recent statistical window.

[0120] If Piy > Pth3, perform a heavy adjustment on the basis of the medium adjustment. The heavy adjustment is to expand the deduplication window of the event time from the default 10 seconds to 60 seconds to eliminate duplicate calculations caused by data delays.

[0121] If the system operating status level L is 0 or 2, the current data cleaning and deduplication strategy will remain unchanged.

[0122] Wherein, Pth2 is the second preset pipeline pressure index, and Pth3 is the third preset pipeline pressure index.

[0123] Preferably, in this embodiment, the second preset pipeline pressure index is 0.75 and the third preset pipeline pressure index is 0.9.

[0124] Specifically, a dynamic mapping mechanism between status levels and cleaning strategies was established. When the pipeline pressure is identified as being caused by data quality degradation, the tolerance of data processing can be automatically increased or the severity of data discarding can be reduced. By sacrificing some late or redundant data in exchange for the release of computing resources and the recovery of latency in the core processing link, a dynamic balance between the timeliness and accuracy of stream computing results is achieved.

[0125] Please continue reading. Figure 1 As shown, the strategy adjustment method further includes:

[0126] Step S53: Adjust the data storage and archiving strategy based on the system operating status level.

[0127] Specifically, at the end of each archiving cycle, the system scans the time-series data of the system operating status level L recorded within that archiving cycle and adjusts the data storage and archiving strategy accordingly.

[0128] If the maximum value of L in the past 24 hours is ≤1: adopt the conventional storage strategy, write the original data to the standard SSD storage layer in the first compression format, and retain it for 30 days;

[0129] If the maximum value of L in the past 24 hours is 2: enable the downgraded write strategy, perform 1% random sampling and discard before writing data, and switch the compression algorithm to the second compression format with a higher compression ratio to reduce storage space occupation and write I / O pressure;

[0130] If the maximum value of L in the past 24 hours is 3: enable the emergency archiving strategy, retain only the average, maximum and minimum values ​​after every 5-minute window, discard the original detailed data, and send a "data integrity degradation" alarm notification to the operation and maintenance monitoring platform.

[0131] Preferably, in this embodiment, the first compression format is Snappy, the second compression format is ZSTD, and the compression level is set to 9.

[0132] Specifically, based on the historical peak performance during the archiving period, the write fidelity and compression intensity of the storage layer are adaptively adjusted. When the system is healthy, full data persistence is guaranteed; when encountering resource congestion or abnormal fluctuations, it automatically degrades to sampling and discarding or aggregating and archiving non-critical details, thereby significantly reducing the write amplification factor and space occupation cost of the back-end storage system while ensuring traceability.

[0133] Please see Figure 3 As shown, the big data-based data optimization device includes:

[0134] The data health assessment unit is used to determine the data health score based on the out-of-order arrival rate and the proportion of null values.

[0135] The distribution drift detection unit is used to determine drift characteristics based on the sliding skewness and kurtosis of the data source input throughput;

[0136] The pipeline pressure calculation unit is used to determine the pipeline pressure characteristics based on the data health score and drift characteristics.

[0137] The resource congestion monitoring unit is used to determine resource congestion characteristics based on CPU throttling time and heap memory utilization.

[0138] The strategy hierarchical execution unit is used to determine the system operating status level based on the data pipeline pressure index and resource congestion characteristics, and then adjust the data cleaning and deduplication strategies and data storage and archiving strategies.

[0139] The big data-based data optimization device provided in this application embodiment can execute the big data-based data optimization method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] From a hardware perspective, to implement the data optimization method based on big data in a computer, this application also provides an electronic device; please refer to [link to relevant documentation]. Figure 4 As shown, it is a schematic diagram of the structure of the electronic device described in this application, including:

[0141] The system comprises a processor 1, a memory 2, a communication interface 3, and a bus 4; wherein the processor 1 and the memory 2, and the memory 2 and the communication interface 3, transmit data via the bus 4; the processor is used to process data in the memory and generate commands, the memory is used to store data, the communication interface is used to receive and send data, and the bus is used to realize data transmission between the processor, the memory, and the communication interface.

[0142] In this embodiment, the big data-based data optimization method can be implemented as a runnable computer program. When the computer program is loaded into the processor or into the memory and processed by the processor via the bus, one or more steps of the big data-based data optimization method can be executed.

[0143] This embodiment also provides a computer-readable storage medium for storing the computer-executable instructions. The computer-readable storage medium is a tangible physical storage medium that can store the computer program and various types of data used in the program. The physical storage medium includes, but is not limited to, existing physical storage media or combinations thereof, such as random access memory, read-only memory, optical disk, and hard disk.

[0144] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A data optimization method based on big data, characterized in that, include: The data health score is determined based on the out-of-order arrival rate and the proportion of null values. The drift characteristics are determined based on the sliding skewness and kurtosis of the data source input throughput. Determine the data pipeline pressure characteristics based on data health scores and drift characteristics; Determine resource congestion characteristics based on CPU throttling time and heap memory utilization; The system's operating status level is determined based on the data pipeline pressure index and resource congestion characteristics, and then the data cleaning and deduplication strategies and data storage and archiving strategies are adjusted accordingly.

2. The data optimization method based on big data according to claim 1, characterized in that, Calculate the ratio of the number of out-of-order events whose timestamps lag behind the watermarks to the total number of events in the current real-time period, denoted as the out-of-order factor Lx; Calculate the ratio of the number of null fields to the total number of fields in the current real-time period, denoted as the missing factor Qm; The data health score Dh is determined based on the disorder factor and the missing factor: Dh=1-[w1×min(1,Lx / LO)+w2×min(1,Qm / Q0)] Where L0 is the out-of-order tolerance threshold, Q0 is the null value tolerance threshold, w1 is the out-of-order weight, w2 is the missing weight, and w1+w2=1.

3. The data optimization method based on big data according to claim 2, characterized in that, At the end of each statistical window, the data distribution of the data source input throughput within that window is analyzed, and the moving skewness and moving kurtosis are calculated: The expression for the slip bias is: ; The expression for the sliding kurtosis is: ； The offset feature Cd is determined based on the sliding skewness Sk and sliding kurtosis Ku of the data source input throughput, and Cd = max(0, (|Sk-Skb| / σSk) + |Ku-Kub| / σKu-γ); Where xi is the i-th sampled value of the data source input throughput within the statistical window, xp is the average value of each sampled value of the data source input throughput within the statistical window, n is the number of sampled points of the data source input throughput within the statistical window, Skb is the sliding skewness baseline value during the calibration phase, Kub is the sliding kurtosis baseline value during the calibration phase, γ is the preset offset adjustment coefficient, and σSk and σKu are the sample standard deviations of the skewness sequence and the kurtosis sequence during the calibration phase.

4. The data optimization method based on big data according to claim 3, characterized in that, At the end of each statistical window, the arithmetic mean of the data health score Dh for all real-time periods within that window is obtained and denoted as Dha. Combined with the drift feature Cd, the data pipeline pressure index Pi is determined. Pi=β1×(1-Dha)+β2×lg(5×Cd+1) / lg6; Where β1 is the quality degradation weight, β2 is the drift effect weight, and β1+β2=1.

5. The data optimization method based on big data according to claim 4, characterized in that, Within each real-time period, the CPU rate limiting time percentage Thr and heap memory utilization Me of the operator instance are read from the container monitoring interface, and then the resource congestion feature Rc is determined, Rc=λ1×max(0,(Thr-Thr0) / (1-Thr0))+λ2×max(0,(Me-Me0) / (1-Me0)). Where Thr0 is the security rate limiting threshold, Me0 is the security memory threshold, λ1 is the rate limiting weight, λ2 is the utilization weight, and λ1+λ2=1.

6. The data optimization method based on big data according to claim 5, characterized in that, At the end of each optimization cycle, the 90th percentile of the data pipeline pressure index Pi generated by all statistical windows included in the optimization cycle is calculated and denoted as Piy, and the 90th percentile of the resource congestion feature Rc generated by all 360 real-time cycles in the optimization cycle is denoted as Rcy. Determine the system operating status level L based on Piy and Rcy: If Piy≤Pth1 and Rcy≤Rth, then L=0; If Piy > Pth1 and Rcy ≤ Rth, then L = 1; If Rcy > Rth and Piy ≤ Pth1, then L = 2; If Piy > Pth1 and Rcy > Rth, then L = 3; Wherein, Pth1 is the first preset pipeline pressure, and Rth is the preset congestion characteristic.

7. The data optimization method based on big data according to claim 6, characterized in that, At the end of each optimization cycle, if the system operating status level L is 1 or 3, the data cleaning and deduplication strategy is adjusted according to the current Piy value: If Pth1 < Piy ≤ Pth2, perform a minor adjustment; If Pth2 < Piy ≤ Pth3, perform a moderate adjustment based on the minor adjustment; If Piy > Pth3, perform a heavy adjustment on top of the moderate adjustment; If the system operating status level L is 0 or 2, the current data cleaning and deduplication strategy will remain unchanged. Wherein, Pth2 is the second preset pipeline pressure index, and Pth3 is the third preset pipeline pressure index.

8. The data optimization method based on big data according to claim 7, characterized in that, At the end of each archiving cycle, the system scans the time-series data of the system operating status level L recorded within that archiving cycle and adjusts the data storage and archiving strategy accordingly. If the maximum value of L in the past 24 hours is ≤1: adopt the conventional storage strategy, write the original data to the standard SSD storage layer in the first compression format, and retain it for 30 days; If the maximum value of L in the past 24 hours is 2: enable the downgraded write strategy, perform 1% random sampling and discard before writing data, and switch the compression algorithm to the second compression format with a higher compression ratio to reduce storage space occupation and write I / O pressure; If the maximum value of L in the past 24 hours is 3: enable the emergency archiving strategy, retain only the average, maximum and minimum values ​​over every 5-minute window, discard the original detailed data, and send a "data integrity degradation" alarm notification to the operation and maintenance monitoring platform.

9. A data optimization device based on big data, applied to the data optimization method based on big data as described in any one of claims 1-8, characterized in that, include: The data health assessment unit is used to determine the data health score based on the out-of-order arrival rate and the proportion of null values. The distribution drift detection unit is used to determine drift characteristics based on the sliding skewness and kurtosis of the data source input throughput; The pipeline pressure calculation unit is used to determine the pipeline pressure characteristics based on the data health score and drift characteristics. The resource congestion monitoring unit is used to determine resource congestion characteristics based on CPU throttling time and heap memory utilization. The strategy hierarchical execution unit is used to determine the system operating status level based on the data pipeline pressure index and resource congestion characteristics, and then adjust the data cleaning and deduplication strategies and data storage and archiving strategies.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data optimization method based on big data as described in any one of claims 1-8.

Images