Key index screening method supporting real-time updating
By combining a stream processing engine with a sliding time window for filtering, a dynamic index correlation network and uncertainty assessment are constructed, which solves the problems of low efficiency and high latency in the key index filtering in the prior art. This enables the identification and updating of key indicators in seconds, and improves the real-time monitoring and decision-making capabilities of the system.
Patent Information
- Application Number
- CN202610048465.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-14
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies struggle to quickly and accurately screen key indicators in the operation monitoring of complex systems. Furthermore, existing real-time analysis systems lack the ability to jointly model the structural dependencies and inherent uncertainties among indicators, resulting in excessive computational overhead and difficulty in meeting low-latency requirements.
The system receives multi-dimensional indicator observation data streams with timestamps through a stream processing engine. It uses sliding time windows for filtering and online updated statistical calculations to dynamically construct indicator correlation networks and uncertainty assessments. It then combines correlation and uncertainty scores for normalization and fusion to achieve second-level identification and updating of key indicators.
It enables real-time identification and updating of key indicators with low computational overhead, improving the accuracy and efficiency of real-time monitoring and decision-making in complex systems, and is suitable for high-speed streaming data scenarios.
Smart Images
Figure CN121502275A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data analysis technology, and in particular to a method for screening key indicators that supports real-time updates. Background Technology
[0002] In the operation and monitoring of complex systems, there are usually a large number of sensors or business indicators. Quickly and accurately filtering the key indicators among these is a core step in achieving real-time system monitoring and decision-making. Traditional key indicator filtering often relies on offline batch calculations, such as assessing stability based on historical coefficients of variation. Such methods cannot adapt to the actual needs of continuous data inflow and dynamic evolution of indicator relationships.
[0003] While existing real-time analysis systems can process streaming data, they often focus only on single-indicator threshold alarms and lack the ability to jointly model the structural dependencies and inherent uncertainties between indicators. Furthermore, frequent full recalculation of correlation matrices and statistics leads to a surge in computational overhead, making it difficult to meet low-latency requirements. Summary of the Invention
[0004] Therefore, it is necessary to provide a key indicator screening method that supports real-time updates, enabling dynamic maintenance of indicator correlation networks and uncertainty assessment with low computational overhead in a streaming data environment, and achieving second-level identification and updating of key indicators, in response to the aforementioned technical problems.
[0005] A method for filtering key metrics that supports real-time updates, the method comprising: The stream processing engine receives multi-dimensional indicator observation data streams containing timestamps, with each observation data containing the current values of multiple indicators; A sliding time window of a preset fixed length is maintained. The multidimensional index observation data stream is filtered according to the sliding time window, and only the latest valid samples within the sliding time window are retained to form a sample dataset. For each indicator in the sample dataset within the sliding time window, the mean, covariance, and cumulative second central moments are updated online to obtain the real-time mean, real-time covariance, and real-time cumulative second central moments of each indicator. The real-time standard deviation of each indicator is calculated based on the real-time mean, real-time covariance, and real-time cumulative second-order central moments of each indicator. Then, the real-time correlation coefficient between any two indicators is calculated based on the real-time covariance and real-time standard deviation. Combined with the preset correlation threshold, the indicator correlation network is dynamically constructed. At the same time, the node connection degree of each indicator in the correlation network is counted, and the node connection degree is used as the correlation score of the corresponding indicator. The real-time coefficient of variation for each indicator is calculated based on the real-time mean and the real-time standard deviation of each indicator, and the real-time coefficient of variation is used as the uncertainty score of the corresponding indicator. The correlation scores and uncertainty scores of each indicator are normalized separately, and then the normalized correlation scores and uncertainty scores are merged and calculated according to preset weights to obtain the dynamic comprehensive score of each indicator. Based on the dynamic comprehensive score of each indicator, key indicators are selected using preset screening rules to obtain a set of key indicators.
[0006] The aforementioned key indicator screening method that supports real-time updates achieves data timeliness control through streaming data access and sliding window screening, reduces computational overhead by leveraging incremental statistical algorithms, and ensures the rationality of screening results by integrating a dual-dimensional evaluation of correlation and uncertainty. It also supports dynamic expansion of indicator dimensions, enabling second-level identification and updating of key indicators. It is suitable for high-speed streaming data scenarios and can effectively improve the accuracy and efficiency of real-time monitoring and decision-making in complex systems. Attached Figure Description
[0007] Figure 1 This is a flowchart illustrating a key indicator filtering method that supports real-time updates in one embodiment. Figure 2 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0008] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0009] In one embodiment, such as Figure 1 As shown, a method for filtering key metrics that supports real-time updates is provided, including the following steps: Step 102: Receive multi-dimensional indicator observation data streams containing timestamps through the stream processing engine. Each observation data contains the current values of multiple indicators.
[0010] The stream processing engine can use mainstream data processing platforms such as Kafka and Flink. The timestamp is used to mark the time when the data was generated, which is the key basis for subsequent sliding time window filtering of data. The multi-dimensional indicator observation values cover the real-time values of various business-related indicators such as sensor data in industrial IoT and transaction indicators in financial risk control.
[0011] Step 104: Maintain a sliding time window with a preset fixed length, filter the multidimensional index observation data stream according to the sliding time window, and retain only the latest valid samples within the sliding time window to form a sample dataset.
[0012] The length of the sliding time window can be flexibly set according to business needs, such as setting it to the most recent 600 records or data within 10 minutes. Its purpose is to ensure that the analysis focuses on the current operating status of the system, eliminate expired and redundant data, reduce the amount of data for subsequent calculations, and improve processing efficiency.
[0013] Step 106: Update the mean, covariance, and cumulative second central moment of each indicator in the sample dataset within the sliding time window online to obtain the real-time mean, real-time covariance, and real-time cumulative second central moment of each indicator.
[0014] Online updates refer to updating statistics immediately upon receiving new observation data, rather than offline batch calculations. The mean reflects the average level of the indicators, the covariance reflects the degree of coordinated change between the two indicators, and the accumulated second central moments are an important intermediate variable for calculating variance. These statistics form the basis for subsequent calculations of correlation and uncertainty.
[0015] Step 108: Calculate the real-time standard deviation of each indicator based on the real-time mean, real-time covariance, and real-time cumulative second-order central moments of each indicator. Then, calculate the real-time correlation coefficient between any two indicators based on the real-time covariance and real-time standard deviation. Combine this with a preset correlation threshold to dynamically construct an indicator correlation network. At the same time, count the node connection degree of each indicator in the correlation network and use the node connection degree as the correlation score of the corresponding indicator.
[0016] The standard deviation reflects the dispersion of the indicators, while the correlation coefficient quantifies the degree of linear correlation between two indicators. The indicator association network uses nodes to represent indicators and edges to represent the relationship between indicators. The degree of node connection reflects the degree of correlation between a certain indicator and other indicators. The higher the degree, the stronger the correlation of the indicator in the indicator system, and the more likely it is to be a key indicator.
[0017] Step 110: Calculate the real-time coefficient of variation for each indicator based on the real-time mean and the real-time standard deviation of each indicator, and use the real-time coefficient of variation as the uncertainty score of the corresponding indicator.
[0018] The coefficient of variation is the ratio of the standard deviation to the mean. It eliminates the influence of dimensions and can compare the degree of fluctuation of different indicators on the same scale. The higher the coefficient of variation, the more drastic the fluctuation of the indicator in the current period, and the more information or risk it contains. As an uncertainty score, it can effectively reflect the dynamic change characteristics of the indicator.
[0019] Step 112: Normalize the correlation score and uncertainty score of each indicator respectively, and then merge the normalized correlation score and uncertainty score according to the preset weight to obtain the dynamic comprehensive score of each indicator.
[0020] Normalization maps scores with different dimensions and ranges to the same interval, preventing a single score from dominating the overall score due to an excessively large numerical range. Preset weights can be adjusted according to business needs, reflecting the different levels of importance of correlation and uncertainty in the selection of key indicators.
[0021] Step 114: Based on the dynamic comprehensive score of each indicator, key indicators are selected using preset screening rules to obtain a set of key indicators.
[0022] Real-time output can be achieved through APIs or message queues to ensure that the selected key indicators can be used by downstream systems or business modules in a timely manner to meet the needs of real-time decision-making.
[0023] The aforementioned method for selecting key indicators that supports real-time updates addresses the issues of offline operation and data redundancy in existing systems by combining a stream processing engine to receive timestamped multidimensional data streams with a sliding time window. The stream processing engine receives continuously flowing multidimensional indicator data in real time, ensuring timely data acquisition and avoiding the lag of traditional offline methods that wait for full data. The sliding time window, with its preset fixed length, retains only the latest valid samples within the window, eliminating outdated and redundant data while ensuring subsequent calculations focus on the current system state, avoiding resource waste caused by processing full stream data in existing real-time systems. Simultaneously, the timestamps in the data stream provide precise time information for window selection, ensuring that the selected sample dataset reflects the latest changes in indicators in real time, laying a data foundation for subsequent dynamic maintenance of correlation networks and uncertainty assessment. Furthermore, in the statistical calculation stage, the scheme employs incremental calculation methods such as the Welford online algorithm for core statistics such as mean, covariance, and the accumulated second-order central moments. Each new data point only requires recursive updates based on the previous statistical result and the current data, significantly reducing computational overhead and meeting low-latency requirements. The correlation coefficient is calculated based on real-time updated covariance and standard deviation. Network edges are dynamically added or removed only based on whether the correlation coefficient exceeds a preset threshold, without reconstructing the entire correlation network, further reducing computational load and enabling real-time dynamic maintenance of the correlation network. Uncertainty assessment is achieved through real-time calculation of the coefficient of variation, which is derived from incrementally updated mean and standard deviation, again without requiring full recalculation, ensuring real-time performance and low overhead in uncertainty assessment. The correlation score is calculated based on the node connection degree of the dynamic correlation network, reflecting changes in structural dependence between indicators in real time; the uncertainty score is calculated based on real-time coefficient of variation, quantifying the inherent volatility of indicators. After normalization, both are merged according to preset weights to form a dynamic comprehensive score, which takes into account both the structural correlation between indicators and the inherent uncertainty of indicators, avoiding the one-sidedness of single-dimensional alarms in existing systems. Finally, through threshold screening rules, a set of key indicators is output in real time, ensuring that the screening results can respond to dynamic changes in indicator relationships and fluctuations within seconds, fully adapting to the needs of real-time monitoring and decision-making in complex systems, and bridging the technical gap between traditional offline methods and existing real-time systems.
[0024] In one embodiment, the monitoring indicator dimension changes, and when a new indicator is detected, the statistical calculation container for the new indicator is automatically initialized; When an existing indicator is found to have no observations for an extended period, the indicator is marked as dormant and its statistical calculation is suspended.
[0025] In a specific embodiment, the statistical calculation container is a carrier used to store statistical data such as the mean and covariance of indicators. Initializing the container when adding new indicators can quickly integrate them into the screening process. Marking indicators that have no data for a long time as dormant can save computing resources and improve the overall operating efficiency of the system.
[0026] In one embodiment, the process of updating the mean online includes: The process of updating the mean online is as follows: ; in, express t -1 time index i The real-time average, express t Time indicators i The real-time average, express t Time indicators i The observed values, express t The number of valid samples within the sliding time window at any given moment.
[0027] In one embodiment, the process of updating the covariance between any two indicators online is as follows: ; in, express t -1 time index i With indicators j Real-time covariance, express t Time indicators i With indicators j covariance, express t Time indicators i The observed values, express t Time indicators j The observed values, express t Time indicators j The real-time average, express t -1 time index i The real-time average, express t The number of valid samples within the sliding time window at any given moment.
[0028] In one embodiment, the process of updating the accumulated second-order central moments of each index online is as follows: ; in, express t -1 time index i The cumulative amount of the second-order central moments, express t Time indicators i The real-time accumulation of the second-order central moments, express t Time indicators i The observed values, express t -1 time index i The real-time average, express t Time indicators i The real-time average.
[0029] In one embodiment, the process of calculating the real-time standard deviation of each indicator is as follows: ; in, express t Time indicators i The real-time accumulation of the second-order central moments, express t Time indicators i The real-time standard deviation express t The number of valid samples within the sliding time window at any given moment.
[0030] In one embodiment, the real-time correlation coefficient between any two indicators is calculated based on the real-time covariance and the real-time standard deviation, including: The real-time correlation coefficient between any two indicators is calculated based on the real-time covariance and real-time standard deviation: ; in, express t Time indicators i With indicators j covariance, express t Time indicators i The real-time standard deviation express t Time indicators j The real-time standard deviation.
[0031] In one embodiment, the real-time coefficient of variation for each indicator is calculated based on the real-time mean and the real-time standard deviation of each indicator, including: The real-time coefficient of variation for each indicator is calculated based on its real-time mean and real-time standard deviation: ; in, express t Time indicators i The real-time standard deviation express t Time indicators i The real-time average, This parameter is used to adjust the values to prevent numerical overflow caused by a zero denominator.
[0032] In one embodiment, the correlation scores and uncertainty scores of each indicator are normalized separately, and then the normalized correlation scores and uncertainty scores are fused together according to preset weights to obtain a dynamic comprehensive score for each indicator, including: The correlation scores and uncertainty scores of each indicator are normalized separately. Then, the normalized correlation scores and uncertainty scores are merged according to preset weights to obtain the dynamic comprehensive score for each indicator: ; in, for t Time indicators i Dynamic comprehensive score, To preset weights, for t Time indicators i The degree of node connectivity in an interconnected network. for t The maximum value of the connection degree of all indicator nodes at any given time. for t Time indicators i The real-time coefficient of variation, for t The minimum value of the real-time coefficient of variation of all indicators at any given time. for t The maximum value of the real-time coefficient of variation for all indicators at any given time.
[0033] In one embodiment, the indicator association network is an undirected graph, where nodes correspond to various indicators. The existence of an edge is determined by whether the real-time correlation coefficient between two corresponding indicators exceeds a preset association threshold. When the absolute value of the real-time correlation coefficient is not less than the preset association threshold, an edge is established between the two corresponding indicator nodes. When the absolute value of the real-time correlation coefficient is less than the preset association threshold, the edge between the two corresponding indicator nodes is deleted. The preset association threshold can be configured online according to different business scenarios. The preset filtering rule is to sort the indicators in descending order according to the dynamic comprehensive score and select the preset quantity indicators that are ranked first as key indicators.
[0034] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0035] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a key indicator screening method that supports real-time updates. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0036] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0037] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0038] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0039] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for filtering key indicators that supports real-time updates, characterized in that, The method includes: The stream processing engine receives multi-dimensional indicator observation data streams containing timestamps, with each observation data containing the current values of multiple indicators; A sliding time window of a preset fixed length is maintained. The multidimensional index observation data stream is filtered according to the sliding time window, and only the latest valid samples within the sliding time window are retained to form a sample dataset. For each indicator in the sample dataset within the sliding time window, the mean, covariance, and cumulative second central moments are updated online to obtain the real-time mean, real-time covariance, and real-time cumulative second central moments of each indicator. The real-time standard deviation of each indicator is calculated based on the real-time mean, real-time covariance, and real-time cumulative second-order central moments of each indicator. Then, the real-time correlation coefficient between any two indicators is calculated based on the real-time covariance and real-time standard deviation. An indicator association network is dynamically constructed in combination with a preset association threshold. At the same time, the node connection degree of each indicator in the association network is counted, and the node connection degree is used as the association degree score of the corresponding indicator. The real-time coefficient of variation for each indicator is calculated based on the real-time mean and the real-time standard deviation of each indicator, and the real-time coefficient of variation is used as the uncertainty score of the corresponding indicator. The correlation scores and uncertainty scores of each indicator are normalized separately, and then the normalized correlation scores and uncertainty scores are merged and calculated according to preset weights to obtain the dynamic comprehensive score of each indicator. Based on the dynamic comprehensive score of each indicator, key indicators are selected using preset screening rules to obtain a set of key indicators.
2. The method according to claim 1, characterized in that, The method further includes: Monitor changes in indicator dimensions; when a new indicator is detected, automatically initialize the statistical calculation container for the new indicator. When an existing indicator is found to have no observations for an extended period, the indicator is marked as dormant and its statistical calculation is suspended.
3. The method according to claim 1, characterized in that, The process of updating the mean online includes: The process of updating the mean online is as follows: in, express t -1 time index i The real-time average, express t Time indicators i The real-time average, express t Time indicators i The observed values, express t The number of valid samples within the sliding time window at any given moment.
4. The method according to claim 1, characterized in that, The process of updating the covariance between any two indicators online is as follows: in, express t -1 time index i With indicators j Real-time covariance, express t Time indicators i With indicators j covariance, express t Time indicators i The observed values, express t Time indicators j The observed values, express t Time indicators j The real-time average, express t -1 time index i The real-time average, express t The number of valid samples within the sliding time window at any given moment.
5. The method according to claim 1, characterized in that, The process of updating the cumulative second-order central moments of each indicator online is as follows: in, express t -1 time index i The cumulative amount of the second-order central moments, express t Time indicators i The real-time accumulation of the second-order central moments, express t Time indicators i The observed values, express t -1 time index i The real-time average, express t Time indicators i The real-time average.
6. The method according to claim 1, characterized in that, The process of calculating the real-time standard deviation of each indicator is as follows: in, express t Time indicators i The real-time accumulation of the second-order central moments, express t Time indicators i The real-time standard deviation express t The number of valid samples within the sliding time window at any given moment.
7. The method according to claim 1, characterized in that, Calculate the real-time correlation coefficient between any two indicators based on real-time covariance and real-time standard deviation, including: The real-time correlation coefficient between any two indicators is calculated based on the real-time covariance and real-time standard deviation: ； in, express t Time indicators i With indicators j covariance, express t Time indicators i The real-time standard deviation express t Time indicators j The real-time standard deviation.
8. The method according to claim 1, characterized in that, The real-time coefficient of variation for each indicator is calculated based on its real-time mean and real-time standard deviation, including: The real-time coefficient of variation for each indicator is calculated based on its real-time mean and real-time standard deviation: in, express t Time indicators i The real-time standard deviation express t Time indicators i The real-time average, This parameter is used to adjust the values to prevent numerical overflow caused by a zero denominator.
9. The method according to claim 1, characterized in that, The correlation scores and uncertainty scores of each indicator are normalized separately, and then the normalized correlation scores and uncertainty scores are fused together according to preset weights to obtain a dynamic comprehensive score for each indicator, including: The correlation scores and uncertainty scores of each indicator are normalized separately, and then the normalized correlation scores and uncertainty scores are fused together according to preset weights to obtain the dynamic comprehensive score of each indicator: in, for t Time indicators i Dynamic comprehensive score, To preset weights, for t Time indicators i The degree of node connectivity in an interconnected network. for t The maximum value of the connection degree of all indicator nodes at any given time. for t Time indicators i The real-time coefficient of variation, for t The minimum value of the real-time coefficient of variation of all indicators at any given time. for t The maximum value of the real-time coefficient of variation for all indicators at any given time.
10. The method according to claim 1, characterized in that, The indicator association network is an undirected graph, where nodes correspond to various indicators. The existence of an edge is determined by whether the real-time correlation coefficient between two corresponding indicators exceeds a preset association threshold. When the absolute value of the real-time correlation coefficient is not less than the preset association threshold, an edge is established between the two corresponding indicator nodes. When the absolute value of the real-time correlation coefficient is less than the preset association threshold, the edge between the two corresponding indicator nodes is deleted. The preset association threshold can be configured online according to different business scenarios. The preset filtering rule is to sort the indicators in descending order according to the dynamic comprehensive score and select the preset quantity indicators that rank highest as key indicators.