Dimensionality reduction compression method, device and storage medium for business index data
Patent Information
- Application Number
- CN202611315808.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-03
AI Technical Summary
In the distributed architecture of banks, existing technologies struggle to balance data traceability and query efficiency when processing massive amounts of business indicator data in monitoring systems, leading to the loss of dimensional information and the breakage of correlations.
By identifying the coupling degree between different dimension pairs under the same indicator name, and filtering out target dimension pairs based on the coupling degree threshold, dimensionality reduction splitting is performed to generate new derived indicator data, while preserving the dimension mapping relationship, thus achieving data dimensionality reduction compression.
It significantly reduced the amount of data, alleviated storage pressure and query burden, while retaining key dimensional information, ensuring data traceability and query flexibility.
Smart Images
Figure CN123034824A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of dimensionality reduction and compression technology for business indicator data, and more specifically, to a method, apparatus, storage medium, and electronic device for dimensionality reduction and compression of business indicator data. Background Technology
[0002] In a bank's distributed architecture, the monitoring system needs to collect massive amounts of business indicator monitoring data. This data is characterized by high concurrency, high cardinality, and time-series characteristics, posing challenges to the monitoring system's data processing and storage. The raw indicators acquired by the collection end are typically presented in the format of "indicator name + dimension key-value pair + value," and the same indicator can generate a massive number of data entries due to multiple dimension combinations. For some monitoring indicators and dimensions that are of particular interest to business personnel, excessive data can lead to extremely low query efficiency. To ensure data query efficiency, the massive amounts of collected data need to be processed to achieve a manageable volume.
[0003] Existing monitoring data processing technologies typically employ a method of directly removing some dimensions and then aggregating and calculating the remaining dimensions to cope with massive, high-cardinality indicator data. While this approach achieves data volume reduction and improved storage efficiency, it leads to the loss of dimensional information and the breakage of the correlation between indicators and dimensions. It fails to ensure data integrity and traceability, causing business personnel to face the problem of insufficient data when conducting full-dimensional queries or refined monitoring. Summary of the Invention
[0004] The main objective of this application is to provide a method, apparatus, storage medium, and electronic device for dimensionality reduction and compression of business indicator data, so as to at least solve the problem that related solutions have difficulty in ensuring data traceability after indicator data compression.
[0005] To achieve the above objectives, according to one aspect of this application, a method for dimensionality reduction and compression of business indicator data is provided, comprising: acquiring original indicator data, wherein the original indicator data includes indicator names, multiple dimension key-value pairs, and indicator values; determining the coupling degree between different dimension pairs under the same indicator name in the original indicator data, determining a target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than a coupling degree threshold; performing dimensionality reduction splitting processing on the dimension key-value pairs in the target original indicator data according to the target dimension, generating newly derived indicator data after splitting, and a dimensionality mapping relationship between the target original indicator data and the newly derived indicator data, wherein the number of dimension key-value pairs in the newly derived indicator data is less than the number of dimension key-value pairs in the target original indicator data.
[0006] Optionally, determining the coupling degree between different dimension pairs with the same indicator name in the original indicator data includes: identifying the same dimension key-value pairs in multiple original indicator data with the same indicator name as common dimension key-value pairs; identifying other dimensions in multiple original indicator data besides the common dimension key-value pairs as dimensions to be evaluated, and determining the coupling degree between each pair of dimensions to be evaluated.
[0007] Optionally, determining the coupling degree between each pair of the dimensions to be evaluated includes: determining the first indicator value corresponding to the first dimension to be evaluated and the second indicator value corresponding to the second dimension to be evaluated; determining the product of the first indicator value and the second indicator value as the total number of Cartesian product combinations of the first dimension to be evaluated and the second dimension to be evaluated; determining the number of pairs of the first dimension to be evaluated and the second dimension to be evaluated, and determining the ratio of the number of pairs to the total number of Cartesian product combinations as the coupling degree.
[0008] Optionally, determining the target dimension pair based on the coupling degree includes: determining two dimensions whose coupling degree is greater than or equal to a coupling degree threshold as the target dimension pair.
[0009] Optionally, after generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, the method further includes: encapsulating the target dimension pair and the dimensional mapping relationship into a dimensionality reduction splitting strategy.
[0010] Optionally, after generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the target original indicator data and the newly derived indicator data, the method further includes: storing the original indicator data, the newly derived indicator data, and the dimensional mapping relationship in a time series database, and setting the storage retention period corresponding to the original indicator data and the newly derived indicator data respectively.
[0011] Optionally, obtaining raw monitoring indicator data includes: obtaining the raw monitoring indicator data in real time through a distributed message queue.
[0012] According to another aspect of this application, a dimensionality reduction and compression device for business indicator data is provided, comprising: an acquisition unit for acquiring original indicator data, wherein the original indicator data includes indicator names, multiple dimension key-value pairs, and indicator values; a determination unit for determining the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determining a target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than a coupling degree threshold; and a processing unit for performing dimensionality reduction and splitting processing on the dimension key-value pairs in the target original indicator data according to the target dimension, generating newly derived indicator data after splitting, and a dimensionality mapping relationship between the target original indicator data and the newly derived indicator data, wherein the number of dimension key-value pairs in the newly derived indicator data is less than the number of dimension key-value pairs in the target original indicator data.
[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute any of the aforementioned dimensionality reduction and compression methods for business indicator data.
[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a dimensionality reduction compression method for performing any of the aforementioned business metric data.
[0015] By applying the technical solution of this application, the coupling degree between different dimension pairs under the same indicator name is determined, and target dimension pairs are selected based on the coupling degree threshold. This avoids the crude mode of blindly aggregating or simply discarding dimensions in traditional technologies. Traditional methods often lead to uncontrollable data volume due to the explosion of dimension combinations, or fail to meet the needs of refined queries due to the loss of dimensional information during aggregation. This application identifies highly coupled dimension pairs, performs dimensionality reduction splitting on the original data according to the target dimension pairs, and generates new derived indicators with fewer dimensions. This decouples strongly correlated dimensions and avoids the Cartesian product effect caused by cross-combination of dimensions. This splitting not only significantly reduces the number of data entries for a single indicator, alleviating the write pressure and storage cost of time-series databases, but also ensures that business personnel can trace the correlation logic of the original data through the new derived indicators by preserving the dimension mapping relationship. Ultimately, this solution significantly compresses the data volume while retaining the key dimensional information required for business monitoring, achieving a balance between storage efficiency and query flexibility, thereby solving the problem that related solutions have difficulty in ensuring data traceability after indicator data compression. Attached Figure Description
[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 A hardware structure block diagram of a mobile terminal for a method of dimensionality reduction and compression of execution business indicator data provided in an embodiment of this application is shown.
[0018] Figure 2 A flowchart illustrating a method for dimensionality reduction and compression of business indicator data according to an embodiment of this application is shown.
[0019] Figure 3 A flowchart illustrating the processing of a dimensionality reduction and compression system for business indicator data provided according to an embodiment of this application is shown.
[0020] Figure 4 A structural block diagram of a business indicator data dimensionality reduction and compression device provided according to an embodiment of this application is shown. Detailed Implementation
[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:
[0025] High-base indicators: These are monitoring indicators that contain a large number of different dimension combinations under the same indicator, resulting in an exponential growth of data entries.
[0026] Stream computing: Employs a real-time streaming processing framework to perform low-latency, high-throughput online computation and processing of monitoring data.
[0027] Time series database: Specifically designed to store time-stamped sequence data, optimizing the performance of writing and querying time-dimensional data.
[0028] As described in the background section, existing solutions struggle to maintain data traceability after compressing indicator data. To address this issue, embodiments of this application provide a method, apparatus, storage medium, and electronic device for dimensionality reduction and compression of business indicator data.
[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0030] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of dimensionality reduction and compression of business indicator data according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0031] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the dimensionality reduction and compression method for business indicator data in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0032] This embodiment provides a method for dimensionality reduction and compression of business indicator data running on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0033] Figure 2 This is a flowchart of a method for dimensionality reduction and compression of business indicator data according to an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:
[0034] Step S201: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0035] Step S202: Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0036] Step S203: Based on the target dimension, perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data to generate the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
[0037] This embodiment, by applying steps S201, S202, and S203, determines the coupling degree between different dimension pairs with the same indicator name and filters out target dimension pairs based on a coupling degree threshold, avoiding the crude approach of blindly aggregating or simply discarding dimensions in traditional technologies. Traditional methods often lead to uncontrollable data volume due to dimension combination explosion or fail to meet refined query needs due to the loss of dimension information during aggregation. This application identifies highly coupled dimension pairs, performs dimensionality reduction splitting on the original data according to the target dimension pairs, generates new derived indicators with fewer dimensions, decouples strongly correlated dimensions, and avoids the Cartesian product effect caused by cross-combination of dimensions. This splitting not only significantly reduces the number of data entries for a single indicator, alleviating the write pressure and storage cost of the time-series database, but also ensures that business personnel can trace the correlation logic of the original data through the new derived indicators by preserving the dimension mapping relationship. Ultimately, this solution significantly compresses the data volume while retaining the key dimension information required for business monitoring, achieving a balance between storage efficiency and query flexibility, thereby solving the problem that related solutions struggle to maintain data traceability after indicator data compression.
[0038] In the specific implementation process, the coupling degree between different dimension pairs with the same indicator name in the original indicator data is determined, including: determining the same dimension key-value pairs in multiple original indicator data with the same indicator name as common dimension key-value pairs; determining other dimensions in multiple original indicator data other than the common dimension key-value pairs as dimensions to be evaluated, and determining the coupling degree between each pair of the above-mentioned dimensions to be evaluated.
[0039] In this embodiment, key-value pairs with the same metric name and the same dimension are identified as common dimension key-value pairs. By extracting the common dimension, other dimensions besides the common dimension are identified as dimensions to be evaluated, and the coupling degree between each pair is calculated. This pairwise comparison strategy avoids the exponential computational complexity of combining all dimensions. If the coupling degree is calculated using all permutations, the computational overhead is enormous when the number of dimensions is large, making it difficult to meet the requirements of real-time stream processing. This solution uses a pairwise evaluation method, reducing the computational complexity from factorial to polynomial, greatly improving the system's response speed and real-time processing capability.
[0040] Specifically, determining the coupling degree between each pair of the aforementioned dimensions to be evaluated includes: determining the first indicator value corresponding to the first dimension to be evaluated and the second indicator value corresponding to the second dimension to be evaluated; determining the product of the first indicator value and the second indicator value as the total number of Cartesian product combinations of the first dimension to be evaluated and the second dimension to be evaluated; determining the number of pairs of the first dimension to be evaluated and the second dimension to be evaluated, and determining the ratio of the number of pairs to the total number of Cartesian product combinations as the coupling degree.
[0041] In this embodiment, by determining the index values of the first and second dimensions to be evaluated respectively and calculating their product as the "total number of Cartesian product combinations," a theoretical upper limit benchmark for dimensional association is mathematically constructed. This benchmark represents the maximum data combination scale that two dimensions can theoretically generate under the current common dimension conditions. Secondly, the coupling degree is determined by statistically comparing the number of actually observed pairings with the theoretical total number of combinations. This ratiometric index eliminates the influence of the dimensional cardinality on the results. This calculation scheme ensures that the splitting operation is triggered only when there are substantial Cartesian product features between dimensions, avoiding increased system complexity due to erroneous splitting, and also ensuring that the data volume can be compressed to the maximum extent when splitting is necessary, achieving optimal dimensionality reduction.
[0042] More specifically, determining the target dimension pair based on the aforementioned coupling degree includes: determining two dimensions whose coupling degree is greater than or equal to the coupling degree threshold as the aforementioned target dimension pair.
[0043] In this embodiment, by defining target dimension pairs as those with a coupling degree greater than or equal to a coupling degree threshold, this threshold mechanism allows system administrators to flexibly adjust based on specific business needs, hardware resource conditions, and data quality. For example, when storage resources are extremely strained, the threshold can be lowered to trigger more splits; conversely, when prioritizing data integrity, the threshold can be raised to reduce splits. This flexibility enables the invention to adapt to the diverse needs of distributed architectures in banks of different sizes. Secondly, only dimension pairs that meet the high coupling condition are selected for splitting. This logic effectively filters out dimension combinations with low coupling and minimal splitting benefits. Splitting all dimension pairs would not only generate a massive number of new metrics, increasing the burden of data management, but could also disrupt existing low-coupling but necessary relationships, leading to more complex query logic.
[0044] Furthermore, after generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, the method further includes: encapsulating the target dimension pairs and the dimensional mapping relationship into a dimensionality reduction splitting strategy.
[0045] In this embodiment, the target dimension pairs and dimension mapping relationships are encapsulated into a dimensionality reduction splitting strategy. This encapsulation decouples the dimensionality reduction logic from the data processing flow, allowing the system to adapt to different metrics or business scenarios by loading different strategy configurations without modifying the core code. Traditional solutions often lose the original correlation between dimensions after aggregation, while this solution preserves the ability to trace back to the original metric data from newly derived metrics through mapping relationships.
[0046] Furthermore, after generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, the method further includes: storing the original indicator data, the newly derived indicator data, and the dimensional mapping relationship in a time series database, and setting the storage retention period for the original indicator data and the newly derived indicator data respectively.
[0047] In this embodiment, the original indicator data, newly derived indicator data, and dimensional mapping relationships are stored separately in a time-series database, with different retention periods set, reflecting the principles of separating hot and cold data and hierarchical value storage. Original indicator data, due to its numerous dimensional combinations, large data volume, and rapid decay of its fine-grained instantaneous value over time, typically only needs to be retained for a short period (e.g., 7 days) for troubleshooting. Newly derived indicator data, after dimensionality reduction, retains core business dimensions, significantly reducing data volume, and its aggregated trend information is of significant value for long-term business analysis, capacity planning, and trend prediction. Therefore, by setting longer retention periods (e.g., 1 year or longer), the system can retain high-value monitoring data for extended periods with limited storage resources. This hierarchical storage mechanism not only solves the storage pressure of massive monitoring data but also improves the long-term availability of data.
[0048] Specifically, obtaining raw monitoring metric data includes: obtaining the aforementioned raw monitoring metric data in real time through a distributed message queue.
[0049] In this embodiment, the method for acquiring raw monitoring indicator data is defined. Raw monitoring indicator data is acquired in real time through a distributed message queue (such as Kafka, Pulsar, etc.), leveraging the high concurrency processing capabilities and peak-shaving / valley-filling characteristics of message queues. In the distributed architecture of large banks, the monitoring system faces the pressure of millions of data inflows per second. The distributed message queue, acting as a buffer layer, can absorb sudden traffic surges and ensure the stability of the data acquisition end. Furthermore, distributed message queues typically have data persistence and retry mechanisms, ensuring that data is not lost even if backend processing nodes are temporarily unavailable, thus guaranteeing data integrity and consistency. This allows the proposed solution to handle the flood of monitoring data during peak periods, maintaining the stable operation and efficient processing capabilities of the entire monitoring system.
[0050] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the dimensionality reduction and compression method for business indicator data of this application will be described in detail below with reference to specific embodiments.
[0051] In a distributed architecture for banks, monitoring systems need to collect massive amounts of business metric monitoring data. This data is characterized by high concurrency, high cardinality, and time-series characteristics, posing challenges to the monitoring system's data processing, storage, and query response capabilities. The raw metrics acquired by the collection end are typically presented in the form of "metric name + dimension key-value pair + value". The same metric can generate a massive number of data entries due to multiple dimension combinations, resulting in the high cardinality problem of the metrics.
[0052] Taking the original indicators as an example:
[0053] AAA{k1=v11,k2=v21,k3=v31} 2;
[0054] AAA{k1=v11,k2=v21,k3=v32} 3;
[0055] AAA{k1=v11,k2=v22,k3=v31} 4;
[0056] AAA{k1=v11,k2=v22,k3=v32} 8.
[0057] The AAA metric carries three dimensions: k1, k2, and k3. k2 has two possible values, and k3 has two possible values. These dimensions are cross-combined to generate four independent original metric records. As business scales up, the number of dimensions and the number of values per dimension increase exponentially, leading to a surge in monitoring data entries. This results in a series of problems, including high storage resource consumption, heavy write pressure on the time-series database, slow metric queries, and high system resource overhead.
[0058] Traditional data processing methods involve data aggregation and calculation, which aggregates specific dimensions and performs calculations. The core calculations include: acquiring raw monitoring indicator data from the data collection end; directly removing some dimensions and performing aggregation calculations on the remaining dimensions; storing the aggregated data and discarding the original collected data.
[0059] Traditional aggregation results:
[0060] AAA1{k1=v11} 17;
[0061] AAA2{k2=v21} 5;
[0062] AAA2{k2=22} 12;
[0063] AAA3{k3=v31} 6;
[0064] AAA3{k3=v32} 12.
[0065] The aggregation results of this application:
[0066] AAA_1{k1=v11,k2=v21} 5;
[0067] AAA_1{k1=v11,k2=v22} 12;
[0068] AAA_2{k1=v11,k3=v31} 6;
[0069] AAA_2{k1=v11,k3=v32} 11.
[0070] The specific implementation scheme of this application is as follows:
[0071] The original indicators collected, under the condition that the values of the same indicator are fixed in the common dimension, show that almost all the values of the X dimension and the values of the Y dimension are paired up, forming a large number of cross-combination records of dimensions.
[0072] Let the metric be M, and select two dimensions X and Y to be evaluated. Choose a fixed set of common dimensions S (in this example, S={k1}, and k1=v11):
[0073] Given a fixed value for the common dimension S, we calculate the number of unique values Xset for dimension X, which is Nx; and the number of unique values Yset for dimension Y, which is Ny. Theoretically, the total number of Cartesian product combinations is Nx. Ny.
[0074] RealCount is the actual number of (X value, Y value) pairs that appear in the valid records that simultaneously satisfy the condition of a fixed common dimension S and carry both X and Y dimensions in the original statistical indicators.
[0075] Configure the system with a configurable coupling threshold, Rate (configurable threshold, e.g., 0.6-1, adjustable according to business needs). If RealCount÷(Nx×Ny)≥Rate, X and Y are determined to be a highly coupled dimension pair, exhibiting a Cartesian product-like phenomenon, triggering the metric splitting and dimensionality reduction process; if the ratio is below the threshold, the two dimensions are considered to have low coupling, and splitting is not performed.
[0076] By splitting the indicators, the two sets of indicators AAA_1 and AAA_2 can be separated into two dimensions, k2 and k3, so that the two dimensions k2 and k3 will not appear in the same indicator at the same time, thereby reducing the amount of data in the indicator.
[0077] This embodiment relates to a specific business indicator data dimensionality reduction and compression system. The system consists of: a system configuration device, a data acquisition device, a configuration information acquisition device, a data dimensionality reduction device, a data output device, and a data storage device.
[0078] Data acquisition device: Acquires raw monitoring indicator data in real time through a distributed message queue, supporting a throughput of millions per second.
[0079] System configuration device: Stores global preset parameters of the system, including coupling degree threshold, common dimension set, derived indicator naming template, and dimensionality reduction splitting strategy template; The system configuration device reads historical sample indicator data within a specified time window from the data cache, completes the calculation and identification of dimensional coupling relationships, outputs dimensional coupling analysis results, dimension pairs to be split, derived indicator information, and aggregation calculation expressions, and generates preset dimensionality reduction strategies on the machine side; At the same time, it retains the system administrator interaction interface, which allows the system administrator to review, modify and supplement the preset dimensionality reduction strategies output by the machine. System administrator configuration is an optional auxiliary means and not a necessary prerequisite for system execution.
[0080] Configuration information acquisition device: acquires configuration information, and will be called again in case of abnormality; if the configuration information is acquired normally, it will generate dimensionality reduction rules for a certain indicator, two dimensions or multiple dimensions of configuration information.
[0081] Data dimensionality reduction device: After obtaining the dimensionality reduction rules and original indicator data, the data dimensionality reduction device performs dimensionality splitting to generate new independent indicators (such as AAA_1, AAA_2), performs aggregation calculations on the split new indicators, and generates compressed data.
[0082] Data output device: The dimensionality-reduced data is written to the data output device, and the data output device writes it to the data storage.
[0083] Data storage device: A time-series database is used, combined with VictoriaMetrics to enable writing and improve query efficiency.
[0084] Unified API: Provides standardized API interfaces for business systems to call the reduced dimensionality indicator data.
[0085] Figure 3 The processing flowchart of the dimensionality reduction and compression system for business indicator data is as follows: Figure 3 As shown, it specifically includes the following:
[0086] The collected monitoring data is written to the data cache, and the data acquisition device consumes the data in the data cache.
[0087] The system retrieves the metric name configured by the administrator. The system configuration device then retrieves data from the data cache for a specific period and performs calculations to generate the coupling rate for two or more dimensions under the common dimension of this metric. If the coupling rate exceeds the threshold, the system configuration device automatically adds a new metric name and downsampling formula. Alternatively, the system administrator can confirm the addition of new metric names and downsampling formulas based on the displayed dimension splitting information.
[0088] The configuration information acquisition device acquires the reduced configuration information and verifies the information. If the information is correct, the configuration data in the acquisition device is synchronized to the data reduction device. If the information is incorrect, the configuration information acquisition device is called again for retry, and corresponding self-monitoring indicator data is generated for subsequent processing by the system self-monitoring module.
[0089] After combining the original monitoring indicators and the correct downsampling strategy information of a certain indicator, the data dimensionality reduction device executes the indicator dimension splitting logic to generate multiple independent derived indicators. It then performs grouping and aggregation calculations on the split derived indicators and outputs compressed time-series data of the derived indicators.
[0090] Newly generated indicators and original indicators are written from the data output device to the data storage device. The retention period for original indicators and newly generated indicators are set to different retention periods according to actual needs. The retention period for original indicators is shorter, and the retention period for newly generated indicators is longer.
[0091] The unified API provides a unified interface to the outside world. External systems can directly request the unified API, which then requests the backend storage, encapsulates the queried data in a unified manner, and returns it to the external system in a standard format.
[0092] This embodiment introduces stream computing, supporting PromQL syntax, time windows, and some functions. Combined with the distributed architecture of a large commercial bank, this embodiment enables the processing of massive amounts of collected monitoring data. It can perform dimensionality reduction calculations on millions of monitoring data points per second in real time and efficiently, improving the correlation between monitoring indicator dimensions, meeting the monitoring needs of business systems, and simultaneously alleviating the storage pressure on monitoring data.
[0093] This application also provides a dimensionality reduction and compression device for business indicator data. It should be noted that this device can be used to execute the dimensionality reduction and compression method for business indicator data provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0094] The following describes the dimensionality reduction and compression device for business indicator data provided in the embodiments of this application.
[0095] Figure 4 This is a schematic diagram of a dimensionality reduction and compression device for business indicator data according to an embodiment of this application. For example... Figure 4 As shown, the device includes:
[0096] The acquisition unit 41 is used to acquire raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0097] The determining unit 42 is used to determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and to determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0098] Processing unit 43 is used to perform dimensionality reduction splitting on the dimension key-value pairs in the target original indicator data according to the target dimension, and generate the split new derived indicator data and the dimension mapping relationship between the target original indicator data and the new derived indicator data, wherein the number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the target original indicator data.
[0099] In this embodiment, by determining the coupling degree between different dimension pairs under the same indicator name and filtering out target dimension pairs based on the coupling degree threshold, the crude mode of blindly aggregating or simply discarding dimensions in traditional technologies is avoided. Traditional methods often lead to uncontrollable data volume due to the explosion of dimension combinations, or fail to meet the needs of refined queries due to the loss of dimension information due to aggregation. This application identifies highly coupled dimension pairs, performs dimensionality reduction splitting on the original data according to the target dimension pairs, and generates new derived indicators with fewer dimensions, decoupling strongly related dimensions and avoiding the Cartesian product effect caused by cross-combination of dimensions. This splitting not only significantly reduces the number of data entries for a single indicator, alleviating the write pressure and storage cost of the time-series database, but also ensures that business personnel can trace the correlation logic of the original data through the new derived indicators by preserving the dimension mapping relationship. Ultimately, this solution significantly compresses the data volume while retaining the key dimension information required for business monitoring, achieving a balance between storage efficiency and query flexibility, thereby solving the problem that related solutions have difficulty in ensuring data traceability after indicator data compression.
[0100] As an optional approach, the determining unit includes a first determining module and a second determining module; the first determining module is used to determine the same dimension key-value pairs in multiple original indicator data with the same indicator name as common dimension key-value pairs; the second determining module is used to determine other dimensions in multiple original indicator data besides the common dimension key-value pairs as dimensions to be evaluated, and to determine the coupling degree between each pair of the aforementioned dimensions to be evaluated.
[0101] In one optional scheme, the second determining module includes a first determining submodule, a second determining submodule, and a third determining submodule; the first determining submodule is used to determine the first indicator value corresponding to the first dimension to be evaluated and the second indicator value corresponding to the second dimension to be evaluated, respectively; the second determining submodule is used to determine the product of the first indicator value and the second indicator value as the total number of Cartesian product combinations of the first dimension to be evaluated and the second dimension to be evaluated; the third determining submodule is used to determine the number of pairs of the first dimension to be evaluated and the second dimension to be evaluated, and the ratio of the number of pairs to the total number of Cartesian product combinations is determined as the coupling degree.
[0102] In an optional scheme, the determining unit further includes a third determining module, used to determine the two dimensions with a coupling degree greater than or equal to the coupling degree threshold as the target dimension pair.
[0103] In one alternative, the apparatus further includes an encapsulation unit for encapsulating the target dimension pairs and the dimension mapping relationship into a dimensionality reduction splitting strategy after generating the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data.
[0104] In one optional embodiment, the apparatus further includes a setting unit, which, after generating the newly derived indicator data after splitting and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, stores the original indicator data, the newly derived indicator data, and the dimensional mapping relationship in a time-series database, and sets the storage retention period corresponding to the original indicator data and the newly derived indicator data respectively.
[0105] In one alternative approach, the acquisition unit includes an acquisition module, which is used to acquire the aforementioned raw monitoring indicator data in real time through a distributed message queue.
[0106] The aforementioned data compression and dimensionality reduction device includes a processor and a memory. The acquisition unit, determination unit, and processing unit are all stored as program units in the memory, and the processor executes these program units to achieve the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.
[0107] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the issue of data traceability after data compression in certain solutions.
[0108] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0109] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute a method for dimensionality reduction and compression of the business indicator data.
[0110] Specifically, methods for dimensionality reduction and compression of business indicator data include:
[0111] Step S201: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0112] Step S202: Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0113] Step S203: Based on the target dimension, perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data to generate the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
[0114] This invention provides a processor for running a program, wherein the program executes a dimensionality reduction and compression method for the aforementioned business indicator data.
[0115] Specifically, methods for dimensionality reduction and compression of business indicator data include:
[0116] Step S201: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0117] Step S202: Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0118] Step S203: Based on the target dimension, perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data to generate the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
[0119] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:
[0120] Step S201: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0121] Step S202: Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0122] Step S203: Based on the target dimension, perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data to generate the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
[0123] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.
[0124] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:
[0125] Step S201: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value;
[0126] Step S202: Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold.
[0127] Step S203: Based on the target dimension, perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data to generate the split new derived indicator data and the dimensional mapping relationship between the original target indicator data and the new derived indicator data. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
[0128] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.
[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0134] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0135] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0137] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0138] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for dimensionality reduction and compression of business indicator data, characterized in that, include: Obtain raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value; Determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold; The dimension key-value pairs in the original target indicator data are split into new derived indicator data based on the target dimension, and the dimension mapping relationship between the original target indicator data and the new derived indicator data is generated. The number of dimension key-value pairs in the new derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
2. The method according to claim 1, characterized in that, Determining the coupling degree between different dimension pairs with the same indicator name in the original indicator data includes: Identify the same dimension key-value pairs in the original indicator data of multiple indicators with the same indicator name as common dimension key-value pairs; Other dimensions in the original indicator data, excluding the common dimension key-value pairs, are identified as dimensions to be evaluated, and the coupling degree between each pair of the dimensions to be evaluated is determined.
3. The method according to claim 2, characterized in that, Determine the pairwise coupling between each of the dimensions to be evaluated, including: Determine the value of the first indicator corresponding to the first dimension to be evaluated, and the value of the second indicator corresponding to the second dimension to be evaluated, respectively; The product of the first indicator value and the second indicator value is determined as the total number of Cartesian product combinations of the first and second dimensions to be evaluated. The number of pairings between the first dimension to be evaluated and the second dimension to be evaluated is determined, and the ratio of the number of pairings to the total number of combinations of the Cartesian product is determined as the coupling degree.
4. The method according to claim 1, characterized in that, Determining the target dimension pair based on the coupling degree includes: The two dimensions whose coupling degree is greater than or equal to the coupling degree threshold are determined as the target dimension pair.
5. The method according to claim 1, characterized in that, After generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, the method further includes: The target dimension pairs and the dimension mapping relationship are encapsulated into a dimensionality reduction splitting strategy.
6. The method according to claim 1, characterized in that, After generating the newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, the method further includes: The original indicator data, the newly derived indicator data, and the dimension mapping relationship are stored in a time series database, and the storage retention period corresponding to the original indicator data and the newly derived indicator data is set respectively.
7. The method according to claim 1, characterized in that, Obtain raw monitoring metric data, including: The raw monitoring metric data is obtained in real time through a distributed message queue.
8. A dimensionality reduction and compression device for business indicator data, characterized in that, include: The acquisition unit is used to acquire raw indicator data, wherein the raw indicator data includes indicator name, multiple dimension key-value pairs, and indicator value; A determining unit is used to determine the coupling degree between different dimension pairs with the same indicator name in the original indicator data, and to determine the target dimension pair based on the coupling degree, wherein the coupling degree of the target dimension pair is greater than the coupling degree threshold; The processing unit is configured to perform dimensionality reduction splitting on the dimension key-value pairs in the original target indicator data according to the target dimension, generate newly derived indicator data after splitting, and the dimensional mapping relationship between the original target indicator data and the newly derived indicator data, wherein the number of dimension key-value pairs in the newly derived indicator data is less than the number of dimension key-value pairs in the original target indicator data.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the dimensionality reduction and compression method for business indicator data as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a dimensionality reduction compression method for performing business indicator data as described in any one of claims 1 to 7.