High-performance index calculation method supporting automatic aggregation of multi-dimensional data

Abstract

The invention discloses a high-performance index calculation method supporting multi-dimensional data automatic aggregation, and relates to the technical field of data processing, and the method comprises the steps: recognizing an abnormal dimension combination of which the page view significantly exceeds a historical threshold value, and obtaining a resource occupation state of the abnormal dimension combination; determining an aggregation load gain of the abnormal dimension combination according to an enhancement effect of resource competition between dimensions in the abnormal dimension combination, and calculating resource release elasticity of the non-abnormal dimension combination in response to resource occupation; calculating aggregation path sensitivity based on a dynamic interaction relationship between aggregation load gain and resource release elasticity, and actively fusing an aggregation path corresponding to the abnormal dimension combination according to the aggregation path sensitivity; updating the aggregation path sensitivity according to the resource release state fed back by the unabnormal dimension combination in real time so as to automatically recover the fused aggregation path; according to the method, system resource overload caused by abnormal access is effectively prevented through dynamic calculation and active fusing of aggregation path sensitivity.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a high-performance index calculation method that supports automatic aggregation of multidimensional data. Background Technology

[0002] With the development of big data technology, enterprises have widely adopted multidimensional data real-time streaming computing platforms for business analysis, indicator calculation, and real-time monitoring. These platforms typically require real-time aggregation and calculation of high-dimensional data combinations to meet diverse needs in business scenarios, such as rapid decision-making, indicator monitoring, and anomaly identification.

[0003] In practical applications, real-time aggregation tasks often involve accessing and calling resources for a large number of dimension combinations. Especially when business traffic fluctuates significantly or the size of dimension combinations is large, the real-time access volume of one or more dimension combinations may be significantly higher than the historical normal level, leading to a sharp increase in the resource consumption of aggregation computing tasks. This sudden increase in resource consumption not only affects the processing efficiency of aggregation tasks for the corresponding abnormal dimension combinations, but also makes it easier for resource contention issues to cause a decline in overall system performance or even the risk of system crashes.

[0004] Current methods for handling anomalies in real-time aggregate computing load are mostly reactive, meaning manual intervention is only implemented when performance issues become significant or the system is on the verge of failure. This approach not only lacks real-time performance but also easily delays intervention opportunities, making it difficult to achieve rapid response and proactive intervention to resource usage anomalies. It fails to meet the growing demands of enterprises for real-time performance, high reliability, and automated operation and maintenance.

[0005] Furthermore, existing methods for identifying abnormal resource usage often rely on simple threshold judgments or single-dimensional resource monitoring, failing to comprehensively consider the enhanced resource competition among members of different dimensions within a multi-dimensional combination, as well as the dynamic load impact of abnormal dimension combinations on system resources. This makes it difficult for existing technologies to accurately and dynamically assess the real-time impact of abnormal combination resource usage on overall platform performance.

[0006] On the other hand, although the circuit breaker operation of the aggregated path caused by abnormal dimension combination resource occupation can quickly reduce the load, in the existing technology, most aggregated paths lack a clear automatic recovery mechanism after circuit breaker is broken, and manual intervention is required for recovery, resulting in low recovery efficiency of the circuit breaker path and affecting business continuity. Summary of the Invention

[0007] The purpose of this invention is to provide a high-performance index calculation method that supports automatic aggregation of multidimensional data, so as to solve the problems in the background art mentioned above.

[0008] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a high-performance index calculation method that supports automatic aggregation of multidimensional data, comprising: S101: Determine the current access volume of each dimension combination based on the real-time multidimensional data stream, identify abnormal dimension combinations whose access volume significantly exceeds the historical threshold, and obtain the resource usage status of abnormal dimension combinations. S102: Based on the enhanced effect of resource competition among dimensions in the abnormal dimension combination, determine the aggregate load gain of the abnormal dimension combination and calculate the resource release elasticity of the response resource occupation of the non-abnormal dimension combination. S103: Calculate the aggregation path sensitivity based on the dynamic interaction between aggregation load gain and resource release elasticity, and proactively circuit break the corresponding aggregation path based on the abnormal dimension combination. S104: Update the aggregation path sensitivity based on the real-time feedback of resource release status of non-abnormal dimension combinations to automatically restore the aggregated path that has been broken.

[0009] Furthermore, the process of identifying abnormal dimension combinations whose access volume significantly exceeds historical thresholds and obtaining the resource usage status of these abnormal dimension combinations includes: Extract the access volume data of each dimension combination in the most recent T historical periods to obtain the historical access baseline of each dimension combination; Calculate the ratio between the access volume of each dimension combination in the current period and the corresponding historical access baseline, and determine the dimension combination whose access volume ratio is higher than the first preset ratio threshold as the initial abnormal candidate combination; The resource consumption level of the initial anomaly candidate combination is calculated based on the aggregated calculation of the call count among the dimension members of the initial anomaly candidate combination. Based on the resource occupancy level being higher than the second preset occupancy threshold, a final abnormal dimension combination is determined from the initial abnormal candidate combination, and the resource occupancy status is determined by the resource occupancy level of the final abnormal dimension combination.

[0010] Further, the calculation of the resource consumption level of the initial abnormal candidate combination includes: Obtain the amount of computing resources used by each dimension member in each initial anomaly candidate combination when participating in the aggregation calculation; Based on the aggregation frequency of the initial abnormal candidate combination in the most recent N aggregation calculations, determine the resource consumption growth rate when each dimension member is aggregated. Based on the superposition effect between the resource consumption growth rates when members of each dimension are aggregated, the cumulative value of resource consumption for the initial abnormal candidate combination is calculated. The resource usage level of the initial abnormal candidate combination is determined based on the cumulative value of resource usage.

[0011] Furthermore, the calculation of the resource release elasticity in response to the resource occupation of abnormal dimension combinations in response to non-abnormal dimension combinations includes: Extract the real-time resource usage of non-abnormal dimension combinations when the resource load gain effect of abnormal dimension combinations reaches a preset load threshold. Calculate the change in current resource usage of non-abnormal dimension combinations relative to historical normal resource usage benchmarks to determine the initial value of resource release elasticity; The initial value of resource release elasticity is adjusted based on the degree of fluctuation in the resource release process of non-abnormal dimension combinations during the period of change in the resource load gain effect of abnormal dimension combinations. Based on the adjusted initial value of resource release elasticity and the resource load gain effect of abnormal dimension combinations, the final resource release elasticity of non-abnormal dimension combinations is determined.

[0012] Furthermore, adjusting the initial value of resource release elasticity based on the fluctuation of the resource release process during the period of change in the resource load gain effect of abnormal dimension combinations includes: Determine the changing trend data of the resource load gain effect of abnormal dimension combination at multiple consecutive time points, and calculate the difference in the rate of load change; Based on the difference in load change rate, extract the release speed data of the non-abnormal dimension combination resource release process at the corresponding time. Adjust the initial value of resource release elasticity based on the fluctuations in release rate data.

[0013] Furthermore, the aggregation path corresponding to the active circuit breaker anomaly dimension combination includes: The load anomaly level of the aggregation path is determined based on the real-time sensitivity of the aggregation path, and the initial circuit breaker strategy of the aggregation path with the combination of anomaly dimensions is determined based on the load anomaly level. Extract the associated dimension dependency data of abnormal dimension combinations based on the initial circuit breaker strategy of the aggregation path, and calculate the resource call redundancy between dimensions in the associated dimension dependency data; The initial circuit breaker strategy for the aggregated path of the abnormal dimension combination is optimized based on the resource call redundancy, and the final circuit breaker strategy for the aggregated path of the abnormal dimension combination is obtained. Based on the final circuit breaker strategy of the aggregation path, the aggregation path corresponding to the combination of abnormal dimensions is actively circuit breaker.

[0014] Furthermore, the initial circuit breaker strategy for the aggregated path, which optimizes the combination of abnormal dimensions based on resource call redundancy, includes: Based on the resource call data between dimensions, determine the degree of overlap in resource calls within abnormal dimension combinations; Extract the target dimension combination with the highest degree of resource call overlap, and analyze the degree of matching between the dimension call order within the target dimension combination and the dimension call order within the abnormal dimension combination; Based on the degree of matching of the dimension call order, the initial circuit breaker strategy of the aggregation path is adjusted, and the final circuit breaker strategy of the aggregation path is optimized.

[0015] Furthermore, the step of updating the aggregated path sensitivity based on the resource release status feedback in real time according to the non-abnormal dimension combination includes: Real-time extraction of resource release status data for non-abnormal dimension combinations during circuit breaker periods for abnormal dimension combinations, and determination of the changing trend of resource release status data; Based on the changing trend of resource release status data, calculate the resource release rate of the non-abnormal dimension combination before the aggregation path is restored; Based on the degree of coupling between the resource release rate and the resource release elasticity of non-abnormal dimension combinations, the real-time sensitivity of the aggregation path is modified.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention automatically identifies abnormal dimension combinations with significantly higher access volumes than historical baselines by real-time monitoring of access volume and resource consumption of various dimension combinations in a multidimensional data stream. Then, by quantifying the dynamic relationship between the enhanced resource competition effect within the dimension combination and the resource release elasticity of non-abnormal dimension combinations, it accurately calculates the real-time sensitivity of the aggregation path and executes an active circuit breaker strategy. This effectively avoids system performance bottlenecks and overall service interruption risks caused by abnormal resource consumption of dimension combinations, significantly improving the stability and reliability of enterprise real-time stream computing platforms.

[0017] This invention further optimizes the scope and intensity of circuit breaking paths by introducing analysis of resource call redundancy and call order matching degree between dimensions in the process of decision-making for circuit breaking of aggregated paths. This makes the circuit breaking strategy more accurate and efficient, avoids unnecessary business performance loss caused by excessive circuit breaking in traditional circuit breaking schemes, realizes refined resource management in abnormal resource usage scenarios, and effectively improves the platform's resource utilization efficiency.

[0018] This invention establishes a dynamic feedback adjustment mechanism between resource release status and aggregation path sensitivity, enabling automatic recovery of broken aggregation paths. This ensures rapid automatic recovery of aggregation tasks and business continuity after anomalies are resolved, effectively reducing manual intervention costs and improving system response speed. It comprehensively enhances the automated operation and maintenance level and business assurance capabilities of enterprise multidimensional real-time stream computing platforms. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of a high-performance index calculation method that supports automatic aggregation of multidimensional data according to the present invention. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more complete and comprehensive, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative illustrations of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0022] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of the exemplary embodiments disclosed in this application. However, those skilled in the art will recognize that the technical solutions disclosed in this application can be practiced with one or more specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the disclosure of this application.

[0023] Example 1

[0024] like Figure 1 As shown in the figure, this embodiment discloses a high-performance index calculation method that supports automatic aggregation of multi-dimensional data, including: S101: Determine the current access volume of each dimension combination based on the real-time multidimensional data stream, identify abnormal dimension combinations whose access volume significantly exceeds the historical threshold, and obtain the resource usage status of abnormal dimension combinations. This embodiment provides an implementation method based on an enterprise's existing multidimensional data real-time stream computing platform. The real-time multidimensional data stream is collected by the data stream processing platform. The real-time data stream includes real-time access records of dimension combinations. Each access record includes, but is not limited to, access timestamps, user identifiers, IP addresses, device identifiers, and product SKUs. The access records are statistically analyzed in real time by the platform's built-in real-time aggregation module to obtain the current access volume of the dimension combinations.

[0025] In specific implementation, identifying abnormal dimension combinations whose access volume significantly exceeds historical thresholds and obtaining the resource usage status of these abnormal dimension combinations includes: Extract the access volume data of each dimension combination in the most recent T historical periods to obtain the historical access baseline of each dimension combination; In practice, the enterprise real-time aggregation platform predefines a historical period length, with the unit being minutes. The specific value is determined by the enterprise's actual business needs. The historical access baseline is the average access volume of each dimension combination over multiple consecutive historical periods, calculated using the following formula:

[0026] In the formula: This represents the historical access baseline for the i-th dimension combination. This represents the number of visits to the i-th dimension combination within the t-th historical period; T represents the sequence number of the latest historical period; N represents the number of historical periods used to calculate the historical visit baseline, which is determined by the actual business scenario of the enterprise.

[0027] Calculate the ratio between the access volume of each dimension combination in the current period and the corresponding historical access baseline, and determine the dimension combination whose access volume ratio is higher than the first preset ratio threshold as the initial abnormal candidate combination; In practice, the formula for calculating the visit volume ratio is as follows:

[0028] In the formula: This represents the ratio of visits to the i-th dimension combination; This represents the number of visits in the current period for the i-th dimension combination; : The baseline for obtaining historical access; When the access ratio of a certain dimension combination is higher than the first preset ratio threshold (the specific value of the first preset ratio threshold is set by the enterprise according to business needs), the dimension combination is determined to be an initial abnormal candidate combination.

[0029] The resource consumption level of the initial anomaly candidate combination is calculated based on the aggregated calculation of the call count among the dimension members of the initial anomaly candidate combination. The calculation of the resource occupancy level of the initial abnormal candidate combination includes: Obtain the amount of computing resources used by each dimension member in each initial anomaly candidate combination when participating in the aggregation calculation; In practice, the number of aggregate computing calls refers to the amount of standard computing resources consumed each time a dimension member participates in an aggregate computing task. The unit of the computing resource quantity is the number of CPU cores. The amount of computing resources occupied by each dimension member is recorded and obtained in real time through the enterprise's existing system monitoring module, which will not be elaborated further.

[0030] Based on the aggregation frequency of the initial abnormal candidate combination in the most recent N aggregation calculations, determine the resource consumption growth rate when each dimension member is aggregated. In practice, the call frequency of the initial abnormal candidate combination in the most recent consecutive aggregation calculation tasks can be directly obtained through system records. Combined with the amount of computing resources occupied by each dimension member during each call, the formula for calculating the resource consumption growth rate is as follows:

[0031] In the formula: This represents the growth rate of resource consumption for the member in the j-th dimension; This represents the resource consumption of the j-th dimension member in the k-th aggregation task; M represents the number of the most recent aggregation tasks, the specific value of which is set by the enterprise according to business needs. This indicates the time interval between each aggregation task; Based on the superposition effect between the resource consumption growth rates when members of each dimension are aggregated, the cumulative value of resource consumption for the initial abnormal candidate combination is calculated. In practice, the resource consumption growth rates of members in each dimension influence each other during the actual aggregation process, resulting in a cumulative effect of resource consumption. The formula for calculating the cumulative resource consumption of the initial abnormal candidate combination is as follows:

[0032] In the formula: This represents the cumulative resource consumption of the initial abnormal candidate combination; Indicates the lth The growth rate of resource consumption by members in each dimension; Indicates the relationship between the j-th dimension member and the j-th dimension member. The correlation strength between the resource consumption growth rate of each dimension member at the time of aggregation is between 0 and 1, and the specific value of the correlation strength is determined by the analysis of the enterprise's historical data; J represents the total number of dimension members in the initial abnormal candidate combination.

[0033] The resource occupancy level of the initial abnormal candidate combination is determined based on the cumulative value of resource occupancy; In practice, based on the company's historical resource monitoring experience and data, a mapping relationship between the cumulative resource usage value and the resource usage level is preset. For example, the resource usage level can be divided into several levels (e.g., low, medium, high), and the range of cumulative resource usage value corresponding to each level can be clearly defined. When the cumulative resource usage is less than or equal to a certain preset low threshold, the resource usage level is low. When the cumulative resource usage value is between the preset low threshold and the high threshold, the resource usage level is medium. When the cumulative resource usage exceeds the preset high threshold, the resource usage level is high. The above mapping relationship of resource usage levels is set by the enterprise based on actual system operation experience, and the specific values ​​are determined by the enterprise itself based on historical data; For example, based on the above cumulative resource usage value of 0.17035 cores / second, if the enterprise sets the low threshold to 0.05 cores / second and the high threshold to 0.15 cores / second, then the cumulative resource usage value is determined to be higher than the high threshold, and therefore the resource usage level is high.

[0034] Based on the resource occupancy level being higher than the second preset occupancy threshold, a final abnormal dimension combination is determined from the initial abnormal candidate combination, and the resource occupancy status is determined by the resource occupancy level of the final abnormal dimension combination; In practice, the second preset occupancy threshold is determined by the enterprise based on the resource management strategy in the actual scenario. This threshold is usually set in the form of a level of resource occupancy (for example, the threshold is set to the resource occupancy level "medium"). When the resource occupancy level of the initial abnormal candidate combination is higher than the second preset occupancy threshold, the initial abnormal candidate combination is determined to be the final abnormal dimension combination, and its resource occupancy status is determined to be a high load state, indicating that the consumption of system resources by the corresponding dimension combination has exceeded the reasonable resource usage threshold set by the enterprise.

[0035] S102: Based on the enhanced effect of resource competition among dimensions in the abnormal dimension combination, determine the aggregate load gain of the abnormal dimension combination and calculate the resource release elasticity of the response resource occupation of the non-abnormal dimension combination. The abnormal dimension combinations described in this embodiment generate significant resource contention in real-time multidimensional data stream processing. Specifically, this resource contention manifests as follows: when multiple dimension members within the combination simultaneously call aggregate computing resources, the actual resource consumption is significantly higher than the simple sum of the resource consumption when each dimension member calls individually. Therefore, this embodiment defines this enhanced resource consumption phenomenon as an enhanced resource contention effect to characterize the degree of mutual resource influence among dimension members.

[0036] In specific implementation, the calculation of the resource release elasticity of non-abnormal dimension combinations in response to abnormal dimension combinations includes: Extract the real-time resource usage of non-abnormal dimension combinations when the resource load gain effect of abnormal dimension combinations reaches a preset load threshold. In specific implementation, the resource load gain effect is defined as:

[0037] In the formula: This refers to the enhanced resource competition effect when dimension member m and dimension member n jointly participate in aggregation calculation; This represents the actual resource usage growth rate when dimension member m and dimension member n simultaneously access resources. and These are the resource usage growth rates when dimension member m and dimension member n call resources individually, respectively. The specific values ​​are automatically calculated based on the historical records of the enterprise resource monitoring platform. For example, in actual business monitoring, if dimension member A alone has a growth rate of 0.04 cores / second, and dimension member B alone has a growth rate of 0.05 cores / second, but when A and B are called simultaneously, the actual growth rate is 0.12 cores / second, then the resource contention enhancement effect is:

[0038] If the enterprise's preset load threshold is 1.2, then it is determined that the load gain effect of abnormal dimension combinations has exceeded the preset threshold, triggering the extraction of real-time resource usage of non-abnormal dimension combinations.

[0039] At this time, the system obtains real-time resource usage index data for non-abnormal dimension combinations through the enterprise's real-time resource monitoring module. This data includes, but is not limited to, CPU utilization and memory utilization. The real-time collection mechanism for the relevant data relies on the enterprise's existing monitoring platform for automatic collection, and the specific acquisition method is provided by the platform.

[0040] Calculate the change in current resource usage of non-abnormal dimension combinations relative to historical normal resource usage benchmarks to determine the initial value of resource release elasticity; In this embodiment, the formula for calculating the initial value of resource release elasticity is as follows:

[0041] in, Set the initial value for resource release elasticity; For historical normal resource usage benchmarks without abnormal dimension combinations, the platform automatically extracts the average resource usage data within the historical period (such as the average CPU usage rate over 5 historical periods). This represents the current real-time resource usage value, collected in real-time by the enterprise monitoring platform. For example: If the historical baseline for normal resource usage (such as CPU utilization) without abnormal dimension combinations is 60%, and the current real-time resource usage is 45%, then the initial value for resource release elasticity is: This indicates that the combination of non-abnormal dimensions has significant potential for resource release.

[0042] The step of adjusting the initial value of resource release elasticity based on the fluctuation of the resource release process of non-abnormal dimension combinations during the period of change in the resource load gain effect of abnormal dimension combinations includes: In real-world business scenarios, the resource occupancy status of abnormal dimension combinations changes dynamically, resulting in a resource load gain effect that is not stable but fluctuates with real-time changes in business access volume. Similarly, non-abnormal dimension combinations exhibit unstable release speed fluctuations during the response to changes in resource occupancy of abnormal dimension combinations. Therefore, this embodiment further incorporates this fluctuation characteristic by dynamically adjusting the initial value of resource release elasticity to more accurately reflect resource release capacity.

[0043] Determine the changing trend data of the resource load gain effect of abnormal dimension combination at multiple consecutive time points, and calculate the difference in the rate of load change; In specific implementation, this embodiment utilizes an enterprise real-time monitoring platform to continuously collect data sequences of the load gain effect of abnormal dimension combinations within a preset sampling period. The data sequence of the load gain effect is as follows: The load gain effect at each moment is calculated using the aforementioned formula; the formula for calculating the difference in load change rate is as follows:

[0044] In the formula: This represents the difference in the rate of load change at the i-th sampling time. Indicates time The resource load gain effect; For three consecutive sampling times.

[0045] Based on the difference in load change rate, extract the release speed data of the non-abnormal dimension combination resource release process at the corresponding time. In practice, the enterprise real-time monitoring platform will collect load gain data for abnormal dimension combinations while simultaneously recording resource release speed data for non-abnormal dimension combinations at the corresponding time. The resource release rate is defined as the rate at which the resource occupancy of the non-abnormal dimension combination decreases.

[0046] Adjust the initial value of resource release elasticity based on the changes and fluctuations in release rate data; In practice, the degree of fluctuation in release rate data is obtained by calculating the standard deviation, and the calculation formula is as follows: ;in, The standard deviation of the fluctuation in the rate of resource release; Let i be the resource release rate at time i. This represents the average resource release rate over multiple consecutive time points; L is the number of time points collected. After obtaining the degree of fluctuation in resource release rate using the above formula, this embodiment adjusts the initial value of resource release elasticity based on the fluctuation in release rate. The formula for the adjusted resource release elasticity value is as follows:

[0047] In the formula, This represents the initial value of the adjusted resource release elasticity; The initial elastic value obtained from the calculation; The impact coefficient of resource release fluctuations is obtained by enterprises based on historical data fitting experience. The standard deviation of the resource release rate fluctuation.

[0048] The initial value of resource release elasticity is adjusted based on the degree of fluctuation in the resource release process of non-abnormal dimension combinations during the period of change in the resource load gain effect of abnormal dimension combinations. Based on the adjusted initial value of resource release elasticity and the resource load gain effect of abnormal dimension combinations, the final resource release elasticity of non-abnormal dimension combinations is determined. In practice, the final resource release elasticity for non-abnormal dimension combinations is determined according to the following formula:

[0049] in, Release elasticity value for final resources; This is the adjusted elasticity value; This represents the maximum value in the resource load gain effect data sequence.

[0050] S103: Calculate the aggregation path sensitivity based on the dynamic interaction between aggregation load gain and resource release elasticity, and proactively circuit break the corresponding aggregation path based on the abnormal dimension combination. In specific implementation, the aggregation path corresponding to the active circuit breaker anomaly dimension combination includes: The load anomaly level of the aggregation path is determined based on its real-time sensitivity, and the initial circuit breaker strategy for the aggregation path based on the combination of anomaly dimensions is determined according to the load anomaly level. In specific implementation, the formula for calculating the real-time sensitivity of the aggregation path is as follows:

[0051] In the formula: This indicates the real-time sensitivity of the aggregation path; a higher value indicates a higher risk of abnormal resource consumption in the aggregation path. This indicates the resource load gain effect of the abnormal dimension combination corresponding to the aggregation path; This represents the final resource release elasticity value for combinations of non-abnormal dimensions; The classification of aggregated path load anomaly levels is based on the enterprise's business experience, with pre-defined threshold ranges for the real-time sensitivity of aggregated paths. For example: when At this time, the load anomaly level of the aggregation path is "low", and circuit breaking is not required; when When the anomaly level is "medium", the basic circuit breaker strategy is executed; when When the anomaly level is "high", an emergency circuit breaker strategy will be executed; Where the threshold and The specific values ​​are determined independently by the company based on its historical operating experience.

[0052] Extract the associated dimension dependency data of abnormal dimension combinations based on the initial circuit breaker strategy of the aggregation path, and calculate the resource call redundancy between dimensions in the associated dimension dependency data; In practice, the dependency relationship data of related dimensions is automatically obtained through the dimension call relationship log recorded in real time inside the platform, including the number of calls, order and resource consumption between dimension members. The platform automatically counts and records this data, which will not be elaborated further. The redundancy of resource calls between dimensions is defined as the proportion of overlapping resource calls between dimension members to the total number of resource calls. The calculation formula is as follows:

[0053] In the formula: Redundancy in resource allocation between dimensions; The platform obtains the amount of resource overlap jointly invoked by dimension member m and dimension member n in real time. The total amount of resources called by dimension member m and dimension member n is obtained by the platform in real time through recording and statistics. The initial circuit breaker strategy for the aggregated path of the abnormal dimension combination is optimized based on the resource call redundancy, and the final circuit breaker strategy for the aggregated path of the abnormal dimension combination is obtained. The initial circuit breaker strategy for aggregated paths based on optimizing the combination of abnormal dimensions according to resource call redundancy includes: Based on the resource call data between dimensions, determine the degree of overlap in resource calls within abnormal dimension combinations; In practice, the resource call overlap degree is defined as the proportion of the total overlap of resource calls among all dimension members within an abnormal dimension combination to the total total resource calls, as shown in the following formula:

[0054] In the formula: denoted as the resource call overlap degree of the abnormal dimension combination; D is the set of all dimension members within the abnormal dimension combination. Extract the target dimension combination with the highest degree of resource call overlap, and analyze the degree of matching between the dimension call order within the target dimension combination and the dimension call order within the abnormal dimension combination; In practice, the platform records the order in which members of each dimension call resources in real time, and defines the matching degree of dimension call order as the proportion of dimension members whose call order is completely consistent between two dimension combinations to the total number of dimension members:

[0055] In the formula, The degree of matching between the call order of the target dimension combination and the abnormal dimension combination; and These are the sets of call orders for dimension members within the target dimension combination and the abnormal dimension combination, respectively. Based on the degree of matching of the dimension call order, the initial circuit breaker strategy of the aggregation path is adjusted, and the final circuit breaker strategy of the aggregation path is optimized. In specific implementation, if A higher value (e.g., ≥0.8) indicates that the target dimension combination and the abnormal dimension combination have highly similar resource call behaviors. The circuit breaker strategy for the aggregate path corresponding to the abnormal combination can be adjusted to only circuit breaker the path of the dimension member with high redundancy in order to reduce the scope of circuit breaker and loss. If the matching degree is low, the initial circuit breaker strategy should be strictly implemented.

[0056] Based on the final circuit breaker strategy of the aggregation path, actively breaker the aggregation path corresponding to the combination of abnormal dimensions. In practice, the proactive circuit breaking of aggregated paths is achieved through the path management module built into the enterprise's existing platform. Based on the optimized final circuit breaking strategy, the system sends a circuit breaking command to the aggregated path management module in real time. The module executes the circuit breaking operation to temporarily close the aggregated paths corresponding to the abnormal dimension combinations, so as to quickly reduce the abnormal load of aggregated paths and protect the platform's operational stability.

[0057] S104: Update the aggregation path sensitivity based on the real-time feedback of resource release status of non-abnormal dimension combinations to automatically restore the broken aggregation path; In specific implementation, the updating of aggregated path sensitivity based on the resource release status feedback in real time according to the non-abnormal dimension combination includes: Real-time extraction of resource release status data for non-abnormal dimension combinations during circuit breaker periods for abnormal dimension combinations, and determination of the changing trend of resource release status data; In practice, the resource release status data includes indicators such as real-time CPU utilization, memory utilization, and aggregate computing task queue length, which are not abnormal. This data is automatically collected in real time through the enterprise's existing real-time resource monitoring platform, with a specific collection frequency of once per second. The collected data is automatically stored in the time-series database within the platform. For example, during the circuit breaker period of abnormal combination paths, a set of data is recorded every second for non-abnormal dimension combinations (Table 1): Table 1: Data Table of Resource Release Status Time (s) CPU utilization (%) Memory utilization (%) Task queue length 1 45 60 10 2 44 58 9 3 42 55 7 4 40 52 6 By performing linear regression analysis on resource usage data at multiple consecutive time points, the changing trend of resource release status data can be determined. For example, in the data above, CPU, memory usage, and task queue length all show a significant downward trend, indicating that the resource release speed is relatively stable.

[0058] Based on the changing trend of resource release status data, calculate the resource release rate of the non-abnormal dimension combination before the aggregation path is restored; In practice, the resource release rate is defined as the average decrease in system resources released per unit time without abnormal dimension combinations, and the calculation formula is as follows:

[0059] In the formula: For resource release rate; This represents the initial resource usage value. This represents the current resource usage value. This indicates the time interval corresponding to changes in resource usage; the higher this value, the faster the non-abnormal dimension combination recovers to normal resource usage levels, and the better the system resource recovery status.

[0060] Based on the degree of coupling between the resource release rate and the resource release elasticity of non-abnormal dimension combinations, the real-time sensitivity of the aggregation path is modified. In practice, the coupling degree between resource release rate and resource release elasticity is defined as follows:

[0061] In the formula: For the degree of coupling, For resource release rate, For combinations of non-abnormal dimensions, the final resource release elasticity; The greater the degree of coupling mentioned above, the closer the actual ability of non-abnormal dimensional combinations to release resources is to their theoretical maximum elasticity; Based on this, the formula for correcting the real-time sensitivity of the aggregation path is:

[0062] In the formula: Real-time sensitivity of the corrected aggregation path; The current sensitivity of the aggregation path; The sensitivity correction coefficient set for the system is obtained by the enterprise based on empirical fitting of historical data; This refers to the degree of coupling between the aforementioned resource release rate and elasticity; Specific example: Assuming the current aggregation path sensitivity Given a resource release rate of 1.67% / s and a final resource release elasticity of 25% (i.e., 0.25), the degree of coupling is:

[0063] If the system sensitivity correction coefficient Setting it to 5 results in the corrected aggregation path sensitivity:

[0064] This indicates that the sensitivity of the aggregation path has decreased, meaning that the risk of the aggregation path has been significantly reduced, and the conditions for resuming the circuit breaker path are met.

[0065] In practice, enterprises pre-set the aggregation path sensitivity recovery threshold within the system. For example, if set to 1.5, the real-time sensitivity of the corrected aggregation path will meet the following conditions:

[0066] If the system resource usage level has been restored to normal, the system will automatically send a recovery command to the aggregation path management module to automatically restore the previously broken aggregation path, thereby achieving automated closed-loop management of the normal execution of aggregation computing tasks. In the specific example, the aforementioned sensitivity correction value is 1.29. If the enterprise's preset recovery threshold is 1.5, then the condition has been met, the system automatically releases the circuit breaker operation on the abnormal dimension combination aggregation path, and resumes normal task execution.

[0067] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters, weights, and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0068] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0069] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A high-performance index calculation method supporting automatic aggregation of multidimensional data, characterized in that, include: S101: Determine the current access volume of each dimension combination based on the real-time multidimensional data stream, identify abnormal dimension combinations whose access volume significantly exceeds the historical threshold, and obtain the resource usage status of abnormal dimension combinations. S102: Based on the enhanced effect of resource competition among dimensions in the abnormal dimension combination, determine the aggregate load gain of the abnormal dimension combination and calculate the resource release elasticity of the response resource occupation of the non-abnormal dimension combination. S103: Calculate the aggregation path sensitivity based on the dynamic interaction between aggregation load gain and resource release elasticity, and proactively circuit break the corresponding aggregation path based on the abnormal dimension combination. S104: Update the aggregation path sensitivity based on the real-time feedback of resource release status of non-abnormal dimension combinations to automatically restore the aggregated path that has been broken.

2. The high-performance index calculation method supporting automatic aggregation of multi-dimensional data according to claim 1, characterized in that, The process of identifying abnormal dimension combinations whose access volume significantly exceeds historical thresholds and obtaining the resource usage status of these abnormal dimension combinations includes: Extract the access volume data of each dimension combination in the most recent T historical periods to obtain the historical access baseline of each dimension combination; Calculate the ratio between the access volume of each dimension combination in the current period and the corresponding historical access baseline, and determine the dimension combination whose access volume ratio is higher than the first preset ratio threshold as the initial abnormal candidate combination; The resource consumption level of the initial anomaly candidate combination is calculated based on the aggregated calculation of the call count among the dimension members of the initial anomaly candidate combination. Based on the resource occupancy level being higher than the second preset occupancy threshold, a final abnormal dimension combination is determined from the initial abnormal candidate combination, and the resource occupancy status is determined by the resource occupancy level of the final abnormal dimension combination.

3. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 2, characterized in that, The calculation of the resource consumption level of the initial abnormal candidate combination includes: Obtain the amount of computing resources used by each dimension member in each initial anomaly candidate combination when participating in the aggregation calculation; Based on the aggregation frequency of the initial abnormal candidate combination in the most recent N aggregation calculations, determine the resource consumption growth rate when each dimension member is aggregated. Based on the superposition effect between the resource consumption growth rates when members of each dimension are aggregated, the cumulative value of resource consumption for the initial abnormal candidate combination is calculated. The resource usage level of the initial abnormal candidate combination is determined based on the cumulative value of resource usage.

4. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 3, characterized in that, The calculation of the resource release elasticity for non-abnormal dimension combinations in response to abnormal dimension combinations includes: Extract the real-time resource usage of non-abnormal dimension combinations when the resource load gain effect of abnormal dimension combinations reaches a preset load threshold. Calculate the change in current resource usage of non-abnormal dimension combinations relative to historical normal resource usage benchmarks to determine the initial value of resource release elasticity; The initial value of resource release elasticity is adjusted based on the degree of fluctuation in the resource release process of non-abnormal dimension combinations during the period of change in the resource load gain effect of abnormal dimension combinations. Based on the adjusted initial value of resource release elasticity and the resource load gain effect of abnormal dimension combinations, the final resource release elasticity of non-abnormal dimension combinations is determined.

5. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 4, characterized in that, The adjustment of the initial value of resource release elasticity based on the fluctuation of the resource release process during the period of change in the load gain effect of abnormal dimension combinations includes: Determine the changing trend data of the resource load gain effect of abnormal dimension combination at multiple consecutive time points, and calculate the difference in the rate of load change; Based on the difference in load change rate, extract the release speed data of the non-abnormal dimension combination resource release process at the corresponding time. Adjust the initial value of resource release elasticity based on the fluctuations in release rate data.

6. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 5, characterized in that, The aggregation path corresponding to the active circuit breaker anomaly dimension combination includes: The load anomaly level of the aggregation path is determined based on the real-time sensitivity of the aggregation path, and the initial circuit breaker strategy of the aggregation path with the combination of anomaly dimensions is determined based on the load anomaly level. Extract the associated dimension dependency data of abnormal dimension combinations based on the initial circuit breaker strategy of the aggregation path, and calculate the resource call redundancy between dimensions in the associated dimension dependency data; The initial circuit breaker strategy for the aggregated path of the abnormal dimension combination is optimized based on the resource call redundancy, and the final circuit breaker strategy for the aggregated path of the abnormal dimension combination is obtained. Based on the final circuit breaker strategy of the aggregation path, the aggregation path corresponding to the combination of abnormal dimensions is actively circuit breaker.

7. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 6, characterized in that, The initial circuit breaker strategy for aggregated paths that optimizes the combination of abnormal dimensions based on resource call redundancy includes: Based on the resource call data between dimensions, determine the degree of overlap in resource calls within abnormal dimension combinations; Extract the target dimension combination with the highest degree of resource call overlap, and analyze the degree of matching between the dimension call order within the target dimension combination and the dimension call order within the abnormal dimension combination; Based on the degree of matching of the dimension call order, the initial circuit breaker strategy of the aggregation path is adjusted, and the final circuit breaker strategy of the aggregation path is optimized.

8. The high-performance index calculation method supporting automatic aggregation of multidimensional data according to claim 7, characterized in that, The method of updating the aggregated path sensitivity based on the real-time feedback of resource release status according to the combination of non-abnormal dimensions includes: Real-time extraction of resource release status data for non-abnormal dimension combinations during circuit breaker periods for abnormal dimension combinations, and determination of the changing trend of resource release status data; Based on the changing trend of resource release status data, calculate the resource release rate of the non-abnormal dimension combination before the aggregation path is restored; Based on the degree of coupling between the resource release rate and the resource release elasticity of non-abnormal dimension combinations, the real-time sensitivity of the aggregation path is modified.

Images