A data platform resource consumption monitoring system

Abstract

The application relates to the technical field of data platform resource management, and discloses a data platform resource consumption monitoring system, which comprises a resource identification generation module, a consumption feature collection module, an abnormality discrimination module and a response adjustment module. The resource identification generation module generates a unique identification corresponding to a key level according to a task access frequency, a data size and an operation type; the consumption feature collection module integrates initial, intermediate and end feature collection values to form a resource consumption feature verification result; the abnormality discrimination module determines an identification abnormal value based on analysis state and time consumption data, and outputs an abnormality monitoring value in combination with a benchmark value; and the response adjustment module uses a multi-dimensional resource abnormality response model to comprehensively optimize a resource path by using a key level and historical data. The system realizes dynamic monitoring of resource consumption, accurate discrimination of abnormalities and intelligent adjustment of a path, and improves resource management efficiency and platform stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data platform resource management, in particular to a data platform resource consumption monitoring system. BACKGROUND

[0002] With the rapid development of big data technology, resource management of data platforms faces great challenges. Data platforms often need to handle massive tasks, which differ significantly in access frequency, data size, and operation type, leading to uneven resource consumption. Traditional resource monitoring systems mostly use static thresholds or simple rules for anomaly detection, which are difficult to dynamically adapt to the key differences of tasks and cannot fully capture the timing characteristics of resource consumption. For example, existing technologies usually only collect the initial and end states of resource nodes, ignoring the dynamic changes in the intermediate process, leading to delayed or false anomaly detection. In addition, resource path adjustment mostly relies on manual intervention, lacks intelligent optimization mechanisms based on multi-dimensional historical data, and is difficult to balance resource occupation and response speed.

[0003] The current system also has the following problems: first, the resource identifier generation is not related to the task criticality level, and it is not possible to allocate a safer identifier for high-priority tasks; second, the anomaly detection relies on a single indicator (such as CPU usage), and does not integrate and analyze multi-dimensional data such as state, time consumption, etc., resulting in a high false negative rate; third, the response adjustment lacks comprehensive analysis of historical characteristics and abnormal values, and the path selection efficiency is low. These problems lead to resource waste, response delay, and even system crashes. Therefore, a monitoring system that can dynamically evaluate task criticality level, real-time collect multi-dimensional consumption characteristics, intelligently detect anomalies, and automatically optimize resource paths is needed to improve the stability and resource utilization of data platforms. SUMMARY

[0004] The purpose of the present application is to provide a data platform resource consumption monitoring system to solve the problems raised in the background art.

[0005] To achieve the above purpose, the present application provides a data platform resource consumption monitoring system, which comprises:

[0006] A resource identifier generation module is configured to generate a unique resource identifier for a data platform resource usage task; determine the criticality level of the resource usage task according to the access frequency attribute, data size attribute, and operation type attribute of the resource usage task, and perform resource identifier generation based on the criticality level;

[0007] The consumption feature collection module is configured to collect and verify the resource consumption behavior, and the specific process includes: integrating the initial feature collection value of the current resource node, the intermediate feature collection value of each fixed time interval in the resource use process, and the end feature collection value of the current resource node and the next node to form a resource consumption feature set; integrating the resource consumption feature set and the resource consumption feature benchmark to form a resource consumption feature verification result;

[0008] The anomaly discrimination module is configured to monitor the identification anomaly state and the consumption anomaly state in the resource use process; determine the resource identification anomaly value according to the analysis state, the analysis time consumption and the analysis data content of the resource identification analysis device; determine the resource consumption anomaly monitoring value according to the resource identification anomaly value and the identification anomaly benchmark value; determine the resource consumption anomaly information according to the resource consumption anomaly monitoring value;

[0009] The response adjustment module is configured to handle the abnormal response and path adjustment in the resource use process; integrate the key level, the historical feature verification result and the historical anomaly monitoring value of the resource path between the resource nodes, use a multi-dimensional resource anomaly response model to adjust the current resource path, and determine the optimal response path.

[0010] Preferably, the specific process of the resource identification generation module based on the key level includes:

[0011] According to the proportion of the failed task amount in the total resource task amount, the access frequency attribute of the resource use task is determined;

[0012] The operation type attribute, the data size attribute and the access frequency attribute of the resource use task are evaluated to determine the key level of the resource use task;

[0013] The key level of the resource use task is integrated to generate a corresponding resource identification for the resource use task based on the coding rule screening, the capacity adaptation screening and the security level screening.

[0014] Preferably, the process of the consumption feature collection module obtaining the resource consumption feature verification result includes:

[0015] According to the resource parameters of the input end and the output end in the current resource node, the initial feature collection value is determined;

[0016] According to the resource parameters of each fixed time interval in the resource use process and the resource parameters before use, the intermediate feature collection value is determined;

[0017] According to the resource parameters of the output end of the current resource node and the input end of the next node, the end feature collection value is determined;

[0018] Integrate the initial feature collection value, the intermediate feature collection value and the tail feature collection value to form the resource consumption feature set;

[0019] Compare the resource consumption feature set with the resource consumption feature benchmark to form a resource consumption feature verification result.

[0020] Preferably, the implementation process of the resource consumption anomaly monitoring value determined by the anomaly discrimination module includes:

[0021] Obtain the parsing state, the parsing time consumption and the parsing data content of the resource identification parsing device in the resource usage process;

[0022] According to the parsing state, the parsing data content, the parsing position information and the parsing data details, respectively determine the parsing state weight value, the position information matching value and the intermediate feature collection value;

[0023] Evaluate the parsing state weight value, the parsing time consumption, the position information matching value and the intermediate feature collection value to determine the resource identification anomaly value;

[0024] Compare the resource identification anomaly value with the identification anomaly benchmark value. If it does not exceed the identification anomaly benchmark value, perform feature matching between real-time resource data in the resource usage process and original data before use to determine the resource consumption anomaly monitoring value; otherwise, determine the resource consumption anomaly monitoring value according to the resource identification anomaly value and feedback an abnormal prompt.

[0025] Preferably, the process of determining the optimal response path by the response adjustment module using the multi-dimensional resource anomaly response model includes:

[0026] Obtain the key level of the resource usage task and the historical feature verification result and the historical anomaly monitoring value of the resource path;

[0027] Construct the multi-dimensional resource anomaly response model, taking reducing response resource occupation and shortening response time length as optimization objectives, and integrate the key level, the historical feature verification result and the historical anomaly monitoring value to determine the comprehensive optimization objective of the multi-dimensional resource anomaly response model;

[0028] According to the comprehensive optimization objective, solve and determine the optimal response path.

[0029] Preferably, the construction process of the comprehensive optimization objective includes:

[0030] Taking reducing response resource occupation as the first optimization direction and shortening response time length as the second optimization direction;

[0031] The key level is taken as a first weight factor, the historical feature verification result is taken as a second weight factor, and the historical abnormal monitoring value is taken as a third weight factor.

[0032] The optimization directions and the weight factors are integrated to form a comprehensive optimization target of the multi-dimensional resource abnormal response model.

[0033] Preferably, the application further comprises a resource abnormal early warning module for generating early warning information before resource consumption abnormality occurs; specifically, historical resource consumption feature verification results and a current resource consumption feature set are obtained, feature change trends in a preset early warning time window are integrated, and a resource consumption abnormal early warning threshold is determined; the current resource consumption feature set is compared with the resource consumption abnormal early warning threshold, and if the early warning condition is reached, resource consumption abnormal early warning information is generated and sent to a management terminal.

[0034] Preferably, the application further comprises a resource data storage module for storing key data information in a resource use process; specifically, data storage priorities are divided based on the key level of a resource use task; resource identifiers generated by a resource identifier generation module, resource parameters obtained by a consumption feature acquisition module, and resource consumption abnormal monitoring values determined by an abnormality discrimination module are stored in different storage media according to the data storage priorities, forming a traceable resource data archive.

[0035] Preferably, the specific process of the response adjustment module adjusting a current resource path comprises: obtaining node load states, resource delay data and bandwidth occupancy rates of the current resource path; integrating the key level, the historical feature verification result and the historical abnormal monitoring value to determine adjustment priorities of each candidate path; load balancing degrees and resource stabilities of candidate paths are evaluated in sequence according to the adjustment priorities, and a path with the optimal load balancing degree and up-to-standard resource stability is selected as the optimal response path.

[0036] Preferably, the application further comprises a resource behavior verification module for verifying the legality of resource use behavior; specifically, resource identifiers generated by a resource identifier generation module are obtained, and verification fields are extracted therefrom; the verification fields are compared with a preset legal identifier library, and if matching is successful, the resource use behavior is marked as legal, and if matching fails, the resource use behavior is marked as having a risk of forgery, and the forgery risk information is fed back to an abnormality discrimination module.

[0037] Compared with the prior art, the application has the following beneficial effects:

[0038] The data platform resource consumption monitoring system provided by the application improves the accuracy and efficiency of resource management through the cooperative work of multiple modules. The resource identifier generation module dynamically generates a unique identifier based on the task criticality level, ensuring that high-priority tasks obtain a higher security level of resource identifier, thereby avoiding resource conflicts or malicious occupation. The consumption feature acquisition module integrates initial, intermediate, and end feature acquisition values to comprehensively capture the timing changes of resource consumption, solving the detection blind spot problem caused by ignoring the intermediate process in traditional methods, making the feature verification result more accurate and reliable.

[0039] The abnormality discrimination module calculates the resource identifier abnormal value through the analysis of state, time consumption, and data content, and dynamically adjusts the abnormal monitoring threshold value combined with the reference value, greatly reducing the false negative and false positive rates. The response adjustment module uses a multi-dimensional resource abnormal response model to reduce resource occupation and shorten response time as the goal, comprehensively considers the key level, historical verification results, and abnormal monitoring values, intelligently selects the optimal response path, and effectively improves the rationality of resource allocation. The system also identifies potential risks in advance through the resource abnormality early warning module, generates early warning information combined with the feature change trend within the preset time window, helps administrators intervene in advance, and avoids system collapse.

[0040] In addition, the resource data storage module stores data according to the key level, forming a traceable resource file for subsequent analysis and optimization. The resource behavior verification module effectively identifies fake behaviors by comparing the verification field with the legal identifier library, enhancing the security of the system. Overall, the application realizes the whole life cycle monitoring of resource consumption, accurate abnormality discrimination, and dynamic optimization of the path, reduces resource waste, improves the stability and response efficiency of the data platform, and is suitable for large-scale, high-concurrency data processing scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The working principle diagram of the data platform resource consumption monitoring system described in the application;

[0042] Figure 2 The design diagram of the multi-source education data collection and collection;

[0043] Figure 3 The design diagram of the consumption feature acquisition module;

[0044] Figure 4 The design diagram of the abnormality discrimination module. DETAILED DESCRIPTION

[0045] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0046] Please refer to Figures 1-4 The present application provides a data platform resource consumption monitoring system, and the specific implementation steps are as follows:

[0047] The resource identification generation module is configured to generate a unique resource identification for a data platform resource use task, determine a key level of the resource use task according to an access frequency attribute, a data size attribute and an operation type attribute of the resource use task, and perform resource identification generation based on the key level.

[0048] The consumption feature collection module is configured to collect and verify features of resource consumption behavior. The specific process includes: integrating an initial feature collection value of a current resource node, an intermediate feature collection value of the resource use process every fixed time interval, and an end feature collection value of the current resource node and the next node to form a resource consumption feature set; and integrating the resource consumption feature set and a resource consumption feature benchmark to form a resource consumption feature verification result.

[0049] The anomaly discrimination module is configured to monitor identification anomaly states and consumption anomaly states in the resource use process. The anomaly discrimination module is configured to determine a resource identification anomaly value according to a parsing state, a parsing time consumption and a parsed data content of a resource identification parsing device, determine a resource consumption anomaly monitoring value according to the resource identification anomaly value and an identification anomaly benchmark value, and determine resource consumption anomaly information according to the resource consumption anomaly monitoring value.

[0050] The response adjustment module is configured to handle abnormal responses and path adjustments in the resource use process. The response adjustment module is configured to integrate the key level, historical feature verification results and historical anomaly monitoring values of resource paths between resource nodes, use a multi-dimensional resource anomaly response model to adjust a current resource path, and determine an optimal response path.

[0051] Embodiment 1;

[0052] The resource identification generation module of the system performs resource identification generation based on the key level, and the specific implementation is as follows:

[0053] The access frequency attribute of the resource usage task is determined, which needs to be based on the proportion of the failed task amount in the total amount of resource tasks. During the operation of the data platform, resource usage tasks will have different execution situations, some tasks are successfully completed, and some tasks may fail. By counting the total amount of resource tasks and the number of failed tasks in a certain period of time, the proportion of the failed task amount in the total amount is calculated, which can reflect the access frequency attribute of the resource usage task. For example, in a certain period of time, the total amount of resource tasks is 1000, and the failed task amount is 100, so the proportion of the failed task amount is 10%, which can be used as an important basis for determining the access frequency attribute.

[0054] The priority attribute, data size attribute and the determined access frequency attribute of the resource usage task are evaluated to determine the critical level of the resource usage task. The priority attribute is set according to the importance of the task in the data platform, such as the priority of some key business process tasks is high, and the priority of some auxiliary tasks is relatively low. The data size attribute is related to the data size processed by the task, and there is a significant difference between tasks processing large data and tasks processing small data. In the evaluation process, the three attributes need to be considered comprehensively. Different weights can be set for each attribute, and then a comprehensive evaluation result is obtained through weighted calculation, which is the critical level of the resource usage task. For example, the weight of the priority attribute is set to 40%, the weight of the data size attribute is set to 30%, and the weight of the access frequency attribute is set to 30%, then each attribute is scored, and the critical level is calculated according to the weight.

[0055] After determining the critical level of the resource usage task, the critical level needs to be integrated, and the corresponding resource identifier is generated for the resource usage task based on the coding rule screening, capacity adaptation screening and security level screening. The coding rule screening needs to comply with the coding specification of the data platform, and the data platform usually has a set of unified coding rules to ensure that the generated resource identifier has consistency and standardization in format and structure. For example, the coding may need to include specific characters, number combinations, or have certain length requirements, etc. The capacity adaptation screening needs to consider the data size of the resource usage task to ensure that the generated resource identifier can adapt to the task in capacity. Different data sizes may require different capacity resource identifiers to carry related information to ensure that the identifier can accurately and completely represent the resource usage task. The security level screening needs to determine the corresponding security level requirement according to the critical level. The resource usage task with high critical level has relatively high security level requirement, and more stringent security measures need to be taken when generating the resource identifier to ensure the security of the resource identifier and prevent the identifier from being tampered with or forged.

[0056] In the encoding rule screening, the appropriate encoding rule for the resource usage task is selected from the encoding specifications of the data platform. For example, for a certain type of resource usage task, it may be specified that the encoding must start with a specific letter, followed by a certain number of digits and letter combinations. In the capacity adaptation screening process, the data volume of the resource usage task is analyzed to determine the required identification capacity. If the data volume is large, a larger capacity identification format needs to be selected to ensure that all relevant information can be accommodated. The security level screening determines the corresponding encryption method, access rights and other security measures according to the key level. For example, a task with a high key level may need to use a higher level encryption algorithm to encrypt the resource identifier, and at the same time limit access to the identifier to only specific users or systems.

[0057] Through the above series of steps, from determining the access frequency attribute to evaluating the key level, and then generating the resource identifier through various screenings, the key level evaluation of the resource usage task and the generation of the resource identifier are realized. This process provides a unique identification basis for subsequent resource consumption monitoring, so that in the operations of feature collection, anomaly discrimination and response adjustment of resource consumption behavior, the specific resource usage task can be accurately corresponded to, thereby realizing effective monitoring and management of data platform resource consumption. In the whole process, each link is closely connected, and the result of the previous link provides the necessary basic information for the next link, ensuring the accuracy and effectiveness of the generation of the resource identifier. For example, accurate access frequency attribute is one of the important factors for evaluating the key level, and reasonable key level affects the selection of encoding rule, capacity adaptation and security level screening, and finally affects the generation quality of the resource identifier.

[0058] Embodiment 2;

[0059] This embodiment describes the process of the resource consumption feature collection module obtaining the resource consumption feature verification result, which is as follows:

[0060] The initial feature collection value is determined according to the resource parameters of the input and output ends of the current resource node. In the resource usage process of the data platform, each resource node has an input end and an output end. The input end receives resources from other nodes, and the output end delivers resources to subsequent nodes. Resource parameters include various types, such as CPU usage, which reflects the occupation of the central processing unit when the node processes resources; memory usage, which represents the degree of use of memory resources during node operation; and data throughput, which represents the amount of data passing through the node per unit time. When determining the initial feature collection value, these resource parameters of the input and output ends need to be collected in real time. For example, at a certain time, the CPU usage of the input end of the current resource node is 30%, the memory usage is 200 MB, and the data throughput is 10 MB / s. The CPU usage of the output end is 25%, the memory usage is 180 MB, and the data throughput is 8 MB / s. These parameter values after processing can be used as the initial feature collection value to record the resource consumption characteristics of the resource node in the initial state.

[0061] In the resource usage process, every fixed period, the resource parameters of the period and the resource parameters before the period need to be obtained, so as to determine the intermediate feature collection value. The fixed period here can be set according to the actual operation of the data platform and the characteristics of resource consumption. For example, for nodes with rapid resource consumption changes, the fixed period can be set to 1 minute to capture the dynamic changes of resource consumption in time; for nodes with relatively stable resource consumption, the fixed period can be set to 5 minutes or longer. At the end of each fixed period, the resource parameters of the current period, such as CPU usage, memory usage, and data throughput, are collected, and the resource parameters before the start of the period are obtained. By comparing the resource parameters of the two periods, the parameter change values are calculated, which are the intermediate feature collection values. For example, before the start of a certain fixed period, the CPU usage of the resource node is 25%, the memory usage is 180 MB, and the data throughput is 8 MB / s. After 1 minute, the CPU usage at the end of the period is 35%, the memory usage is 220 MB, and the data throughput is 12 MB / s. The change value of the CPU usage is 10%, the change value of the memory usage is 40 MB, and the change value of the data throughput is 4 MB / s. These change values constitute the intermediate feature collection value of the fixed period, which reflects the changes in resource consumption in this time period.

[0062] According to the resource parameters of the output end of the current resource node and the input end of the next node, the tail feature collection value is determined. When the resource is processed in the current node, it will be passed to the next node. At this time, the resource parameter transmission between the output end of the current node and the input end of the next node is very important. The resource parameters of the output end of the current node and the input end of the next node need to be collected, such as CPU usage, memory usage, data throughput, etc. of the output end of the current node, and the corresponding parameters of the input end of the next node. By analyzing these parameters, the tail feature collection value can be determined. For example, the CPU usage of the output end of the current node is 30%, the memory usage is 200MB, the data throughput is 10MB / s, the CPU usage of the input end of the next node is 32%, the memory usage is 210MB, and the data throughput is 9MB / s. After processing these parameter values, they can be used as tail feature collection values to represent the consumption characteristics of resource transmission between the current node and the next node.

[0063] After obtaining the initial feature collection value, the intermediate feature collection value and the tail feature collection value, these values need to be integrated to form a resource consumption feature set. The integration process is to combine the feature collection values at different stages according to certain order and rules, so that it becomes a complete set. This set records all the feature changes of the resource from the initial state to the intermediate change and then to the transmission to the next node in the use process. For example, the parameters in the initial feature collection value, the intermediate feature collection value of each fixed period and the parameters in the tail feature collection value are summarized to form a set containing multiple time points and multiple resource parameters, which can fully reflect the whole process characteristics of resource consumption.

[0064] The resource consumption feature set is compared with the resource consumption feature benchmark to form the resource consumption feature verification result. The resource consumption feature benchmark is a standard range preset according to historical normal resource consumption data, which contains the reasonable fluctuation range of each resource parameter under normal circumstances. In the comparison, each feature value in the resource consumption feature set is compared with the corresponding standard in the benchmark to determine whether it is within the reasonable range. If all feature values are within the benchmark range, it means that the current resource consumption feature is normal; if some feature values exceed the benchmark range, it means that the resource consumption may have abnormal conditions. For example, the normal range of CPU usage in the resource consumption feature benchmark is 20%-80%, the normal range of memory usage is 100MB-500MB, and the normal range of data throughput is 5MB / s-15MB / s. When a CPU usage value in the resource consumption feature set is 85%, which exceeds the benchmark range, the abnormal condition is recorded in the verification result, providing a basis for subsequent abnormal discrimination.

[0065] Through the above series of steps, from determining the initial feature collection value to obtaining the intermediate feature collection value, the end feature collection value, and then integrating to form the resource consumption feature set and comparing with the benchmark, the feature collection and verification of resource consumption behavior are realized. This process can comprehensively and accurately record the feature changes of resource consumption, providing an important basis for the data platform resource consumption monitoring system to judge whether the resource consumption is normal. In the whole process, each link needs to accurately collect and process data to ensure the accuracy and reliability of the feature collection value, so as to ensure the effectiveness of the resource consumption feature verification result. For example, accurate collection of the initial feature collection value can provide a correct starting point for subsequent feature change analysis, timely and accurate acquisition of the intermediate feature collection value can reflect the dynamic process of resource consumption, accurate determination of the end feature collection value can reflect the resource transfer between nodes, and reasonable setting of the resource consumption feature benchmark is the key to judging normality or not.

[0066] Embodiment 3;

[0067] The implementation process of the abnormality discrimination module to determine the resource consumption abnormality monitoring value is as follows:

[0068] The parsing state, parsing time consumption and parsing data content of the resource identification parsing device in the resource usage process need to be obtained. The resource identification parsing device is responsible for parsing the resource identification in the data platform to obtain the information contained therein. The parsing state reflects the working condition of the parsing device in the parsing process, such as whether it is normally parsed, whether there is an error, etc.; the parsing time consumption is the time spent from starting to parse the resource identification to completing the parsing; the parsing data content contains specific information carried in the resource identification, such as parsing location information, parsing data details, etc. In the resource usage process, these parameters of the parsing device need to be monitored in real time, such as obtaining the identification of the parsing state (such as normal, abnormal) through system log recording or real-time monitoring program, the specific value of the parsing time consumption (in milliseconds) and the detailed information of the parsing data content.

[0069] According to the analysis state, the analysis position information in the analysis data content and the analysis data details, the analysis state weight value, the position information matching value and the intermediate characteristic collection value are determined respectively. The determination of the analysis state weight value is based on the working state of the analysis device. For example, when the analysis state is normal, a higher weight value can be set, such as 0.7; when the analysis state appears partial error, the weight value can be adjusted to 0.4; if the analysis state is completely wrong, the weight value is set to 0.1. The analysis position information refers to the specific position of data analysis in the resource identifier, which is compared with the preset standard analysis position information, and the position information matching value is determined according to the matching degree. For example, the preset analysis position information is a certain specific field order and position range, if the actual analysis position information completely matches the preset information, the matching value is 1; if there is partial deviation, the matching value is given according to the deviation degree, 0.5-0.9; if there is serious deviation, the matching value is 0.1-0.4. The analysis data details contain the specific data content in the resource identifier, and the intermediate characteristic collection value comes from the intermediate characteristic data recorded by the consumption characteristic collection module in the resource use process, such as the change value of CPU usage rate, the change value of memory occupation, etc., which need to be associated with the related parameters in the current analysis data content to obtain.

[0070] The analysis state weight value, the analysis time consumption, the position information matching value and the intermediate characteristic collection value are evaluated to determine the resource identifier abnormal value. In the evaluation process, the corresponding weight needs to be set for each parameter, and the weight setting is based on the influence degree of each parameter on the resource identifier abnormality. For example, the weight of the analysis state weight value can be set to 30%, the weight of the analysis time consumption is 25%, the weight of the position information matching value is 25%, and the weight of the intermediate characteristic collection value is 20%. Then, the values of each parameter are multiplied by the corresponding weight and added to obtain the resource identifier abnormal value. For example, the analysis state weight value is 0.7 (corresponding to the weight 30%), the analysis time consumption is 150 milliseconds (assuming that the normal time consumption range is 100-200 milliseconds, which is standardized to 0.8 corresponding to the weight 25%), the position information matching value is 0.9 (corresponding to the weight 25%), and the intermediate characteristic collection value is 0.6 (corresponding to the weight 20%), then the resource identifier abnormal value is 0.7x0.3+0.8x0.25+0.9x0.25+0.6x0.2=0.21+0.2+0.225+0.12=0.755.

[0071] The resource identification abnormal value is compared with an identification abnormal benchmark value. The identification abnormal benchmark value is a threshold value preset according to historical normal analysis data, and is used to determine whether the resource identification is abnormal. If the resource identification abnormal value does not exceed the identification abnormal benchmark value, it indicates that the analysis state of the resource identification is basically normal, but it is still necessary to further determine whether the resource consumption is abnormal. At this time, real-time resource data in the resource usage process is matched with original data before usage to determine a resource consumption abnormal monitoring value. The real-time resource data includes the current CPU usage rate, memory occupancy, data throughput, etc., and the original data is the initial resource parameter recorded before the resource usage task starts. The resource consumption abnormal monitoring value is determined by calculating the difference degree of the real-time resource data and the original data, such as the change rate and fluctuation range of each parameter. For example, the real-time CPU usage rate is increased by 20% compared with the original data, the memory occupancy is increased by 15%, and the data throughput is increased by 10%. These change values are integrated through a certain algorithm to obtain the resource consumption abnormal monitoring value.

[0072] If the resource identification abnormal value exceeds the identification abnormal benchmark value, it indicates that the analysis of the resource identification is obviously abnormal, at which time the resource consumption abnormal monitoring value is directly determined according to the resource identification abnormal value, and an abnormal prompt is fed back. The abnormal prompt can include specific conditions of the resource identification abnormality, such as the analysis state, analysis time consumption, position information matching condition, etc., so that the relevant personnel can timely understand the problem. For example, the resource identification abnormal value is 0.9, which exceeds the benchmark value 0.8, at which time the resource consumption abnormal monitoring value can be directly set to 0.9, and abnormal prompt information such as “resource identification analysis abnormality, analysis state error, analysis time consumption too long, and position information matching failure” is generated and sent to the system management interface or the terminal device of the relevant personnel.

[0073] Through the above steps, from obtaining the parameters of the analysis device to determining each value, and then comparing the benchmark value and determining the resource consumption abnormal monitoring value, the monitoring of the identification abnormal state and the consumption abnormal state in the resource usage process is realized. This process can timely find the abnormal condition in the resource identification analysis process, and determine whether the resource consumption is abnormal, thereby providing an important abnormality discrimination basis for the data platform resource consumption monitoring system. In the whole process, the accuracy and timeliness of each link are crucial, for example, the real-time acquisition of the parameters of the analysis device can ensure the timeliness of the data, the reasonable setting of the weight and benchmark value of each parameter can improve the accuracy of the abnormality discrimination, and the accurate feature matching and abnormal prompt can provide strong support for the subsequent response adjustment.

[0074] Embodiment 4;

[0075] The process in which the response adjustment module determines the optimal response path by using the multi-dimensional resource abnormality response model is as follows:

[0076] The key level of the resource usage task and the historical feature verification result and historical abnormal monitoring value of the resource path need to be obtained. For example, there is a resource usage task in a certain data platform, which is to process the core transaction data of users. According to the priority attribute, data size attribute and access frequency attribute, it is determined that the key level is high. At the same time, the historical feature verification result of the resource path in the past period of time shows that the CPU usage and memory occupancy are close to the upper limit of the benchmark many times when the resource consumption feature set is compared with the benchmark in the peak period. The historical abnormal monitoring value shows that the resource identifier abnormal value has exceeded the benchmark value in some period, and the corresponding resource consumption abnormal monitoring value is also high.

[0077] A multi-dimensional resource abnormal response model is constructed, which takes reducing response resource occupation and shortening response time as optimization objectives. When constructing the model, how to integrate the two optimization objectives to form a comprehensive optimization system needs to be considered. For example, reducing response resource occupation can be achieved by reducing the use of CPU, memory and other resources, while shortening response time requires accelerating the transmission and processing speed of resources between nodes.

[0078] The key level, historical feature verification result and historical abnormal monitoring value are integrated to determine the comprehensive optimization objective of the multi-dimensional resource abnormal response model. The key level is taken as the first weight factor, the historical feature verification result is taken as the second weight factor, and the historical abnormal monitoring value is taken as the third weight factor. For example, for the resource usage task with high key level, the weight of the first weight factor can be set to 40%, because the task with high key level has higher requirements for resource response; the historical feature verification result shows that the resource consumption of the resource path is close to the upper limit of the benchmark in the peak period, indicating that the resource occupation needs to be focused on, and the weight of the second weight factor is set to 35%; the historical abnormal monitoring value shows that the path has appeared abnormal situation, and the weight of the third weight factor is set to 25%.

[0079] When integrating the optimization directions and weight factors, the two optimization directions of reducing response resource occupation and shortening response time need to be combined with the three weight factors. For example, for the optimization direction of reducing response resource occupation, in the case of high key level, the resource occupation needs to be more strictly controlled to ensure the normal operation of the core task; and in the case of resource consumption close to the upper limit of the benchmark in the historical feature verification result, the optimization of reducing resource occupation needs to be focused on. For the optimization direction of shortening response time, the task with high key level usually has higher requirements for response speed, and the abnormal situation in the historical abnormal monitoring may also be related to the too long response time, so the shortening of response time needs to be considered in optimization.

[0080] According to the comprehensive optimization goal, the optimal response path is determined. In the solving process, optimization algorithms such as genetic algorithm, particle swarm optimization algorithm, etc. can be used. Taking the particle swarm optimization algorithm as an example, each particle represents a candidate path, the position of the particle represents the parameters of the path, and the speed represents the adjustment direction and amplitude of the path. Through continuous iteration, the position and speed of the particle are adjusted to make the particle move in the direction of the optimal comprehensive optimization goal, and finally the optimal path that meets the comprehensive optimization goal is found.

[0081] For example, in the data platform, there are multiple candidate paths, the resource occupation of path 1 is relatively low, but the response time is long; the response time of path 2 is short, but the resource occupation is high; the resource occupation and response time of path 3 are relatively balanced. Through the particle swarm optimization algorithm, these candidate paths are evaluated and optimized, combined with the key level of high, the historical feature verification result and the historical abnormal monitoring value, and finally path 3 is determined as the optimal response path, because it achieves a good balance between resource occupation and response time, and can meet the requirements of key level high tasks on resource response, while considering historical characteristics and abnormal situations.

[0082] In the solving process, each candidate path needs to be scored according to the comprehensive optimization goal. The scoring method is to calculate the scores of reducing response resource occupation and shortening response time in two optimization directions, and the scores are weighted according to the weights of the weight factors. For example, the score of path 3 in reducing response resource occupation is 80 points, and the score in shortening response time is 75 points. According to the weights of key level 40%, historical feature verification result 35%, and historical abnormal monitoring value 25%, the comprehensive score is calculated as 80x40%+75x35%+(adjusting score considering historical abnormal situation)x25%, and a higher comprehensive score is finally obtained, so it is determined as the optimal response path.

[0083] Through the above steps, from obtaining relevant data to building a model, determining the comprehensive optimization goal, and then solving the optimal response path, the adjustment of the current resource path is realized. This process can consider the optimization goals of reducing response resource occupation and shortening response time according to the key level of resource use tasks and historical data, find the optimal response path, and improve the efficiency of resource use. In the whole process, accurate data support and reasonable parameter setting are needed for each link, such as accurate evaluation of key level, complete record of historical feature verification result and historical abnormal monitoring value, reasonable selection and parameter adjustment of optimization algorithm, etc., to ensure that the optimal response path determined can truly meet the needs of data platform resource consumption monitoring.

[0084] Example 5;

[0085] The system in this embodiment includes multiple expansion modules, and the implementation of each module is as follows. The resource anomaly early warning module is used to generate early warning information before resource consumption anomaly occurs. The specific process is to obtain historical resource consumption characteristic verification results and a current resource consumption characteristic set. The historical resource consumption characteristic verification results are records of comparison between the resource consumption characteristic set in the past period of time and the benchmark, and the current resource consumption characteristic set is the resource consumption characteristic value collected in real time. For example, in a certain data platform, the historical resource consumption characteristic verification results show that the CPU usage rate gradually increases from 3 pm to 5 pm every day in the past week, and approaches the upper limit of the benchmark; the current resource consumption characteristic set shows that the CPU usage rate has reached 70% of the benchmark value at 2 pm on the current day.

[0086] The trend of characteristic change in the preset early warning time window is integrated to determine the resource consumption anomaly early warning threshold. The preset early warning time window can be set according to actual needs, such as 1 hour or half a day. In the time window, the trend of change of the characteristic value in the historical characteristic verification results, such as the rate of increase or decrease, the fluctuation range, etc., is analyzed, and the change of the current resource consumption characteristic set is combined to determine an early warning threshold. For example, the preset early warning time window is 1 hour, the historical data shows that the CPU usage rate increases by 10% per hour on average in the same time period in the past week, and the current CPU usage rate is 70% of the benchmark value, so the early warning threshold can be set to 90% of the benchmark value, that is, the early warning is triggered when the CPU usage rate may reach 90% within 1 hour.

[0087] The current resource consumption characteristic set is compared with the resource consumption anomaly early warning threshold, and if the early warning condition is reached, resource consumption anomaly early warning information is generated and sent to the management terminal. For example, the current CPU usage rate increases from 70% to 85% within half an hour, and according to the trend, it is expected to exceed the early warning threshold of 90% within 1 hour, at which time the early warning information is generated, including the current resource consumption characteristic, the time when the anomaly is expected to be reached, the possible anomaly type, etc., and is sent to the management terminal, such as the computer or mobile phone of the system administrator, so that the management personnel can take timely measures, such as adjusting resource allocation or starting the standby node.

[0088] The resource data storage module is used to store key data information in the resource usage process. Specifically, based on the key level of the resource usage task, the data storage priority is divided. The resource usage task with high key level has high data storage priority, and needs to use more reliable storage medium and more frequent backup strategy; the task with low key level has low data storage priority, and can use ordinary storage medium. For example, resource usage task A is core business data processing, the key level is high, and the data storage priority is set to level one; task B is log data recording, the key level is low, and the storage priority is set to level three.

[0089] The resource identifier generated by the resource identifier generation module, the resource parameter obtained by the resource parameter acquisition module, and the resource consumption anomaly monitoring value determined by the anomaly determination module are stored in different storage media according to the data storage priority, forming a traceable resource data archive. The data of the first storage priority is stored in a high-performance solid-state hard disk array and is backed up in real time; the data of the second storage priority is stored in a general hard disk and is backed up once a day; the data of the third storage priority is stored in a cheap storage device and is backed up once a week. For example, the resource identifier, detailed resource parameters (such as CPU usage per second and memory usage) and anomaly monitoring values of task A are stored in the first storage medium; the resource identifier, resource parameters per hour and anomaly monitoring values of task B are stored in the third storage medium. When storing, a timestamp and related metadata are added to each data record for subsequent query and traceability.

[0090] The specific process of the response adjustment module adjusting the current resource path is to obtain the node load state, resource delay data and bandwidth occupancy rate of the current resource path. The node load state includes the CPU usage, memory usage and task queue length of each node; the resource delay data is the time delay of resource transmission between nodes; and the bandwidth occupancy rate is the proportion of network bandwidth occupied by the current path. For example, the current resource path includes node 1, node 2 and node 3, the CPU usage of node 1 is 80%, the memory usage is 900MB, and the task queue length is 10; the CPU usage of node 2 is 60%, the memory usage is 500MB, and the task queue length is 5; the CPU usage of node 3 is 70%, the memory usage is 600MB, and the task queue length is 8; the resource delay data shows that the delay from node 1 to node 2 is 50ms, and the delay from node 2 to node 3 is 30ms; and the bandwidth occupancy rate is 70%.

[0091] The adjustment priority of each candidate path is determined by integrating the key level, historical feature verification result and historical anomaly monitoring value. Tasks with high key level need to be adjusted in priority to ensure their normal operation; the historical feature verification result shows that the path often appears abnormal, and the adjustment priority is also high; the path with high historical anomaly monitoring value also needs to be adjusted in priority. For example, the key level of the resource usage task is high, the historical feature verification result shows that the resource consumption of the current path often approaches the upper limit of the benchmark during peak hours, and the historical anomaly monitoring value also shows that the path has appeared abnormal many times, so the adjustment priority of each candidate path is determined by comprehensive evaluation according to these factors, the adjustment priority of candidate path 1 is the highest, the adjustment priority of candidate path 2 is the second, and the adjustment priority of candidate path 3 is the lowest.

[0092] The load balancing degree and resource stability of the candidate paths are evaluated in turn according to the adjusted priority, and a path with the optimal load balancing degree and up-to-standard resource stability is selected as the optimal response path. The load balancing degree evaluates whether the load of each node in the candidate path is uniform, so as to avoid that some nodes have too high load and affect the overall performance. The resource stability evaluates whether the resource consumption of each node in the path is stable and whether it is easy to fluctuate. For example, the load balancing degree of each node of the candidate path 1 is 85 points (100 points in total), and the resource stability is 90 points; the load balancing degree of the candidate path 2 is 70 points, and the resource stability is 80 points; the load balancing degree of the candidate path 3 is 60 points, and the resource stability is 75 points. According to the adjusted priority, the candidate path 1 is evaluated first, and the load balancing degree and the resource stability of the candidate path 1 both reach the set standard (for example, the load balancing degree is not less than 80 points, and the resource stability is not less than 85 points), so the candidate path 1 is selected as the optimal response path.

[0093] The resource behavior verification module is used to verify the legality of the resource use behavior, and specifically acquires the resource identifier generated by the resource identifier generation module and extracts the verification field therein. The resource identifier contains a field specially used for verification, which is generated by a specific algorithm. For example, the resource identifier is "RS20250629001_Verify123", wherein "Verify123" is the verification field.

[0094] The verification field is compared with a preset legal identifier library, if the comparison is successful, the resource use behavior is marked as legal, if the comparison fails, the resource use behavior is marked as having a risk of forgery, and the risk information of forgery is fed back to the abnormality discrimination module. The preset legal identifier library stores all legal verification fields and their corresponding resource use task information. For example, the verification field "Verify123" exists in the legal identifier library and is consistent with the information of the resource use task, so the behavior is marked as legal; if the verification field does not exist in the legal identifier library or is inconsistent with the task information, it is marked as having a risk of forgery, and the risk information such as resource identifier, verification field, matching result, etc. is fed back to the abnormality discrimination module, so that the abnormality discrimination module further processes, such as triggering more stringent monitoring or preventing the resource use behavior.

[0095] Through the implementation of the above each expansion module, the early warning of resource consumption, the storage management of data, the adjustment of path and the verification of behavior are realized, the function of the data platform resource consumption monitoring system is further improved, and the reliability and security of the system are improved. In the implementation process, each link of each module needs accurate data processing and reasonable parameter setting to ensure the normal operation of the module and the effective play of the function. For example, the reasonable setting of the preset early warning time window and the early warning threshold in the resource abnormality early warning module directly affects the accuracy and timeliness of the early warning; the division of the storage priority and the selection of the storage medium in the resource data storage module are related to the safety and traceability of the data; the evaluation standard of each candidate path and the determination of the adjustment priority in the response adjustment module affect the selection of the optimal response path and the improvement of the resource use efficiency; the integrity of the legal identification library and the reliability of the verification algorithm in the resource behavior verification module determine the accuracy and security of the resource use behavior verification.

[0096] It should be noted that, in this text, relational terms such as first and second are used only to distinguish one entity or action from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0097] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A data platform resource consumption monitoring system, characterized in that, The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption behavior monitoring method and device. The application relates to a resource consumption 2. The data platform resource consumption monitoring system of claim 1, wherein, The consumption feature acquisition module obtains the resource consumption feature verification result in the following process: According to the resource parameters of the input end and the output end in the current resource node, determine the initial feature acquisition value; According to the resource parameters of each interval fixed period in the resource use process and the resource parameters before use, determine the intermediate feature acquisition value; According to the resource parameters of the output end of the current resource node and the input end of the next node, determine the end feature acquisition value; Integrate the initial feature acquisition value, the intermediate feature acquisition value and the end feature acquisition value to form the resource consumption feature set; Compare the resource consumption feature set with the resource consumption feature benchmark to form the resource consumption feature verification result.

3. The data platform resource consumption monitoring system of claim 1, wherein, The implementation process of the abnormality discrimination module to determine the resource consumption abnormality monitoring value includes: Obtain the analysis state, analysis time consumption and analysis data content of the resource identification analysis device in the resource use process; According to the analysis state, analysis position information and analysis data details in the analysis data content, determine the analysis state weight value, position information matching value and intermediate feature acquisition value respectively; Evaluate the analysis state weight value, analysis time consumption, position information matching value and intermediate feature acquisition value to determine the resource identification abnormality value; Compare the resource identification abnormality value with the identification abnormality benchmark value. If it does not exceed the identification abnormality benchmark value, perform feature matching on the real-time resource data in the resource use process and the original data before use to determine the resource consumption abnormality monitoring value; otherwise, determine the resource consumption abnormality monitoring value according to the resource identification abnormality value and feedback abnormality prompt.

4. The data platform resource consumption monitoring system of claim 1, wherein, The construction process of the comprehensive optimization target includes: Taking reducing response resource occupation as the first optimization direction and shortening response time as the second optimization direction; Taking the key level as the first weight factor, the historical feature verification result as the second weight factor, and the historical abnormality monitoring value as the third weight factor; Integrate each optimization direction and weight factor to form the comprehensive optimization target of the multi-dimensional resource abnormality response model.

5. The data platform resource consumption monitoring system of claim 1, wherein, It also includes a resource abnormality early warning module for generating early warning information before resource consumption abnormality occurs; specifically: obtain the historical resource consumption feature verification result and the current resource consumption feature set, integrate the feature change trend in the preset early warning time window, and determine the resource consumption abnormality early warning threshold; Compare the current resource consumption feature set with the resource consumption abnormality early warning threshold. If the early warning condition is met, generate resource consumption abnormality early warning information and send it to the management terminal.

6. The data platform resource consumption monitoring system of claim 1, wherein, It also includes a resource data storage module for storing key data information in the resource use process; specifically: based on the key level of the resource use task, divide the data storage priority; store the resource identification generated by the resource identification generation module, the resource parameters obtained by the consumption feature acquisition module, and the resource consumption abnormality monitoring value determined by the abnormality discrimination module to different storage media according to the data storage priority, forming a traceable resource data archive.

7. The data platform resource consumption monitoring system of claim 1, wherein, The application also comprises a resource behavior verification module for verifying the legitimacy of the resource use behavior, specifically: obtaining the resource identifier generated by the resource identifier generation module, extracting the verification field therein; comparing the verification field with a preset legitimate identifier library, marking the resource use behavior as legitimate if the comparison is successful, marking the resource use behavior as having a risk of forgery if the comparison fails, and feeding back the forgery risk information to the abnormality discrimination module.

Images