Big data automatic management platform capable of interactively displaying various indexes

Abstract

The invention relates to the technical field of big data, and discloses a big data automatic management platform capable of interactively displaying various indexes. Comprising an index database construction module, an index data model definition module, a data information extraction early warning module, a comprehensive data interaction display module, a management platform environment monitoring module, a platform operation risk judgment module and a big data platform management module. A time sequence database is adopted to process large-scale time sequence data, the data are stored in the database, data security is detected according to a data security detection mathematical model, and the data subjected to security detection are interactively displayed in different sections according to different classifications. A platform operation load index is calculated through a platform load monitoring mathematical model, the operation condition of the management platform is monitored, a big data platform management module carries out platform management on a big data platform according to early warning information and a risk judgment result, large-scale data can be processed in parallel, and high-performance data processing capacity is provided.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more specifically to a big data automation management platform that can interactively display various indicators. Background Art

[0002] With the popularization of technologies such as the Internet, the Internet of Things, and social media, the amount of global data has shown explosive growth. This data comes from various channels, including sensors, log files, social media, transaction records, etc., and is diverse, real-time, and massive. In order to meet the challenges of big data, data processing technology is constantly innovating. The emergence of technologies such as distributed computing, cloud computing, and in-memory computing has made large-scale data processing and analysis possible. At the same time, the development of data storage technologies such as data warehouses and data lakes has provided strong support for data storage and management.

[0003] In a highly competitive market environment, companies need to make decisions quickly and accurately to respond to market changes. However, there are inconsistent data definitions between different data sources and systems. This leads to errors in the definition of the same indicators during the data integration process, affecting the accuracy and reliability of data analysis. Traditional big data automation management platforms lack platform monitoring systems, resulting in the inability to grasp the platform's operating status and performance indicators in real time, affecting the judgment and work efficiency of operation and maintenance personnel. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a big data automation management platform that can interactively display various indicators to solve the problems existing in the above-mentioned background technology.

[0005] The present invention provides the following technical solution: a big data automated management platform capable of interactively displaying various indicators, comprising: an indicator database construction module, an indicator data model definition module, a data information extraction and early warning module, a comprehensive data interactive display module, a management platform environment monitoring module, a platform operation risk judgment module, and a big data platform management module;

[0006] The indicator database construction module uses a time series database to process large-scale time series data and stores the data in the database, and transmits the data to the indicator data model definition module;

[0007] The indicator data model definition module includes a model layer unit, a meta-model layer unit and a comprehensive model layer unit, obtains the indicator corresponding data information through unit processing, and transmits the indicator corresponding data information to the data information extraction and early warning module;

[0008] The data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information and displays early warnings for risk data;

[0009] The comprehensive data interactive display module interactively displays the data detected by the data information extraction and early warning module in different sections according to different categories;

[0010] The management platform environment monitoring module calculates the platform operation load index through the platform load monitoring mathematical model to monitor the operation status of the management platform, and transmits the platform operation load index to the platform operation risk judgment module;

[0011] The platform operation risk judgment module judges the platform operation risk based on the platform operation load index transmitted by the management platform environment monitoring module, and transmits the judgment result to the big data platform management module;

[0012] The big data platform management module receives the warning information from the data information extraction and warning module and the judgment results from the platform operation risk judgment module, and performs platform management on the big data platform.

[0013] Preferably, in the indicator data model definition module, the model layer unit defines indicators for time series data and transmits the defined indicator data information to the meta-model layer unit. The meta-model layer unit receives the indicator data information defined by the model layer unit, completes the extraction and sorting of the indicator data information according to the task priority, and transmits the sorted indicator data information extraction task to the comprehensive model layer unit. The comprehensive model layer unit receives the sorted indicator data information extraction task transmitted by the meta-model layer unit, and extracts the data information corresponding to the indicator through the comprehensive model layer.

[0014] Preferably, the content of the indicator definition includes defining the basic attributes of the indicator and the hierarchical relationship between the indicators. The basic attributes include the indicator name, indicator alias, indicator description, indicator caliber and indicator calculation standard. The hierarchical relationship includes parent indicators, child indicators, and related indicators, and an indicator database is formed based on the indicators.

[0015] Preferably, the calculation formula for the task priority is: Where α represents the task priority, t max Indicates the maximum waiting time for a task, t ex Indicates the waiting time of the task, and a, b, and c are calculation coefficients.

[0016] Preferably, the data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information and displays the specific content of the risk data early warning as follows:

[0017] Step S01: Construct a data set (x i ,y i ), which contains the label description value x for each data i And the characteristic description value y of each data i ,(X i , Y i ) represents a preset data training set, where i=1, 2, 3, ..., n;

[0018] Step S02: Calculate the similarity between the two data sets according to step S01. The calculation formula is: Where Pz represents the similarity between the two data sets;

[0019] Step S03: Compare the similarity Pz of the two data sets with the preset similarity threshold θ1. If the similarity Pz of the two data sets is greater than the preset similarity threshold θ1, the data information is judged to be safe and the feature description value is extracted. If the similarity Pz of the two data sets is less than or equal to the preset similarity threshold θ1, the data information is judged to be risky and the feature description value cannot be extracted, and a warning for the risk data is displayed.

[0020] Preferably, in the comprehensive data interactive display module, the extracted feature description values are displayed on the administrator interface through a visualization tool, and an interactive interface is provided to interactively display the data detected as safe in the data information extraction early warning module in different sections according to different categories, allowing administrators to view interactive data by clicking and querying.

[0021] Preferably, in the management platform environment monitoring module, the specific content of monitoring the operation status of the management platform by calculating the platform operation load index through the platform load monitoring mathematical model is as follows:

[0022] Step S01: Assuming that the number of servers included in the management platform is k, where k = 1, 2, 3, ..., K, calculate the bandwidth utilization of different servers using the following formula: where r k represents the bandwidth utilization of the kth server, v k represents the data transmission rate of the kth server, d k Indicates the bandwidth allocated to the k-th server;

[0023] Step S02: Calculate the average bandwidth utilization of the management platform according to step S01. The calculation formula is: in Indicates the average bandwidth utilization;

[0024] Step S03: Calculate the network bandwidth load during the operation of the management platform. The calculation formula is: Where lr represents the network bandwidth load during the operation of the management platform;

[0025] Step S04: Calculate the CPU usage during the operation of the management platform. The calculation formula is: Where cx represents the CPU usage during the operation of the management platform, t k1 represents the idle time of the kth server, t k Indicates the total running time of the kth server;

[0026] Step S05: Calculate the storage capacity load during the operation of the management platform. The calculation formula is: Where pz represents the storage capacity load during the operation of the management platform, p k1 represents the storage capacity used by the kth server, p k represents the total storage capacity of the kth server;

[0027] Step S06: Calculate the platform operation load index, the calculation formula is: Fz = log2(ω1×lr+ω2×cx+ω3×pz), where Fz represents the platform operation load index, ω1 represents the weight of the network bandwidth load, ω2 represents the weight of the CPU usage, and ω3 represents the weight of the storage capacity load.

[0028] Preferably, in the platform operation risk judgment module, the platform operation load index Fz is transmitted based on the management platform environment monitoring module, and the platform operation load index Fz is compared with the preset platform operation load threshold θ2. If the platform operation load index Fz is greater than the preset platform operation load threshold θ2, the platform operation is judged to be risk-free; if the platform operation load index Fz is less than or equal to the preset platform operation load threshold θ2, the platform operation is judged to be risky.

[0029] Preferably, the big data platform management module receives the warning information in the data information extraction warning module, performs preventive management on the risk data, receives the judgment results in the platform operation risk judgment module, and takes countermeasures, including increasing server resources, optimizing network configuration, and adjusting task scheduling strategies.

[0030] Technical effects and advantages of the present invention:

[0031] The present invention is provided with an indicator database construction module, an indicator data model definition module, a data information extraction and early warning module, a comprehensive data interactive display module, a management platform environment monitoring module, a platform operation risk judgment module and a big data platform management module. It uses a time series database to process large-scale time series data and stores the data in a database, detects data security based on a data security detection mathematical model, and interactively displays the detected security data in different sections according to different classifications, calculates the platform operation load index through a platform load monitoring mathematical model to monitor the operation status of the management platform, and the big data platform management module judges the results based on early warning information and risks. In short, a big data automation management platform that can interactively display various indicators processes large-scale data in parallel through data definition, uniformly manages and integrates data from different sources, provides high-performance data processing capabilities, provides comprehensive data support for enterprises, and helps to realize information sharing and intercommunication; and improves data processing efficiency based on priority sorting, and monitors the system's operation load in real time so that operation and maintenance personnel can promptly discover system performance bottlenecks and avoid system crashes or performance degradation due to excessive load. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of a big data automation management platform that can interactively display various indicators. DETAILED DESCRIPTION

[0033] The technical solutions of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the present invention. In addition, the forms of the various structures described in the following embodiments are merely examples. The present invention involves a big data automation management platform that can interactively display various indicators and is not limited to the various structures described in the following embodiments. All other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the present invention provides a big data automation management platform that can interactively display various indicators, including: an indicator database construction module, an indicator data model definition module, a data information extraction and early warning module, a comprehensive data interactive display module, a management platform environment monitoring module, a platform operation risk judgment module, and a big data platform management module;

[0035] The indicator database construction module is connected to the indicator data model definition module, the indicator data model definition module is connected to the data information extraction and early warning module and the management platform environment monitoring module, the data information extraction and early warning module is connected to the comprehensive data interactive display module and the big data platform management module, the management platform environment monitoring module is connected to the platform operation risk judgment module, the comprehensive data interactive display module is connected to the big data platform management module, and the platform operation risk judgment module is connected to the big data platform management module;

[0036] The indicator database construction module uses a time series database to process large-scale time series data and stores the data in the database, and transmits the data to the indicator data model definition module;

[0037] A time series database is a database system specifically designed for storing, processing, and querying time series data. Time series data refers to a series of data generated continuously over time. Each data point contains a timestamp for indexing, aggregation, and sampling.

[0038] The indicator data model definition module includes a model layer unit, a meta-model layer unit and a comprehensive model layer unit, obtains the indicator corresponding data information through unit processing, and transmits the indicator corresponding data information to the data information extraction and early warning module;

[0039] The data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information and displays early warnings for risk data;

[0040] The comprehensive data interactive display module interactively displays the data detected by the data information extraction and early warning module in different sections according to different categories;

[0041] The management platform environment monitoring module calculates the platform operation load index through the platform load monitoring mathematical model to monitor the operation status of the management platform, and transmits the platform operation load index to the platform operation risk judgment module;

[0042] The platform operation risk judgment module judges the platform operation risk based on the platform operation load index transmitted by the management platform environment monitoring module, and transmits the judgment result to the big data platform management module;

[0043] The big data platform management module receives the warning information from the data information extraction and warning module and the judgment results from the platform operation risk judgment module, and performs platform management on the big data platform.

[0044] In this embodiment, it should be specifically explained that in the indicator data model definition module, the model layer unit defines indicators for the time series data, and transmits the defined indicator data information to the meta-model layer unit. The meta-model layer unit receives the indicator data information defined by the model layer unit, completes the extraction and sorting of the indicator data information according to the task priority, and transmits the sorted indicator data information extraction task to the comprehensive model layer unit. The comprehensive model layer unit receives the sorted indicator data information extraction task transmitted by the meta-model layer unit, and extracts the corresponding data information of the indicator through the comprehensive model layer.

[0045] In this embodiment, it should be specifically explained that the content of the indicator definition includes defining the basic attributes of the indicator and the hierarchical relationship between the indicators. The basic attributes include the indicator name, indicator alias, indicator description, indicator caliber and indicator calculation standard. The hierarchical relationship includes parent indicators, child indicators, and related indicators, and an indicator database is formed based on the indicators.

[0046] In this embodiment, it should be specifically explained that the calculation formula for the task priority is: Where α represents the task priority, t max Indicates the maximum waiting time for a task, t ex Indicates the waiting time of the task, and a, b, and c are calculation coefficients.

[0047] In this embodiment, it should be specifically explained that the data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information, and displays the risk data early warning. The specific content is as follows:

[0048] Step S01: Construct a data set (x i ,y i ), which contains the label description value x for each data i And the characteristic description value y of each data i ,(X i , Y i ) represents a preset data training set, where i=1, 2, 3, ..., n;

[0049] Step S02: Calculate the similarity between the two data sets according to step S01. The calculation formula is: Where Pz represents the similarity between the two data sets;

[0050] Step S03: Compare the similarity Pz of the two data sets with the preset similarity threshold θ1. If the similarity Pz of the two data sets is greater than the preset similarity threshold θ1, the data information is judged to be safe and the feature description value is extracted. If the similarity Pz of the two data sets is less than or equal to the preset similarity threshold θ1, the data information is judged to be risky and the feature description value cannot be extracted, and a warning for the risk data is displayed.

[0051] In this embodiment, it should be specifically explained that in the comprehensive data interactive display module, the extracted feature description values are displayed on the administrator interface through a visualization tool, and an interactive interface is provided to interactively display the data detected as safe in the data information extraction and early warning module in different sections according to different categories, allowing administrators to view the interactive data by clicking and querying.

[0052] In this embodiment, it should be specifically explained that, in the management platform environment monitoring module, the specific contents of calculating the platform operation load index by using the platform load monitoring mathematical model to monitor the operation status of the management platform are as follows:

[0053] Step S01: Assuming that the number of servers included in the management platform is k, where k = 1, 2, 3, ..., K, calculate the bandwidth utilization of different servers using the following formula: where r k represents the bandwidth utilization of the kth server, v k represents the data transmission rate of the kth server, d k Indicates the bandwidth allocated to the k-th server;

[0054] Step S02: Calculate the average bandwidth utilization of the management platform according to step S01. The calculation formula is: in represents the average bandwidth utilization, r k represents the bandwidth utilization of the kth server;

[0055] Step S03: Calculate the network bandwidth load during the operation of the management platform. The calculation formula is: Where lr represents the network bandwidth load during the operation of the management platform. represents the average bandwidth utilization, r k represents the bandwidth utilization of the kth server;

[0056] Step S04: Calculate the CPU usage during the operation of the management platform. The calculation formula is: Where cx represents the CPU usage during the operation of the management platform, t k1 represents the idle time of the kth server, t k Indicates the total running time of the kth server;

[0057] Step S05: Calculate the storage capacity load during the operation of the management platform. The calculation formula is: Where pz represents the storage capacity load during the operation of the management platform, p k1 represents the storage capacity used by the kth server, p k represents the total storage capacity of the kth server;

[0058] Step S06: Calculate the platform operation load index, the calculation formula is: Fz = log2(ω1×lr+ω2×cx+ω3×pz), where Fz represents the platform operation load index, lr represents the network bandwidth load during the operation of the management platform, cx represents the CPU usage during the operation of the management platform, pz represents the storage capacity load during the operation of the management platform, ω1 represents the weight of the network bandwidth load, ω2 represents the weight of the CPU usage, and ω3 represents the weight of the storage capacity load.

[0059] In this embodiment, it should be specifically explained that in the platform operation risk judgment module, the platform operation load index Fz is transmitted based on the management platform environment monitoring module, and the platform operation load index Fz is compared with the preset platform operation load threshold θ2. If the platform operation load index Fz is greater than the preset platform operation load threshold θ2, the platform operation is judged to be risk-free; if the platform operation load index Fz is less than or equal to the preset platform operation load threshold θ2, the platform operation is judged to be risky.

[0060] In this embodiment, it needs to be specifically explained that in the big data platform management module, the warning information in the data information extraction warning module is received, risk data is prevented and managed, the judgment results in the platform operation risk judgment module are received, and response measures are taken, including increasing server resources, optimizing network configuration, adjusting task scheduling strategies, and ensuring that key tasks are executed first when tasks on the platform are reasonably scheduled to avoid task delays or failures due to insufficient resources, and according to the scheduling results of task scheduling, the computing resources, storage resources and network resources of the platform are optimized to improve resource utilization and reduce operating costs.

[0061] In this embodiment, it should be specifically explained that the difference between this embodiment and the prior art lies in that this embodiment is provided with an indicator database construction module, an indicator data model definition module, a data information extraction and early warning module, a comprehensive data interactive display module, a management platform environment monitoring module, a platform operation risk judgment module and a big data platform management module. A time series database is used to process large-scale time series data and store the data in a database. Data security is detected based on a data security detection mathematical model, and the detected security data is interactively displayed in different sections according to different classifications. The platform operation load index is calculated by a platform load monitoring mathematical model to monitor the operation status of the management platform. The big data platform management module is based on early warning information and risk judgment results. In short, a big data automation management platform that can interactively display various indicators processes large-scale data in parallel through data definition, uniformly manages and integrates data from different sources, provides high-performance data processing capabilities, provides comprehensive data support for enterprises, and helps to realize information sharing and interoperability; and improves data processing efficiency based on priority sorting. Real-time monitoring of the system's operating load enables operation and maintenance personnel to promptly discover system performance bottlenecks and avoid system crashes or performance degradation due to excessive load.

[0062] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0063] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A big data automation management platform that can interactively display various indicators, characterized by: include: Indicator database construction module, indicator data model definition module, data information extraction and early warning module, comprehensive data interactive display module, management platform environment monitoring module, platform operation risk judgment module and big data platform management module; The indicator database construction module uses a time series database to process large-scale time series data and stores the data in the database, and transmits the data to the indicator data model definition module; The indicator data model definition module includes a model layer unit, a meta-model layer unit and a comprehensive model layer unit, obtains the indicator corresponding data information through unit processing, and transmits the indicator corresponding data information to the data information extraction and early warning module; The data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information and displays early warnings for risk data; The comprehensive data interactive display module interactively displays the data detected by the data information extraction and early warning module in different sections according to different categories; The management platform environment monitoring module calculates the platform operation load index through the platform load monitoring mathematical model to monitor the operation status of the management platform, and transmits the platform operation load index to the platform operation risk judgment module; The platform operation risk judgment module judges the platform operation risk based on the platform operation load index transmitted by the management platform environment monitoring module, and transmits the judgment result to the big data platform management module; The big data platform management module receives the warning information from the data information extraction and warning module and the judgment results from the platform operation risk judgment module, and performs platform management on the big data platform.

2. A big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: In the indicator data model definition module, the model layer unit defines indicators for time series data and transmits the defined indicator data information to the meta-model layer unit. The meta-model layer unit receives the indicator data information defined by the model layer unit, completes the extraction and sorting of the indicator data information according to the task priority, and transmits the sorted indicator data information extraction task to the comprehensive model layer unit. The comprehensive model layer unit receives the sorted indicator data information extraction task transmitted by the meta-model layer unit, and extracts the data information corresponding to the indicator through the comprehensive model layer.

3. The big data automated management platform capable of interactively displaying various indicators according to claim 2, characterized in that: The content of the indicator definition includes defining the basic attributes of the indicator and the hierarchical relationship between indicators. The basic attributes include indicator name, indicator alias, indicator description, indicator caliber and indicator calculation standard. The hierarchical relationship includes parent indicator, child indicator, and related indicator, and an indicator database is formed based on the indicators.

4. The big data automated management platform capable of interactively displaying various indicators according to claim 2, characterized in that: The calculation formula for the task priority is: Where α represents the task priority, t max Indicates the maximum waiting time for a task, t ex Indicates the waiting time of the task, and a, b, and c are calculation coefficients.

5. The big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: The data information extraction and early warning module receives the indicator corresponding data information transmitted by the indicator data model definition module, detects data security through the data security detection mathematical model, extracts security information and displays the specific content of the risk data early warning as follows: Step S01: Construct a data set (x i ,y i ), which contains the label description value x for each data i And the characteristic description value y of each data i ,(X i , Y i ) represents a preset data training set, where i=1, 2, 3, ..., n; Step S02: Calculate the similarity between the two data sets according to step S01. The calculation formula is: Where Pz represents the similarity between the two data sets; Step S03: Compare the similarity Pz of the two data sets with the preset similarity threshold θ1. If the similarity Pz of the two data sets is greater than the preset similarity threshold θ1, the data information is judged to be safe and the feature description value is extracted. If the similarity Pz of the two data sets is less than or equal to the preset similarity threshold θ1, the data information is judged to be risky and the feature description value cannot be extracted, and a warning for the risk data is displayed.

6. The big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: In the comprehensive data interactive display module, the extracted feature description values are displayed on the administrator interface through a visualization tool, and an interactive interface is provided to interactively display the data detected as safe in the data information extraction and early warning module in different sections according to different categories, allowing administrators to view interactive data by clicking and querying.

7. The big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: In the management platform environment monitoring module, the specific contents of monitoring the operation status of the management platform by calculating the platform operation load index through the platform load monitoring mathematical model are as follows: Step S01: Assuming that the number of servers included in the management platform is k, where k = 1, 2, 3, ..., K, calculate the bandwidth utilization of different servers using the following formula: where r k represents the bandwidth utilization of the kth server, v k represents the data transmission rate of the kth server, d k Indicates the bandwidth allocated to the k-th server; Step S02: Calculate the average bandwidth utilization of the management platform according to step S01. The calculation formula is: in Indicates the average bandwidth utilization; Step S03: Calculate the network bandwidth load during the operation of the management platform. The calculation formula is: Where lr represents the network bandwidth load during the operation of the management platform; Step S04: Calculate the CPU usage during the operation of the management platform. The calculation formula is: Where cx represents the CPU usage during the operation of the management platform, t k1 represents the idle time of the kth server, t k Indicates the total running time of the kth server; Step S05: Calculate the storage capacity load during the operation of the management platform. The calculation formula is: Where pz represents the storage capacity load during the operation of the management platform, p k1 represents the storage capacity used by the kth server, p k represents the total storage capacity of the kth server; Step S06: Calculate the platform operation load index, the calculation formula is: Fz = log2(ω1×lr+ω2×cx+ω3×pz), where Fz represents the platform operation load index, ω1 represents the weight of the network bandwidth load, ω2 represents the weight of the CPU usage, and ω3 represents the weight of the storage capacity load.

8. The big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: In the platform operation risk judgment module, the platform operation load index Fz is transmitted based on the management platform environment monitoring module, and the platform operation load index Fz is compared with the preset platform operation load threshold θ2. If the platform operation load index Fz is greater than the preset platform operation load threshold θ2, the platform operation is judged to be risk-free. If the platform operation load index Fz is less than or equal to the preset platform operation load threshold θ2, the platform operation is judged to be risky.

9. The big data automated management platform capable of interactively displaying various indicators according to claim 1, characterized in that: The big data platform management module receives the warning information from the data information extraction warning module, performs preventive management on risk data, receives the judgment results from the platform operation risk judgment module, and takes response measures, including increasing server resources, optimizing network configuration, and adjusting task scheduling strategies.

Images