An AI-based big data platform operation and maintenance management system
Through data feature division and machine learning model prediction based on artificial intelligence, abnormal parameters are identified and optimization instructions are formulated, the problems of data chaos and loss in the operation and maintenance management of big data platform are solved, and the accuracy and efficiency of operation and maintenance management are improved, ensuring that The security and stability of the platform.
Patent Information
- Application Number
- CN202411171637.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-26
AI Technical Summary
When the existing big data platform operation and maintenance management system analyzes massive data, it leads to data confusion or loss, and cannot be effectively optimized, reducing operation and maintenance management efficiency and security stability.
The operation and maintenance management system of big data platform based on artificial intelligence is adopted to predict operation and maintenance status through data feature division and machine learning model, identify abnormal parameters and formulate optimization instructions to achieve independent management and optimization of data sets.
It effectively avoids data chaos and loss, improves the accuracy and efficiency of operation and maintenance management, and ensures the stable state and security stability of the big data platform.
Smart Images

Figure CN118673396B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and more specifically, to an artificial intelligence-based big data platform operation and maintenance management system. Background Art
[0002] A big data platform refers to a platform that can manage data sets captured, managed and processed within a certain time frame using conventional software tools. By collecting, analyzing and processing massive amounts of operation and maintenance data, the big data platform provides operation and maintenance personnel with comprehensive system monitoring, performance analysis, fault warning and automatic repair functions. Due to the variety of types of big data stored in the big data platform, the difficulty of operation and maintenance management increases, and data confusion or loss is prone to occur. In order to improve the security and stability of the operation and maintenance management of the big data platform, it is necessary to combine artificial intelligence technology to assist in operation and maintenance management.
[0003] The patent application with reference publication number CN109933705A discloses a big data platform operation and maintenance management system, including: a data collection module, a data analysis module and a data classification module; the data collection module is used to collect user data; the data analysis module is used to compare and analyze the collected user data to obtain user behavior data through comparison and analysis; the data classification module is used to classify the user behavior data obtained through comparison and analysis; it can push commercial information associated with the user behavior data to the user based on the user behavior data after deleting the data with low closeness to the user behavior data, and can realize pushing commercial information associated with the user behavior data to the user, the commercial information has high accuracy, meets the requirements of the user behavior data, and improves the success rate of commercial promotion;
[0004] The prior art has the following deficiencies:
[0005] The existing big data platform operation and maintenance management system achieves the overall operation and maintenance management effect of big data by conducting overall analysis and comparison on the massive big data in the database. When massive and multi-type big data are analyzed and processed as a whole, different types of data are easily crossed, resulting in confusion or even loss of some data, which in turn causes the operation and maintenance management status of the big data platform to be poor. At the same time, it is impossible to formulate targeted optimization instructions based on the operation and maintenance management status, resulting in low efficiency of the operation and maintenance management of the big data platform, and reducing the security and stability of the operation and maintenance management of the big data platform.
[0006] In view of this, the present invention proposes an artificial intelligence-based big data platform operation and maintenance management system to solve the above problems. Summary of the invention
[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: an artificial intelligence-based big data platform operation and maintenance management system, applied to a big data platform, comprising:
[0008] A set partitioning module is used to extract data features of target data and partition the target data into data sets based on the data features, wherein the data features include type features and attribute features;
[0009] A data collection module is used to collect historical training data of a data set. The historical training data includes comprehensive operation and maintenance parameters and operation and maintenance status values. The comprehensive operation and maintenance parameters include capacity occupancy rate, data encryption ratio, unit processing rate, and abnormal access frequency.
[0010] Model training module, used to train a machine learning model that predicts operation and maintenance status values based on historical training data;
[0011] The prediction and judgment module is used to collect real-time comprehensive operation and maintenance parameters, input them into the trained machine learning model, predict the real-time operation and maintenance status value, and determine whether there are abnormal sets in the data set;
[0012] The instruction formulation module is used to compare the comprehensive operation and maintenance parameters with the corresponding standard values, identify abnormal parameters, and formulate data optimization instructions based on the abnormal parameters.
[0013] Furthermore, the method for extracting type features and attribute features includes:
[0014] Through natural language processing technology, the content semantics of i target data in the database are identified one by one to obtain i semantic words;
[0015] Mark the first part corresponding to the text and the second part corresponding to the number in the semantic word, and establish a split point between the first part and the second part;
[0016] Split i semantic words from the split point through word segmentation technology to obtain i content bits and i type bits, and record the characters in the type bits to obtain i type features;
[0017] The attributes of i original data are queried through the data attribute table, and the text parts in the i attributes are recorded as attribute features to obtain i attribute features.
[0018] Furthermore, the data set partitioning method includes:
[0019] Aggregate the target data with the same type of features, query the storage time of the target data, and arrange the target data in the order of the storage time to generate u data columns;
[0020] Aggregate the target data with the same attribute characteristics in the data column to obtain data sets, with The target data in the database is divided into Subsets;
[0021] Will The type features and attribute features corresponding to the subsets are combined one by one to obtain Collection notes;
[0022] Respectively Create a comment box at the end of each subset and Import the corresponding Collection notes, get A data set.
[0023] Furthermore, the method for obtaining the data encryption ratio includes:
[0024] At time T1, Randomly grab an equal amount of target data from each data set, record them as first detection data, and count the number of first detection data;
[0025] At time T2, Randomly grab an equal amount of target data from each data set, record them as second detection data, and count the number of second detection data;
[0026] The encryption management system queries the encryption status of the first detection data and the second detection data one by one, and marks the target data whose encryption status is encrypted, to obtain x first encrypted data and x second encrypted data;
[0027] Query the storage time and encryption time of the x first encrypted data and the x second encrypted data one by one through the timestamp, and record the time from the storage time to the encryption time as the encryption time, to obtain x first encryption time and x second encryption time;
[0028] The first encrypted data and the second encrypted data whose first encryption duration and the second encryption duration are less than the preset duration threshold are respectively recorded as the first available data and the second available data, and the number of the first available data and the number of the second available data are counted;
[0029] Compare the amount of the first available data with the amount of the first detected data to obtain The first proportion;
[0030] Compare the amount of the second available data with the amount of the second detected data to obtain The second percentage;
[0031] Will The first proportions are respectively Add the second proportions and find the average, and we get The percentage of data encryption.
[0032] Furthermore, the method for obtaining the unit processing rate includes:
[0033] Respectively from Mark an equal amount of target data in the data set and record it as the data to be marked;
[0034] The keywords of the data to be labeled are identified one by one through natural language processing technology, and the keywords are marked on the data to be labeled to obtain the processed data and count the total amount of the processed data;
[0035] Importing all processed data into the data processor at the same time, and after a preset processing time, receiving and recording the processed data exported by the data processor;
[0036] Record the exported processed data without tags as processed data, and count the number of processed data;
[0037] Will The number of processed data sets and Compare the total amount of processed data of the data sets to obtain Sub-processing rate;
[0038] Query through system monitoring tools The fixed delay time of each data set is set, and the preset processing time is compared with After subtracting the fixed delay time, we get Effective processing time;
[0039] Will The processing rate of each sub- Compare the effective processing time and obtain Unit processing rate.
[0040] Furthermore, the method for obtaining abnormal access frequency includes:
[0041] Query one by one through the access management system The time when the target data in the data set first appears in the access record is obtained an initial moment;
[0042] When no access records appear within the preset access period, The time when the last access record appears in the data set is recorded as the end time, and the an ending moment;
[0043] Will Initial time to The time between the end times is recorded as the effective access time, and the The effective access time;
[0044] Mark them one by one Collect all access records of a data set within the effective access time, and query the access status value of the access record one by one;
[0045] The access records with access status values of unauthorized are recorded as abnormal accesses, and the number of abnormal accesses is counted to obtain outlier values;
[0046] Will The abnormal value is Compare the effective access time and obtain Abnormal access frequency.
[0047] Furthermore, the training method of the machine learning model for predicting the operation and maintenance status value includes:
[0048] Convert the comprehensive operation and maintenance parameters into a corresponding set of feature vectors, convert the operation and maintenance status values into digits, convert the operation and maintenance safety status into 0, and convert the operation and maintenance abnormal status into 1;
[0049] The feature vector is used as the input of the machine learning model, and the operation and maintenance status value corresponding to each set of comprehensive operation and maintenance parameters is used as the output of the machine learning model. The operation and maintenance status value is used as the prediction target, and minimizing the sum of prediction errors of all training data is used as the training target. The machine learning model is trained until the sum of prediction errors converges, and the training is stopped to obtain a machine learning model that can predict the operation and maintenance status value.
[0050] Furthermore, the method for determining whether there is an abnormal set in the data set includes:
[0051] When the output of the machine learning model is 0, the real-time operation and maintenance status value is the operation and maintenance safety status, and it is determined that there is no abnormal set in the data set;
[0052] When the output of the machine learning model is 1, the real-time operation and maintenance status value is an abnormal operation and maintenance status, and it is determined that there is an abnormal set in the data set.
[0053] Furthermore, the method for identifying abnormal parameters includes:
[0054] Compare the capacity occupancy rate with the occupancy standard value, and when the capacity occupancy rate is greater than the occupancy standard value, identify the capacity occupancy rate as an abnormal parameter;
[0055] Compare the data encryption ratio with the standard ratio value, and when the data encryption ratio is less than the standard ratio value, identify the data encryption ratio as an abnormal parameter;
[0056] Compare the unit processing rate with the rate standard value, and when the unit processing rate is less than the rate standard value, identify the unit processing rate as an abnormal parameter;
[0057] The abnormal access frequency is compared with the frequency standard value, and when the abnormal access frequency is greater than the frequency standard value, the abnormal access frequency is identified as an abnormal parameter.
[0058] Further, the data optimization instructions include instructions for reducing capacity occupancy, instructions for increasing data encryption ratio, instructions for increasing unit processing rate, and instructions for reducing abnormal access frequency;
[0059] The method for formulating the instruction to reduce the capacity occupancy rate, the instruction to increase the data encryption rate, the instruction to increase the unit processing rate, and the instruction to reduce the abnormal access frequency includes:
[0060] When the abnormal parameter is the capacity occupancy rate, an instruction to reduce the capacity occupancy rate is formulated;
[0061] When the abnormal parameter is the data encryption ratio, an instruction to increase the data encryption ratio is formulated;
[0062] When the abnormal parameter is the unit processing rate, an instruction to increase the unit processing rate is formulated;
[0063] When the abnormal parameter is an abnormal access frequency, an instruction to reduce the abnormal access frequency is formulated.
[0064] The technical effects and advantages of the big data platform operation and maintenance management system based on artificial intelligence of the present invention are as follows:
[0065] The present invention extracts data features of target data, and based on the data features, divides the target data into data sets, collects historical training data of the data sets, the historical training data includes comprehensive operation and maintenance parameters and operation and maintenance status values, trains a machine learning model that predicts the operation and maintenance status values based on the historical training data, collects real-time comprehensive operation and maintenance parameters, inputs them into the trained machine learning model, predicts the real-time operation and maintenance status values, determines whether there is an abnormal set in the data set, compares the comprehensive operation and maintenance parameters with the corresponding standard values, identifies the abnormal parameters, and formulates data optimization instructions based on the abnormal parameters; compared with the prior art, the present invention can divide the massive target data in the big data platform into multiple independent data sets , achieving the classification effect of breaking down massive data into small pieces, reducing the computational burden of each data set, and also avoiding the phenomenon of data confusion or even loss when a large number of different types of data are mixed together. By collecting comprehensive operation and maintenance parameters and combining machine learning models, it can accurately predict the real-time operation and maintenance status value of the data set, and quickly and accurately judge the operation and maintenance management quality of each data set, thereby playing an accurate predictive effect on the operation and maintenance management of the big data platform, and can formulate corresponding data optimization instructions, and perform targeted data optimization processing on the data in the big data platform, ensuring that the data in the big data platform can remain in a stable state, thereby improving the security and stability of the operation and maintenance management of the big data platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 A schematic diagram of an artificial intelligence-based big data platform operation and maintenance management system provided in Example 1 of the present invention;
[0067] Figure 2 A flowchart of a big data platform operation and maintenance management method based on artificial intelligence provided in Example 2 of the present invention;
[0068] Figure 3 A schematic diagram of a structural diagram of an electronic device provided in Embodiment 3 of the present invention;
[0069] Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided in Example 4 of the present invention. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0071] Example 1: Please refer to Figure 1As shown, the big data platform operation and maintenance management system based on artificial intelligence described in this embodiment is applied to the big data platform, including:
[0072] A set partitioning module extracts data features of target data and partitions the target data into data sets based on the data features;
[0073] Target data refers to the big data of various types and attributes recorded and stored in the database within a specific period of time without any processing. It is used to directly represent the original big data and serve as the original sample of the big data stored in the database. Since the types and attributes of these target data are not uniform, the meanings represented by different target data are not consistent. In order to accurately identify the target data, it is necessary to extract the true meaning of the target data, that is, the data features;
[0074] Data features include type features and attribute features; type features are used to represent the data type of target data, and attribute features are used to represent the data attributes of target data;
[0075] The extraction methods of type features and attribute features include:
[0076] Through natural language processing technology, the content semantics of i target data in the database are identified one by one to obtain i semantic words; content semantics is used to express the true meaning of the target data, so that complex and lengthy data can be simplified and accurately expressed;
[0077] Mark the first part corresponding to the text and the second part corresponding to the number in the semantic word, and establish a split point between the first part and the second part; the split point is used to indicate the split position of different parts in the semantic word, that is, the text and numbers in the semantic word can be effectively distinguished, so as to accurately identify the text and numbers in the semantic word;
[0078] Split i semantic words from the split point through word segmentation technology to obtain i content bits and i type bits, and record the characters in the type bits to obtain i type features;
[0079] The attributes of i original data are queried through the data attribute table, and the text parts in the i attributes are recorded as attribute features to obtain i attribute features.
[0080] After the type features and attribute features are extracted, the target data needs to be classified and distinguished according to the type features and attribute features, so that the target data with the same type features and attribute features can be divided into the same data set, so that the massive target data can be classified into small pieces, avoiding the high burden brought by the overall analysis and calculation of the massive target data;
[0081] Methods for partitioning data sets include:
[0082] Aggregate the target data with the same type of features, query the storage time of the target data, and arrange the target data in the order of the storage time to generate u data columns;
[0083] Aggregate the target data with the same attribute characteristics in the data column to obtain data sets, with The target data in the database is divided into A subset is the smallest unit that constitutes a data set and is used to summarize and represent the target data after preliminary screening.
[0084] Will The type features and attribute features corresponding to the subsets are combined one by one to obtain Collection notes;
[0085] Respectively Create a comment box at the end of each subset and Import the corresponding Collection notes, get A data set; the set note is used to represent the combination of type characteristics and attribute characteristics represented in the data set, that is, to accurately reflect the specific type and attribute of the target data in each data set, and the note box is used to provide a framework for importing the set note to ensure the integrity of the data set;
[0086] It should be noted that after the data set is obtained, the meaning of the target data in the data set itself has not changed. The target data of different types and attributes are only classified. Therefore, the meaning represented by the target data in the data set is consistent with that before the division, thereby avoiding changes in the target data after division.
[0087] The historical data collection module collects historical training data of the data set, and the historical training data includes comprehensive operation and maintenance parameters and operation and maintenance status values; the comprehensive operation and maintenance parameters include capacity occupancy rate, data encryption ratio, unit processing rate and abnormal access frequency;
[0088] Comprehensive operation and maintenance parameters refer to various parameters of the target data contained in the data set that will have a positive or negative impact on the data set in daily operation and maintenance management, so as to comprehensively represent the operation and maintenance stability of the target data in the data set and serve as the basic data for subsequent quality analysis of the operation and maintenance management of the big data platform and formulation of instruction measures;
[0089] Comprehensive operation and maintenance parameters include capacity utilization, data encryption ratio, unit processing rate, and abnormal access frequency;
[0090] The capacity utilization rate refers to the ratio between the space occupied by the target data in a data set and the total space of the data set. It can be used to numerically represent the memory usage of the target data in the data set at the current moment. The larger the capacity utilization rate, the larger the space occupied by the target data in the data set, and the smaller the space in the data set that can provide backup and cache for the target data. At this time, the operation and maintenance management status of the big data platform is worse. The capacity utilization rate is monitored and queried through system monitoring tools.
[0091] The data encryption ratio refers to the ratio of the number of encrypted target data in the data set to the total amount of target data, which can be used to numerically represent the number of encrypted target data. The larger the data encryption ratio, the more encrypted target data in the data set, and the better the operation and maintenance management status of the big data platform.
[0092] Methods for obtaining the data encryption ratio include:
[0093] At time T1, Randomly grab an equal amount of target data from each data set, record them as first detection data, and count the number of first detection data;
[0094] At time T2, An equal amount of target data is randomly captured from each data set, recorded as the second detection data, and the number of the second detection data is counted; time T2 is the next time after time T1, and the time between time T2 and time T1 is sufficient for the target data in the data set to perform at least one complete data encryption operation, so as to ensure that the data counted at time T2 can be encrypted;
[0095] The encryption management system queries the encryption status of the first detection data and the second detection data one by one, and marks the target data whose encryption status is encrypted, to obtain x first encrypted data and x second encrypted data;
[0096] Query the storage time and encryption time of the x first encrypted data and the x second encrypted data one by one through the timestamp, and record the time from the storage time to the encryption time as the encryption time, to obtain x first encryption time and x second encryption time;
[0097] The first encrypted data and the second encrypted data whose first encryption duration and the second encryption duration are less than a preset duration threshold are respectively recorded as first available data and second available data, and the number of the first available data and the number of the second available data are counted; the preset duration threshold refers to the maximum duration required for data encryption of the target data marked as available data, so that the duration used for each data encrypted can be limited;
[0098] Compare the amount of the first available data with the amount of the first detected data to obtain The first proportion;
[0099] The expression of the first proportion is:
[0100] ;
[0101] In the formula, For the The first percentage of the data set, No. The number of first available data for a data set, For the The number of first detection data of a data set;
[0102] Compare the amount of the second available data with the amount of the second detected data to obtain The second percentage;
[0103] The expression of the second ratio is:
[0104] ;
[0105] In the formula, For the The second percentage of the data set, No. The number of second available data for a data set, For the the number of second detection data of a data set;
[0106] Will The first proportions are respectively Add the second proportions and find the average, and we get The percentage of data encryption;
[0107] The expression of data encryption ratio is:
[0108] ;
[0109] In the formula, For the The percentage of data encrypted in a data set.
[0110] The unit processing rate refers to the speed of change of the ratio between the target data processed by the data set and the total amount of target data in a unit time. It can be used to numerically represent the current operating efficiency of the data set. The faster the unit processing rate is, the faster the data set processes the target data in a unit time, and the better the operation and maintenance management status of the big data platform is.
[0111] Methods for obtaining unit processing rate include:
[0112] Respectively from Mark an equal amount of target data in the data set and record it as the data to be marked;
[0113] The keywords of the data to be labeled are identified one by one through natural language processing technology, and the keywords are marked on the data to be labeled to obtain the processed data and count the total amount of the processed data;
[0114] All processed data are imported into the data processor at the same time, and after a preset processing time, the processed data exported by the data processor is received and recorded; the preset processing time refers to the minimum time taken by the data processor to complete at least one complete data processing, which can ensure that the processed data exported by the data processor can have enough time for data processing, thereby avoiding the problem that the data processing cannot be completed due to insufficient data processing time; the preset processing time is obtained by collecting the minimum value of the time taken by a large amount of historical processing data to complete at least one complete data processing, and then calculating the average value;
[0115] Record the exported processed data without tags as processed data, and count the number of processed data;
[0116] Will The number of processed data sets and Compare the total amount of processed data of the data sets to obtain Sub-processing rate;
[0117] The expression of sub-processing rate is:
[0118] ;
[0119] In the formula, For the The sub-processing rate of a data set, For the The number of processed data sets, For the The total amount of processed data for a data set;
[0120] Query through system monitoring tools The fixed delay time of each data set is set, and the preset processing time is compared with After subtracting the fixed delay time, we get Effective processing time;
[0121] The expression of effective processing time is:
[0122] ;
[0123] In the formula, For the The effective processing time, is the preset processing time. For the The fixed delay length of each data set;
[0124] Will The processing rate of each sub- Compare the effective processing time and obtain Unit processing rate;
[0125] The expression of unit processing rate is:
[0126] ;
[0127] In the formula, For the The unit processing rate of a data set.
[0128] Abnormal access frequency refers to the number of times that unauthorized access records occur to the target data in the data set per unit time. The greater the abnormal access frequency, the more times that unauthorized access records occur to the target data in the data set per unit time, and the worse the operation and maintenance management status of the big data platform.
[0129] Methods for obtaining abnormal access frequency include:
[0130] Query one by one through the access management system The time when the target data in the data set first appears in the access record is obtained an initial moment;
[0131] When no access records appear within the preset access period, The time when the last access record appears in the data set is recorded as the end time, and the The preset access cycle refers to the maximum time length corresponding to two adjacent access records when they are identified and recorded, so as to provide a reasonable time range for the screening of access records in the data set and effectively reduce the number of access records collected; the preset access cycle is obtained by collecting the maximum time length corresponding to a large number of adjacent access records in history when they are identified and recorded, and then calculating the average value;
[0132] Will Initial time to The time between the end times is recorded as the effective access time, and the The effective access time;
[0133] Mark them one by one Collect all access records within the effective access time of a data set, and query the access status value of the access record one by one; the access status value is used to indicate whether the access record is authorized or not, and the access status value includes authorized status and unauthorized status. The authorized status indicates that the access record has been reasonably and correctly authorized, and the unauthorized status indicates that the access record has not been reasonably and correctly authorized;
[0134] The access records with access status values of unauthorized are recorded as abnormal accesses, and the number of abnormal accesses is counted to obtain outlier values;
[0135] Will The abnormal value is Compare the effective access time and obtain Abnormal access frequency;
[0136] The expression of abnormal access frequency is:
[0137] ;
[0138] In the formula, For the The abnormal access frequency of the data set, For the The outlier value of a data set, For the The effective access duration of a data set.
[0139] The operation and maintenance status value is used to represent the operation and maintenance management security corresponding to the real-time type and attributes of the target data in the data set. The operation and maintenance status value includes the operation and maintenance safety status and the operation and maintenance abnormality status. The operation and maintenance safety status is used to represent that the data set is in a safe operation and maintenance management state, and the operation and maintenance abnormality status is used to represent that the data set is in an abnormal operation and maintenance management state. The operation and maintenance status value is obtained by collecting the capacity occupancy rate, data encryption ratio, unit processing rate and abnormal access frequency corresponding to the operation and maintenance safety status and the operation and maintenance abnormality status.
[0140] Model training module, which trains a machine learning model to predict operation and maintenance status values based on historical training data;
[0141] After obtaining the historical training data, a machine learning model that can predict the operation and maintenance status value can be trained based on the historical training data, so that the machine learning model can be used as a basis for subsequent prediction of the operation and maintenance status of the data set;
[0142] The training methods of the machine learning model for predicting the operation and maintenance status value include:
[0143] Convert the comprehensive operation and maintenance parameters into a corresponding set of feature vectors; convert the operation and maintenance status values into digital values, for example, convert the operation and maintenance safety status into 0, and convert the operation and maintenance abnormal status into 1;
[0144] The feature vector is used as the input of the machine learning model, and the operation and maintenance status value corresponding to each set of comprehensive operation and maintenance parameters is used as the output of the machine learning model. The operation and maintenance status value is used as the prediction target, and minimizing the sum of prediction errors of all training data is used as the training target. The machine learning model is trained until the sum of prediction errors converges, and the training is stopped to obtain a machine learning model that can predict the operation and maintenance status value.
[0145] Exemplarily, the machine learning model is any one of a CNN neural network model or AlexNet;
[0146] The calculation formula for the prediction error is:
[0147] ;
[0148] In the formula, is the prediction error, is the group number of the eigenvector; For the The predicted state value corresponding to the group feature vector, For the The actual state value corresponding to the group training data;
[0149] In the machine learning model, the feature vector is the comprehensive operation and maintenance parameter, and the state value is the operation and maintenance state value;
[0150] Other model parameters of the machine learning model, target loss value, optimization algorithm, training set test set validation set ratio, and loss function optimization are all achieved through actual engineering implementation and continuous experimental tuning.
[0151] The prediction and judgment module collects real-time comprehensive operation and maintenance parameters, inputs them into the trained machine learning model, predicts the real-time operation and maintenance status value, and determines whether there are abnormal sets in the data set;
[0152] By using the above-mentioned method for obtaining historical comprehensive operation and maintenance parameters, real-time comprehensive operation and maintenance parameters are obtained, and the real-time comprehensive operation and maintenance parameters are input into the trained machine learning model, so as to predict the real-time operation and maintenance status value of the data set;
[0153] By analyzing the real-time operation and maintenance status value, it is possible to determine whether each data set has a poor operation and maintenance management status, and mark the data set with a poor operation and maintenance management status as an abnormal set, so that the abnormal set can be used as the object of subsequent big data platform optimization processing;
[0154] Methods for determining whether there are abnormal sets in a data set include:
[0155] When the output of the machine learning model is 0, it indicates that the data set is in a safe state, and the real-time operation and maintenance state value is an operation and maintenance safe state. At this time, it is determined that there is no abnormal set in the data set;
[0156] When the output of the machine learning model is 1, it indicates that the data set is in an abnormal state, and the real-time operation and maintenance status value is an abnormal operation and maintenance state. At this time, it is determined that there is an abnormal set in the data set.
[0157] The instruction formulation module compares the comprehensive operation and maintenance parameters with the corresponding standard values, identifies abnormal parameters, and formulates data optimization instructions based on the abnormal parameters; the data optimization instructions include instructions for reducing capacity occupancy, increasing data encryption ratio, increasing unit processing rate, and reducing abnormal access frequency;
[0158] Abnormal parameters refer to the comprehensive operation and maintenance parameters that cause the operation and maintenance status value of the data set to be in an abnormal operation and maintenance state and exceed the corresponding safety value, so that the abnormal parameters can be used as the object of subsequent optimization of the target data in the data set, and improve the operation and maintenance management status of the data set, thereby improving the safety and stability of the operation and maintenance management of the big data platform;
[0159] Methods for identifying abnormal parameters include:
[0160] Compare the capacity occupancy rate with the occupancy standard value; the occupancy standard value refers to the maximum value that the capacity occupancy rate can reach when the data set is not identified as an abnormal set, which can limit the maximum value of the capacity occupancy rate under normal circumstances and serve as a numerical basis for distinguishing the size of the capacity occupancy rate; the occupancy standard value is obtained by collecting a large number of historical maximum values corresponding to the capacity occupancy rate when it is not identified as an abnormal set, and then calculating their average value;
[0161] When the capacity occupancy rate is greater than the occupancy standard value, it means that the real-time capacity occupancy rate exceeds the normal value range. At this time, the capacity occupancy rate will have a greater negative impact on the data set, and the capacity occupancy rate is identified as an abnormal parameter;
[0162] Compare the data encryption ratio with the ratio standard value; the ratio standard value refers to the minimum value that the data encryption ratio can reach when the data set is not identified as an abnormal set, which can limit the minimum value of the data encryption ratio under normal circumstances and serve as a numerical basis for distinguishing the size of the data encryption ratio; the ratio standard value is obtained by collecting a large number of historical minimum values corresponding to the data encryption ratio when the data set is not identified as an abnormal set, and then calculating the average value;
[0163] When the data encryption ratio is less than the standard value, it means that the real-time data encryption ratio exceeds the normal value range. At this time, the data encryption ratio will have a greater negative impact on the data set, and the data encryption ratio will be identified as an abnormal parameter;
[0164] Compare the unit processing rate with the rate standard value; the rate standard value refers to the minimum value that the unit processing rate can reach when the data set is not identified as an abnormal set, which can limit the minimum value of the unit processing rate under normal circumstances and serve as a numerical basis for distinguishing the size of the unit processing rate; the rate standard value is obtained by collecting a large number of historical minimum values corresponding to the unit processing rate when it is not identified as an abnormal set, and then calculating their average value;
[0165] When the unit processing rate is less than the rate standard value, it means that the real-time unit processing rate exceeds the normal value range. At this time, the unit processing rate will have a greater negative impact on the data set, and the unit processing rate will be identified as an abnormal parameter;
[0166] Compare the abnormal access frequency with the frequency standard value; the frequency standard value refers to the maximum value that the abnormal access frequency can reach when the data set is not identified as an abnormal set, which can limit the maximum value of the abnormal access frequency under normal circumstances and serve as a numerical basis for distinguishing the size of the abnormal access frequency; the frequency standard value is obtained by collecting a large number of historical maximum values corresponding to the abnormal access frequency when it is not identified as an abnormal set, and then calculating the average value;
[0167] When the abnormal access frequency is greater than the frequency standard value, it means that the real-time abnormal access frequency exceeds the normal value range. At this time, the abnormal access frequency will have a greater negative impact on the data set, and the abnormal access frequency will be identified as an abnormal parameter.
[0168] Data optimization instructions refer to instruction measures that can optimize and improve target data within abnormal parameters, improve the security and stability of target data within the data set, ensure that the data set can maintain a relatively stable state, and thus facilitate safe and stable operation and maintenance management of the big data platform, avoiding confusion and loss of data within the big data platform;
[0169] Data optimization instructions include instructions for reducing capacity occupancy, instructions for increasing data encryption ratio, instructions for increasing unit processing rate, and instructions for reducing abnormal access frequency;
[0170] The method for formulating the instruction to reduce the capacity occupancy rate, the instruction to increase the data encryption rate, the instruction to increase the unit processing rate, and the instruction to reduce the abnormal access frequency includes:
[0171] When the abnormal parameter is capacity occupancy, it means that the space occupied by the target data in the data set is too large, which will cause the operation and maintenance status of the data set to deteriorate and have a significant negative impact on the operation and maintenance management of the big data platform. In this case, an instruction to reduce the capacity occupancy is formulated.
[0172] When the abnormal parameter is the data encryption ratio, it means that the number of target data that has been encrypted in the data set is too small, which will cause the operation and maintenance status of the data set to deteriorate and have a significant negative impact on the operation and maintenance management of the big data platform. In this case, an instruction to increase the data encryption ratio is formulated;
[0173] When the abnormal parameter is the unit processing rate, it means that the number of target data processed per unit time in the data set is too small, which will cause the operation and maintenance status of the data set to deteriorate and have a greater negative impact on the operation and maintenance management of the big data platform. In this case, an instruction to increase the unit processing rate is formulated;
[0174] When the abnormal parameter is abnormal access frequency, it means that there are too many unauthorized access records in the data set, which will cause the operation and maintenance status of the data set to deteriorate and have a greater negative impact on the operation and maintenance management of the big data platform. In this case, an instruction to reduce the abnormal access frequency is formulated.
[0175] It should be noted that when the instructions to reduce capacity occupancy, increase data encryption ratio, increase unit processing rate and reduce abnormal access frequency are formulated, these instructions need to be executed through specific means and measures within the big data platform, so as to optimize and improve the abnormal parameters and maintain the security of the operation and maintenance management of the big data platform;
[0176] Exemplarily, when an instruction to reduce capacity occupancy is formulated, one or more databases can be added within the big data platform to store more target data. When an instruction to increase the data encryption ratio is formulated, the workload of the encryption management system is increased, and more target data is encrypted. When an instruction to increase the unit processing rate is formulated, the performance of the data processor is improved, and the amount of data processed per unit time is increased. When an instruction to reduce the frequency of abnormal access is formulated, the security verification level of the access management system is improved, and more unauthorized access is denied.
[0177] In this embodiment, by extracting data features of target data, and based on the data features, dividing the target data into data sets, collecting historical training data of the data sets, the historical training data includes comprehensive operation and maintenance parameters and operation and maintenance status values, training a machine learning model that predicts the operation and maintenance status values based on the historical training data, collecting real-time comprehensive operation and maintenance parameters, inputting them into the trained machine learning model, predicting the real-time operation and maintenance status values, and determining whether there is an abnormal set in the data set, comparing the comprehensive operation and maintenance parameters with the corresponding standard values, identifying the abnormal parameters, and formulating data optimization instructions based on the abnormal parameters; compared with the prior art, the massive target data in the big data platform can be divided into multiple independent data sets It achieves the classification effect of breaking down massive data into small pieces, reduces the computational burden of each data set, and avoids data confusion or even loss when a large number of different types of data are mixed together. By collecting comprehensive operation and maintenance parameters and combining machine learning models, it can accurately predict the real-time operation and maintenance status value of the data set, and quickly and accurately judge the operation and maintenance management quality of each data set, thereby playing an accurate predictive effect on the operation and maintenance management of the big data platform, and can formulate corresponding data optimization instructions, and perform targeted data optimization processing on the data in the big data platform, ensuring that the data in the big data platform can remain in a stable state, thereby improving the security and stability of the operation and maintenance management of the big data platform.
[0178] Example 2: Please refer to Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, and a big data platform operation and maintenance management method based on artificial intelligence is provided, which is applied to a big data platform and is implemented based on an artificial intelligence-based big data platform operation and maintenance management system, including:
[0179] S1: extracting data features of target data and dividing the target data into data sets based on the data features, where the data features include type features and attribute features;
[0180] S2: collect historical training data of the data set, which includes comprehensive operation and maintenance parameters and operation and maintenance status values. The comprehensive operation and maintenance parameters include capacity occupancy rate, data encryption ratio, unit processing rate and abnormal access frequency;
[0181] S3: Based on historical training data, train a machine learning model to predict the operation and maintenance status value;
[0182] S4: Collect real-time comprehensive operation and maintenance parameters, input them into the trained machine learning model, predict the real-time operation and maintenance status value, and determine whether there is an abnormal set in the data set; if there is an abnormal set, execute S5; if there is no abnormal set, repeat S4;
[0183] S5: Compare the comprehensive operation and maintenance parameters with the corresponding standard values, identify abnormal parameters, and formulate data optimization instructions based on the abnormal parameters.
[0184] Example 3: Please refer to Figure 3 As shown, this embodiment discloses an electronic device, including a processor and a memory;
[0185] Wherein, the memory stores a computer program that can be called by the processor;
[0186] The processor executes the artificial intelligence-based big data platform operation and maintenance management method by calling the computer program stored in the memory.
[0187] Since the electronic device introduced in this embodiment is an electronic device used to implement a big data platform operation and maintenance management method based on artificial intelligence in Example 2 of this application, based on the big data platform operation and maintenance management method based on artificial intelligence introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation method of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not introduced in detail here. As long as the technical personnel of this field implement the electronic device used in the big data platform operation and maintenance management method based on artificial intelligence in the embodiment of this application, it belongs to the scope of protection of this application.
[0188] Example 4: Please refer to Figure 4 As shown, this embodiment discloses a computer-readable storage medium on which a rewritable computer program is stored;
[0189] When the computer program is run, it implements the artificial intelligence-based big data platform operation and maintenance management method.
[0190] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. An AI-based big data platform operation and maintenance management system, applied to big data platforms, It is characterized in that include: A set partitioning module is used to extract data features of target data and partition the target data into data sets based on the data features, wherein the data features include type features and attribute features; The extraction methods of type features and attribute features include: Through natural language processing technology, the content semantics of i target data in the database are identified one by one to obtain i semantic words; Mark the first part corresponding to the text and the second part corresponding to the number in the semantic word, and establish a split point between the first part and the second part; Split i semantic words from the split point through word segmentation technology to obtain i content bits and i type bits, and record the characters in the type bits to obtain i type features; Query the attributes of i original data through the data attribute table, and record the text part of the i attributes as attribute features to obtain i attribute features; Methods for partitioning data sets include: Aggregate the target data with the same type of features, query the storage time of the target data, and arrange the target data in the order of the storage time to generate u data columns; Aggregate the target data with the same attribute characteristics in the data column to obtain data sets, with The target data in the database is divided into Subsets; Will The type features and attribute features corresponding to the subsets are combined one by one to obtain Collection notes; Respectively Create a comment box at the end of each subset and Import the corresponding Collection notes, get data sets; A data collection module is used to collect historical training data of a data set. The historical training data includes comprehensive operation and maintenance parameters and operation and maintenance status values. The comprehensive operation and maintenance parameters include capacity occupancy rate, data encryption ratio, unit processing rate, and abnormal access frequency. Methods for obtaining the data encryption ratio include: At time T1, Randomly grab an equal amount of target data from each data set, record them as first detection data, and count the number of first detection data; At time T2, Randomly grab an equal amount of target data from each data set, record them as second detection data, and count the number of second detection data; The encryption management system queries the encryption status of the first detection data and the second detection data one by one, and marks the target data whose encryption status is encrypted, to obtain x first encrypted data and x second encrypted data; Query the storage time and encryption time of the x first encrypted data and the x second encrypted data one by one through the timestamp, and record the time from the storage time to the encryption time as the encryption time, to obtain x first encryption time and x second encryption time; The first encrypted data and the second encrypted data whose first encryption duration and the second encryption duration are less than the preset duration threshold are respectively recorded as the first available data and the second available data, and the number of the first available data and the number of the second available data are counted; Compare the amount of the first available data with the amount of the first detected data to obtain The first proportion; Compare the amount of the second available data with the amount of the second detected data to obtain The second percentage; Will The first proportions are respectively Add the second proportions and find the average, and we get The percentage of data encryption; Methods for obtaining unit processing rate include: From Mark an equal amount of target data in the data set and record it as the data to be marked; The keywords of the data to be labeled are identified one by one through natural language processing technology, and the keywords are marked on the data to be labeled to obtain the processed data and count the total amount of the processed data; Importing all processed data into the data processor at the same time, and after a preset processing time, receiving and recording the processed data exported by the data processor; Record the exported processed data without tags as processed data, and count the number of processed data; Will The number of processed data sets and Compare the total amount of processed data of the data sets to obtain Sub-processing rate; Query through system monitoring tools The fixed delay time of each data set is set, and the preset processing time is compared with After subtracting the fixed delay time, we get Effective processing time; Will The processing rate of each sub- Compare the effective processing time and obtain Unit processing rate; Methods for obtaining abnormal access frequency include: Query one by one through the access management system The time when the target data in the data set first appears in the access record is obtained an initial moment; When no access records appear within the preset access period, The last time an access record appears in a data set is recorded as the end time, and the an ending moment; Will Initial time to The time between the end times is recorded as the effective access time, and the The effective access time; Mark them one by one Collect all access records of a data set within the effective access time, and query the access status value of the access record one by one; The access records with access status values of unauthorized are recorded as abnormal accesses, and the number of abnormal accesses is counted to obtain outlier values; Will The abnormal value is Compare the effective access time and obtain Abnormal access frequency; Model training module, used to train a machine learning model that predicts operation and maintenance status values based on historical training data; The training methods of the machine learning model for predicting the operation and maintenance status value include: Convert the comprehensive operation and maintenance parameters into a corresponding set of feature vectors, convert the operation and maintenance status values into digits, convert the operation and maintenance safety status into 0, and convert the operation and maintenance abnormal status into 1; The feature vector is used as the input of the machine learning model, and the operation and maintenance status value corresponding to each set of comprehensive operation and maintenance parameters is used as the output of the machine learning model. The operation and maintenance status value is used as the prediction target, and the sum of the prediction errors of all training data is minimized as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the training is stopped to obtain a machine learning model that can predict the operation and maintenance status value. The prediction and judgment module is used to collect real-time comprehensive operation and maintenance parameters, input them into the trained machine learning model, predict the real-time operation and maintenance status value, and determine whether there are abnormal sets in the data set; The instruction formulation module is used to compare the comprehensive operation and maintenance parameters with the corresponding standard values, identify abnormal parameters, and formulate data optimization instructions based on the abnormal parameters.
2. According to the big data platform operation and maintenance management system based on artificial intelligence according to claim 1, It is characterized in that The method for determining whether there is an abnormal set in the data set includes: When the output of the machine learning model is 0, the real-time operation and maintenance status value is the operation and maintenance safety status, and it is determined that there is no abnormal set in the data set; When the output of the machine learning model is 1, the real-time operation and maintenance status value is an abnormal operation and maintenance status, and it is determined that there is an abnormal set in the data set.
3. According to the big data platform operation and maintenance management system based on artificial intelligence according to claim 2, It is characterized in that The method for identifying abnormal parameters includes: Compare the capacity occupancy rate with the occupancy standard value, and when the capacity occupancy rate is greater than the occupancy standard value, identify the capacity occupancy rate as an abnormal parameter; Compare the data encryption ratio with the standard ratio value, and when the data encryption ratio is less than the standard ratio value, identify the data encryption ratio as an abnormal parameter; Compare the unit processing rate with the rate standard value, and when the unit processing rate is less than the rate standard value, identify the unit processing rate as an abnormal parameter; The abnormal access frequency is compared with the frequency standard value, and when the abnormal access frequency is greater than the frequency standard value, the abnormal access frequency is identified as an abnormal parameter.
4. According to the big data platform operation and maintenance management system based on artificial intelligence according to claim 3, It is characterized in that The data optimization instructions include instructions for reducing capacity occupancy, instructions for increasing data encryption ratio, instructions for increasing unit processing rate, and instructions for reducing abnormal access frequency; The method for formulating the instruction to reduce the capacity occupancy rate, the instruction to increase the data encryption rate, the instruction to increase the unit processing rate, and the instruction to reduce the abnormal access frequency includes: When the abnormal parameter is the capacity occupancy rate, an instruction to reduce the capacity occupancy rate is formulated; When the abnormal parameter is the data encryption ratio, an instruction to increase the data encryption ratio is formulated; When the abnormal parameter is the unit processing rate, an instruction to increase the unit processing rate is formulated; When the abnormal parameter is an abnormal access frequency, an instruction to reduce the abnormal access frequency is formulated.
