Comprehensive analysis and management system for data platform

Abstract

The invention provides a comprehensive analysis and management system for a data platform. The comprehensive analysis and management system comprises a central data processing unit and a plurality of local data processing nodes, the local nodes collect data and send the data to the central unit, and the central unit comprises a data analysis module and a data mining module. The data analysis module constructs a data model training sample set, and a data model is obtained through training and used for predicting the future trend of local node data. And the data mining module classifies the data according to the data storage and access frequency and the future trend, and optimizes the storage and access positions. And the central unit sends the trained model and parameters to local nodes, and the local nodes predict future data trends based on the model and put forward optimization suggestions. And the central unit is also responsible for issuing the data classification result to the local nodes so as to provide data storage and access position suggestions. The access efficiency can be improved, and the storage cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a comprehensive analysis and management system for a data platform. Background Art

[0002] In the field of data processing technology, existing data platform comprehensive analysis and management systems generally face challenges in data management efficiency and accuracy. These systems usually need to process and analyze large amounts of data from different data sources to support decision making and business operations. However, with the surge in data volume and the diversification of data types, traditional data processing methods have shown obvious limitations in processing speed, storage optimization, and analysis accuracy. Especially in the era of big data, existing data platforms often find it difficult to achieve efficient data access and storage management when processing heterogeneous data and large-scale data sets. In addition, existing systems also have deficiencies in data trend prediction and environmental monitoring, resulting in the inability to effectively guarantee the suitability of the data storage environment, affecting the security and availability of data.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the existing data platform comprehensive analysis and management system lacks an intelligent processing mechanism in data classification and storage optimization, resulting in low data access efficiency and high storage costs; at the same time, the existing system relies on a single prediction model for data trend prediction, which lacks accuracy and robustness; in addition, the existing system also fails to fully utilize sensor technology and real-time data processing capabilities in environmental monitoring and data access management, and is unable to respond to environmental changes and optimize data access in a timely manner. Summary of the invention

[0004] The present invention provides a data platform comprehensive analysis and management system, comprising:

[0005] Central data processing unit, local data processing nodes;

[0006] Each local data processing node sends the acquired data from its own data source to the central data processing unit;

[0007] The central data processing unit includes a data analysis module and a data mining module; the data analysis module constructs a training sample set of the data model according to various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model;

[0008] The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model;

[0009] The data mining module obtains the data storage and access mode according to the storage and access frequency of each type of data of each local data processing node, and classifies each type of data according to the future trend of each type of data of each local data processing node and the data storage and access mode to obtain the data classification results of each local data processing node;

[0010] The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions;

[0011] The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node, and each local data processing node provides data storage and access location suggestions based on the data classification results.

[0012] Furthermore, the data mining module classifies the data through the following steps:

[0013] Step 1: Collect the storage and access frequencies of various types of data at each local data processing node;

[0014] Step 2: Analyze storage and access frequency to determine the activity level of data;

[0015] Step 3: Divide the data into three categories: high-frequency access data, common data, and low-frequency access data according to the future trend and activity level of the data;

[0016] Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

[0017] Furthermore, when a local data processing node has a need to access low-frequency data but has insufficient storage, it sends data shortage information to the central data processing unit. The central data processing unit searches the storage of other local data processing nodes. If the storage of other local data processing nodes contains the low-frequency access data, the central data processing unit issues a scheduling suggestion to schedule the low-frequency data from the storage of other local data processing nodes to the local data processing node in need.

[0018] Furthermore, the data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model;

[0019] The weights are assigned based on the prediction accuracy of the three models.

[0020] The weights for each model are:

[0021]

[0022] Among them, k = 1, 2, 3, is the prediction accuracy of each model.

[0023] Furthermore, each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit;

[0024] The central data processing unit calculates the access efficiency of each storage location according to the access time of each storage location, and divides the internal storage locations into a fast access area, a general access area, and a low-speed access area based on the access efficiency; the central data processing unit sends the internal storage location divisions of each data source to the local data processing node, and the local data processing node gives data storage location recommendations based on the data classification results sent by the central data processing unit; high-frequency access data is placed in the fast access area; ordinary data is placed in the general access area; and low-frequency access data is placed in the low-speed access area.

[0025] Further, the access data of each type of data sent by each local data processing node includes: the number of accesses of each type of data per month;

[0026] The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including:

[0027] The number of accesses to different types of data in different months is recorded in tensor, and the tensor is used as a training sample set for the neural network model;

[0028] The number of visits to different types of data in different months is formed into a data frame, which is used as a training sample set for the random forest and GBDT models.

[0029] Furthermore, the random forest model training process includes:

[0030] Determine the interval range of the number and depth of the model parameter trees, where the number and depth of the trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of the trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of the parameter trees through the grid search method, that is, under different combinations of the number and depth of the trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of the trees when the sum of the mean square errors of various data is the smallest as the required random forest model parameter value.

[0031] Furthermore, the neural network model includes a plurality of neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located; the training process of the neural network sub-model of each local data processing node includes:

[0032] Input the visit data of each type of data in each data frame in different months into the corresponding neural network sub-model for training;

[0033] A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

[0034] Furthermore, the various data information of the data source stored in the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with early creation time.

[0035] Furthermore, the data source is installed with multiple sensors for acquiring environmental monitoring data in real time and uploading the data to a local data processing node; the local data processing node monitors whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the data exceeds the temperature, humidity, and light intensity range suitable for data storage, an alarm is issued to prompt the administrator to take measures to adjust the environment.

[0036] The above-mentioned embodiments according to the present invention have at least the following beneficial effects: the data platform comprehensive analysis and management system described in the present invention can realize efficient processing and analysis of various data sources through the collaborative work of the central data processing unit and the local data processing nodes. The system uses integrated data models, such as random forest models, gradient boosting decision tree models GBDT and neural network models, to conduct in-depth mining and trend prediction of data, thereby providing accurate optimization suggestions for local data processing nodes. In addition, the system divides the data into high-frequency access data, ordinary data and low-frequency access data through intelligent data classification, and guides the optimization of data storage and access location according to the activity level and future trends of the data, so as to improve data access efficiency and reduce storage costs. This intelligent data management method enables the data platform to respond to data storage and access needs more flexibly, while ensuring the suitability of the data storage environment, which can improve the security and availability of data.

[0037] In addition, by collecting and analyzing the storage and access frequencies of each local data processing node, the system can timely identify and schedule low-frequency access data and optimize the allocation of data storage resources. In terms of environmental monitoring, the system uses sensors to obtain data in real time, monitor the temperature, humidity and light intensity of the data storage environment, and immediately issues an alarm and prompts the administrator to take measures once an abnormality is detected, thereby ensuring the stability of the data storage environment and the integrity of the data. This comprehensive analysis and management mechanism can not only improve the level of intelligent data processing, but also enhance the system's ability to adapt to environmental changes and ensure data security, and can provide users with an efficient, reliable and economical data management solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0039] Figure 1 A schematic diagram of the structure of a comprehensive analysis and management system for a data platform provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0041] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0042] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0043] Reference below Figure 1 , Figure 1 This is a schematic diagram of the structure of a comprehensive analysis and management system for a data platform provided by an embodiment of the present invention. Figure 1 As shown, a data platform comprehensive analysis and management system 100 includes:

[0044] Central data processing unit 101, local data processing node 102;

[0045] Each local data processing node sends the acquired data from its own data source to the central data processing unit;

[0046] The central data processing unit includes a data analysis module and a data mining module; the data analysis module constructs a training sample set of the data model according to various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model;

[0047] The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model;

[0048] The data mining module obtains the data storage and access mode according to the storage and access frequency of each type of data of each local data processing node, and classifies each type of data according to the future trend of each type of data of each local data processing node and the data storage and access mode to obtain the data classification results of each local data processing node;

[0049] The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions;

[0050] The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node, and each local data processing node provides data storage and access location suggestions based on the data classification results.

[0051] It should be noted that the system includes a central data processing unit and multiple local data processing nodes. The central data processing unit is responsible for centralized processing and analysis of data collected by local data processing nodes. Local data processing nodes refer to data processing units distributed in different geographical locations or different data sources. They are responsible for collecting and preliminarily processing data and then sending the data to the central unit. The central data processing unit is the core of the system. It has powerful data processing capabilities and can conduct in-depth analysis and mining of large amounts of data.

[0052] Specifically, the central data processing unit includes a data analysis module and a data mining module. The data analysis module is responsible for building a training sample set of a data model based on various types of data sent by local data processing nodes. The data model here refers to a statistical or machine learning model used to predict future trends of data. The training sample set is a set of historical data used to train the data model. The data mining module obtains the storage and access pattern of the data based on the storage and access frequency of the data, and predicts future trends.

[0053] Preferably, the central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes. These model parameters include but are not limited to key parameters such as model weights and thresholds. The local data processing nodes predict the future data trends of their respective data sources based on these trained data models, and make optimization suggestions based on the prediction results, such as adjusting the data collection frequency, optimizing the data storage structure, etc.

[0054] Furthermore, the central data processing unit will also send the data classification results output by the data mining module to each local data processing node to guide them to optimize the data storage and access location.

[0055] In some embodiments, the data mining module classifies the data by:

[0056] Step 1: Collect the storage and access frequencies of various types of data at each local data processing node;

[0057] Step 2: Analyze storage and access frequency to determine the activity level of data;

[0058] Step 3: Divide the data into three categories: high-frequency access data, common data, and low-frequency access data according to the future trend and activity level of the data;

[0059] Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

[0060] It should be noted that the data mining module classifies data through a series of steps, including collecting the storage and access frequency of the data, analyzing these frequencies to determine the activity level of the data, and then classifying the data into different categories based on the future trend and activity level of the data. Data classification here refers to dividing data into different groups based on its characteristics and usage patterns in order to more effectively manage and optimize data storage and access.

[0061] Specifically, the data mining module first collects the storage and access frequencies of various types of data in each local data processing node. These frequencies refer to the number of times the data is accessed or modified in a specific time period. Then, the module analyzes these frequencies to determine the activity level of the data. The activity level refers to the frequency of data access, which can be high, medium, or low. Then, based on the future trend of the data and the activity level, the data is divided into three categories: high-frequency access data, ordinary data, and low-frequency access data. These categories will guide the optimization of data storage and access locations.

[0062] Preferably, the data mining module can also consider other factors during the classification process, such as the importance of the data, the life cycle of the data, etc., to further improve the efficiency of data management. For example, for high-frequency access data, it can be stored on a faster storage medium, such as a solid-state hard drive, to increase access speed; while for low-frequency access data, it can be stored on a hard drive with lower cost and larger capacity.

[0063] Furthermore, the module can also dynamically adjust data classification and storage strategies based on data usage patterns and business needs to adapt to the ever-changing data environment.

[0064] In some embodiments, when a local data processing node has a need to access low-frequency data but has insufficient storage, it sends data shortage information to the central data processing unit, and the central data processing unit searches the storage of other local data processing nodes. If the storage of other local data processing nodes contains the low-frequency access data, the central data processing unit issues a scheduling suggestion to schedule the low-frequency data from the storage of other local data processing nodes to the local data processing node in need.

[0065] It should be noted that when a local data processing node faces the need for low-frequency access to data but has insufficient storage resources, the node will send data shortage information to the central data processing unit. The low-frequency access data here refers to data that is not frequently accessed or queried, and the data shortage information is a request signal used to inform the central unit that the local node needs more storage resources to process such data.

[0066] Specifically, when the local data processing node detects that the storage resources are insufficient to meet the demand for low-frequency access to data, it will generate and send a data shortage message containing the required data type and quantity to the central data processing unit.

[0067] More specifically, after receiving the information, the central data processing unit will perform a series of search operations to query the storage resources of other local data processing nodes to determine whether the required low-frequency access data exists. During this process, the central unit may use specific algorithms to optimize search efficiency and ensure rapid response to the needs of local nodes.

[0068] Preferably, the central data processing unit will issue a scheduling suggestion when it finds that other local data processing nodes have the required low-frequency access data. The scheduling suggestion will include detailed information of data migration, such as source node and target node, amount of data migration and recommended migration time.

[0069] Furthermore, the central unit can also dynamically adjust the scheduling strategy according to factors such as network conditions, urgency and importance of data to ensure the efficiency and reliability of data migration. When implementing data migration, encryption and compression technology can also be used to protect data security and improve transmission efficiency.

[0070] In some embodiments, the data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model;

[0071] The weights are assigned based on the prediction accuracy of the three models.

[0072] The weights for each model are:

[0073]

[0074] Among them, k = 1, 2, 3, which is the prediction accuracy of each model.

[0075] It should be noted that the data model uses an integrated model of random forest model, gradient boosted decision tree model GBDT and neural network model, which are weighted by their respective prediction accuracy. The integrated model is a method of combining the prediction results of multiple different models to improve the overall prediction performance. Here, random forest, GBDT and neural network are three different machine learning models, each of which has unique algorithms and advantages.

[0076] Specifically, the prediction accuracy of these three models is used to determine their weights in the integrated model. Prediction accuracy refers to the degree of consistency between the model's prediction results and the actual results, which is usually obtained through methods such as cross-validation. The weight allocation is based on the accuracy of each model, according to the formula

[0077]

[0078] Calculation, where i = 1, 2, 3 represent random forest, GBDT and neural network models respectively. This weight distribution mechanism ensures that the prediction performance of the model is proportional to its influence in the integrated model.

[0079] Preferably, in order to further improve the prediction performance of the integrated model, more machine learning models can be introduced or the parameters of the existing models can be adjusted. For example, it is possible to consider adding a support vector machine (SVM) or a deep learning model as an additional prediction model and assign weights according to their prediction accuracy.

[0080] Furthermore, feature selection and feature engineering can be used to optimize the model input and improve the model's prediction accuracy. In practical applications, the weights of each model can be dynamically adjusted according to the characteristics of the data and business needs to achieve the best prediction effect.

[0081] In some embodiments, each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit;

[0082] The central data processing unit calculates the access efficiency of each storage location according to the access time of each storage location, and divides the internal storage locations into a fast access area, a general access area, and a low-speed access area based on the access efficiency; the central data processing unit sends the internal storage location divisions of each data source to the local data processing node, and the local data processing node gives data storage location recommendations based on the data classification results sent by the central data processing unit; high-frequency access data is placed in the fast access area; ordinary data is placed in the general access area; and low-frequency access data is placed in the low-speed access area.

[0083] It should be noted that the local data processing node uploads the access time of each storage location within the stored data source to the central data processing unit. Access time refers to the time it takes to access data, and this indicator can be used to measure the performance of the storage location. The central data processing unit uses this data to calculate the access efficiency of each storage location and then classify the storage locations.

[0084] Specifically, after receiving the access time data uploaded by the local data processing node, the central data processing unit will calculate the average access time of each storage location based on this data. The access efficiency can be determined by comparing the average access time of different storage locations, and the location with the shortest access time is considered to have high access efficiency.

[0085] More specifically, based on this information, the central data processing unit divides the internal storage location into a fast access area, a general access area, and a slow access area. This classification helps optimize the storage and access of data and ensures that high-frequency access data is stored in a location with high access efficiency.

[0086] Preferably, the central data processing unit may also consider other factors when classifying storage locations, such as the reliability of the storage location, the importance of the data, and the access mode, etc. For example, for critical data that requires high reliability storage, even if its access efficiency is not the highest, it may be allocated to the fast access area.

[0087] Furthermore, the central data processing unit can also dynamically adjust the classification of storage locations and data storage strategies according to the actual operation of the system and predicted future data access trends to adapt to changing business needs and technical environments.

[0088] In some embodiments, the access data of each type of data sent by each local data processing node includes: the number of accesses of each type of data per month;

[0089] The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including:

[0090] The number of accesses to different types of data in different months is recorded in tensor, and the tensor is used as a training sample set for the neural network model;

[0091] The number of visits to different types of data in different months is formed into a data frame, which is used as a training sample set for the random forest and GBDT models.

[0092] It should be noted that the access data of various types of data sent by the local data processing nodes include the number of accesses to each type of data per month. The number of accesses refers to the number of times data is requested or queried within a specific time period. This indicator is crucial for understanding data usage patterns and optimizing data management. The central data processing unit builds a training sample set of the data model based on these access times for effective data analysis and prediction.

[0093] Specifically, after the central data processing unit receives the number of visits to each type of data each month sent by the local data processing node, it will use this data to construct a training sample set. For the neural network model, the training sample set is recorded in tensor, which is a multidimensional array structure used to represent and process data in the neural network. For the random forest and GBDT models, the training sample set forms a data frame, which is a two-dimensional labeled data structure commonly used to store and manage structured data. These training sample sets will be used to train and optimize data models to predict future access trends of data.

[0094] Preferably, in addition to the number of visits, other relevant indicators can be incorporated into the construction of the training sample set, such as the size, type, source, etc. of the data. This additional information can provide a more comprehensive perspective and help improve the accuracy of model predictions.

[0095] Furthermore, incremental learning or online learning methods can be used to enable the model to adapt to changes in data access patterns, thereby dynamically updating model parameters. During implementation, it is also possible to consider using a distributed computing framework to process large-scale data sets to improve the efficiency and scalability of data processing.

[0096] In some embodiments, the random forest model training process includes:

[0097] Determine the interval range of the number and depth of the model parameter trees, where the number and depth of the trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of the trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of the parameter trees through the grid search method, that is, under different combinations of the number and depth of the trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of the trees when the sum of the mean square errors of various data is the smallest as the required random forest model parameter value.

[0098] It should be noted that the random forest model training process includes determining the interval range of the number and depth of the model parameter trees, where the number and depth of the trees are both integers. Random forest is an ensemble learning method that builds multiple decision trees for classification or regression prediction. During the model training process, the number and depth of decision trees need to be set to control the complexity and predictive power of the model. The number of trees refers to the number of decision trees in the random forest, and the depth of the tree refers to the maximum number of layers of the decision tree.

[0099] Specifically, when initializing the random forest model, the number and depth of trees and the time period s are input into the model, and the data frame dataframe is also input into the model. Before model training begins, it is necessary to determine the possible value range of the number and depth of trees. The setting of these parameters will affect the performance of the model. The time period s refers to the time limit for model training, which is used to control the duration of the training process. The data frame dataframe contains historical data used to train the model and is the basis for building a random forest. The number and depth of parameter trees are determined by the grid search method, that is, under different combinations of the number and depth of trees, the mean square error of the predicted value and the true value of the data access volume is calculated, and the combination of the number and depth of trees when the sum of the mean square errors of various data is minimized is found as the required random forest model parameter value.

[0100] Preferably, in addition to the number and depth of trees, other parameters can also be optimized, such as the splitting criteria of the decision tree, the minimum number of leaf node samples, etc. Adjustment of these parameters can further improve the prediction accuracy and robustness of the model.

[0101] Furthermore, a cross-validation method can be used to evaluate the performance of the model under different parameter combinations, so as to select the best parameter configuration. In actual operation, automated hyperparameter optimization techniques, such as Bayesian optimization, can also be used to find the optimal parameter combination to improve the efficiency and effect of model training.

[0102] In some embodiments, the neural network model includes a plurality of neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located; the training process of the neural network sub-model of each local data processing node includes:

[0103] Input the visit data of each type of data in each data frame in different months into the corresponding neural network sub-model for training;

[0104] A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

[0105] It should be noted that the neural network model includes multiple neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located. A neural network is a computing model that mimics the neuron connections in the human brain. It predicts or classifies by learning the complex relationships between input data. Here, the neural network is decomposed into multiple sub-models, each of which focuses on predicting the access volume of a specific type of data to improve the accuracy and efficiency of the prediction.

[0106] Specifically, the training process of the neural network sub-model of each local data processing node includes inputting the visit volume data of each data in each data frame in different months into the corresponding neural network sub-model for training. The data frame is a two-dimensional data structure used to store and process tabular data. Here, it contains the visit volume information of various data in different months. The neural network sub-model makes predictions by learning the patterns of these data. Genetic algorithm is a search algorithm that simulates the natural selection process and is used to optimize the parameters of the neural network sub-model, including the learning rate and the number of hidden layer neurons. The learning rate determines the amplitude of the network weight update, while the number of hidden layer neurons determines the complexity of the network.

[0107] Preferably, the training of the neural network sub-model can use a variety of optimization techniques to improve performance. For example, batch normalization can be used to speed up the training process, or dropout technology can be used to prevent overfitting.

[0108] Furthermore, early stopping can be used to avoid overtraining, that is, training is stopped when the performance of the validation set no longer improves. In the setting of the parameter search space, a finer granularity can be used, or a more reasonable range can be set in combination with domain knowledge. In the setting of the number of iterations, an adaptive iteration strategy can be used to dynamically adjust the number of iterations according to the progress of model training. These optimization measures can further improve the prediction accuracy and robustness of the neural network sub-model.

[0109] In some embodiments, the various types of data information of the data source stored by the local data processing node also include the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with the earliest creation time.

[0110] It should be noted that the various data information of the data source stored by the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with the earliest creation time. The creation time here refers to the time point when the data was first generated or recorded. It is a basic attribute of the data and is usually used for data version control and data access optimization.

[0111] Specifically, after receiving the data, the local data processing node will extract the creation time of each piece of data and sort the data according to the time information. The sorting can adopt various algorithms, such as quick sort, merge sort, etc., to ensure that the data is arranged in the order of creation time.

[0112] More specifically, when a data access request arrives, the node will give priority to data that was created earlier. This can ensure the timing and consistency of the data, which is especially important when processing historical data or performing data retrospective analysis.

[0113] Preferably, in addition to sorting by creation time, the access priority of data can also be adjusted according to the importance of data, access frequency or other business rules. For example, for business-critical data, even if it is created later, its priority can be increased to ensure fast access to critical data.

[0114] Furthermore, an intelligent pre-fetching mechanism can be introduced to load data that may be accessed into a fast storage area in advance based on data access patterns and prediction algorithms to reduce waiting time and improve the overall performance of the system. In actual operations, it is also possible to combine data lifecycle management strategies to regularly clean up or archive old data to optimize the use of storage resources.

[0115] In some embodiments, the data source is installed with multiple sensors for acquiring environmental monitoring data in real time and uploading the data to a local data processing node; the local data processing node monitors whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the data exceeds the temperature, humidity, and light intensity range suitable for data storage, an alarm is issued to prompt the administrator to take measures to adjust the environment.

[0116] It should be noted that the data source is equipped with multiple sensors for real-time environmental monitoring data. These sensors are devices used to monitor and record various parameters of the environment in which the data source is located. Sensors can detect environmental factors such as temperature, humidity, and light intensity, and upload these data to local data processing nodes. Environmental monitoring data is crucial to ensure the security and reliability of data storage, because improper environmental conditions may damage the data storage medium or affect the integrity of the data.

[0117] Specifically, the local data processing node will monitor whether the temperature, humidity, and light intensity of the data storage environment are within a range suitable for data storage based on the data provided by the sensor. The setting of these parameters is usually based on the technical specifications and optimal operating conditions of the data storage device. For example, the temperature may need to be maintained at 20-25 degrees Celsius, the humidity may need to be controlled between 45%-55%, and the light intensity may need to be maintained below a certain standard. The local data processing node will monitor the environmental status in real time based on these set parameter ranges and compare it with the preset standards.

[0118] Preferably, if the monitored environmental factors exceed the range suitable for data storage, the local data processing node will issue an alarm, prompting the administrator to take measures to adjust the environment. The alarm can be issued in a variety of ways, such as sending an email, SMS notification, or displaying a warning message on the system interface. After receiving the alarm, the administrator can take corresponding measures, such as adjusting the air conditioning temperature, controlling the humidity, or blocking excessive light.

[0119] Furthermore, the system can also automatically execute some preset adjustment measures, such as starting the dehumidification system or adjusting the curtains, to quickly respond to environmental changes and ensure the stability of the data storage environment and the security of the data.

[0120] The above-mentioned various embodiments of the present invention have the following beneficial effects: The design of the central data processing unit and the local data processing node of the data platform comprehensive analysis and management system described in the present invention can realize efficient processing and intelligent analysis of data. The system constructs a training sample set of the data model, and uses the random forest model, the gradient boosting decision tree model GBDT and the neural network model to conduct comprehensive analysis and trend prediction on the data, which can improve the accuracy of the prediction and the efficiency of data processing. In addition, the data mining module of the system can classify the data into different activity levels according to the storage and access frequency of the data, which can optimize the storage and access location of the data, improve the data access efficiency and reduce the storage cost. This design can also achieve the optimal allocation of data resources through the scheduling suggestions of the central data processing unit when the local data processing node is insufficient in storage.

[0121] The system can further improve the prediction accuracy of the data model and the personalized management of data access by integrating model weight distribution and uploading data access status of local data processing nodes. In addition, the system analyzes the access time of each storage location within the data source, divides the access efficiency of the storage location, and provides more reasonable storage location suggestions for high-frequency access data, ordinary data, and low-frequency access data. The system can also prioritize access according to the creation time of the data to improve the timeliness of data access. At the same time, the data storage environment is monitored in real time through sensors to ensure that the temperature, humidity, light intensity and other conditions of data storage are suitable, and alarms can be issued in time to prompt administrators to take measures, thereby improving the security and reliability of data. These designs make the entire data platform have significant intelligent and efficient advantages in data processing, analysis, storage and environmental monitoring.

[0122] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0123] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention to form a technical solution.

Claims

1. A data platform comprehensive analysis and management system, characterized in that the system include: Central data processing unit, local data processing nodes; Each local data processing node sends the acquired data from its own data source to the central data processing unit; The central data processing unit includes a data analysis module and a data mining module; The data analysis module constructs a training sample set of the data model according to various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model; The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model; The data mining module obtains the data storage and access mode according to the storage and access frequency of each type of data of each local data processing node, and classifies each type of data according to the future trend of each type of data of each local data processing node and the data storage and access mode to obtain the data classification results of each local data processing node; The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions; The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node, and each local data processing node provides data storage and access location suggestions based on the data classification results.

2. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data mining module classifies the data by following the steps below: Step 1: Collect the storage and access frequencies of various types of data at each local data processing node; Step 2: Analyze storage and access frequency to determine the activity level of data; Step 3: Divide the data into three categories: high-frequency access data, common data, and low-frequency access data according to the future trend and activity level of the data; Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

3. The data platform comprehensive analysis and management system according to claim 1, characterized in that: When a local data processing node has a need to access low-frequency data but has insufficient storage, it sends data shortage information to the central data processing unit. The central data processing unit searches the storage of other local data processing nodes. If the storage of other local data processing nodes contains the low-frequency access data, the central data processing unit issues a scheduling suggestion to schedule the low-frequency data from the storage of other local data processing nodes to the local data processing node in need.

4. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model; The weights are assigned based on the prediction accuracy of the three models. The weights for each model are: Among them, k = 1, 2, 3, is the prediction accuracy of each model.

5. The data platform comprehensive analysis and management system according to claim 1, characterized in that: Each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit; The central data processing unit calculates the access efficiency of each storage location according to the access time of each storage location, and divides the internal storage location into a fast access area, a general access area, and a low-speed access area based on the access efficiency; the central data processing unit sends the internal storage location division of each data source to the local data processing node, and the local data processing node gives data storage location suggestions based on the data classification results sent by the central data processing unit; high-frequency access data is placed in the fast access area; ordinary data is placed in the general access area; Infrequently accessed data is placed in the low-speed access area.

6. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The access data of each type of data sent by each local data processing node includes: the number of accesses to each type of data per month; The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including: The number of accesses to different types of data in different months is recorded in tensor, and the tensor is used as a training sample set for the neural network model; The number of visits to different types of data in different months is formed into a data frame, which is used as a training sample set for the random forest and GBDT models.

7. The data platform comprehensive analysis and management system according to claim 6, characterized in that: The random forest model training process includes: Determine the interval range of the number and depth of the model parameter trees, where the number and depth of the trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of the trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of the parameter trees through the grid search method, that is, under different combinations of the number and depth of the trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of the trees when the sum of the mean square errors of various data is the smallest as the required random forest model parameter value.

8. The data platform comprehensive analysis and management system according to claim 6, characterized in that: The neural network model includes a plurality of neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located; the training process of the neural network sub-model of each local data processing node includes: Input the visit data of each type of data in each data frame in different months into the corresponding neural network sub-model for training; A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

9. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The various data information of the data source stored in the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with the earliest creation time.

10. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data source is installed with multiple sensors to obtain environmental monitoring data in real time and upload the data to the local data processing node; the local data processing node will monitor whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the temperature, humidity, and light intensity range is exceeded, an alarm will be issued to prompt the administrator to take measures to adjust the environment.

Images