A comprehensive analysis and management system for data platforms

Abstract

The present invention provides a comprehensive analysis and management system for a data platform, comprising a central data processing unit and a plurality of local data processing nodes. The local nodes collect data and send it to the central unit, which includes a data analysis and data mining module. The data analysis module constructs a data model training sample set, trains a data model, and uses it to predict future trends of local node data. The data mining module classifies data according to data storage and access frequency, as well as future trends, and optimizes storage and access locations. The central unit sends the trained model and parameters to the local nodes, which predict future data trends based on the model and make optimization suggestions. The central unit is also responsible for sending the data classification results to the local nodes to provide data storage and access location recommendations. The present invention can improve access efficiency and reduce storage costs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a comprehensive analysis and management system for a data platform. Background Art

[0002] In the field of data processing technology, existing data platform comprehensive analysis and management systems generally face challenges in data management efficiency and accuracy. These systems often need to process and analyze large amounts of data from different data sources to support decision-making and business operations. However, with the surge in data volume and the diversification of data types, traditional data processing methods have shown significant limitations in processing speed, storage optimization, and analysis accuracy. Especially in the era of big data, existing data platforms often struggle to achieve efficient data access and storage management when processing heterogeneous data and large-scale data sets. In addition, existing systems also have shortcomings in data trend prediction and environmental monitoring, resulting in the inability to effectively ensure the suitability of the data storage environment, affecting data security and availability.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the existing technology: the existing data platform comprehensive analysis and management system lacks an intelligent processing mechanism in data classification and storage optimization, resulting in low data access efficiency and high storage costs; at the same time, the existing system relies on a single prediction model for data trend prediction, which lacks accuracy and robustness; in addition, the existing system also fails to fully utilize sensor technology and real-time data processing capabilities in environmental monitoring and data access management, and is thus unable to respond to environmental changes and optimize data access in a timely manner. Summary of the Invention

[0004] The present invention provides a comprehensive analysis and management system for a data platform, comprising:

[0005] Central data processing unit, local data processing nodes;

[0006] Each local data processing node sends the data obtained from its own data source to the central data processing unit;

[0007] The central data processing unit includes a data analysis module and a data mining module; the data analysis module constructs a training sample set of the data model based on various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model;

[0008] The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model;

[0009] The data mining module obtains the data storage and access patterns according to the storage and access frequencies of various types of data at each local data processing node, and classifies various types of data according to the future trends, storage and access patterns of various types of data at each local data processing node to obtain the data classification results of each local data processing node;

[0010] The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions;

[0011] The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node, and each local data processing node provides data storage and access location recommendations based on the data classification results.

[0012] Furthermore, the data mining module classifies the data through the following steps:

[0013] Step 1: Collect the storage and access frequencies of various types of data at each local data processing node;

[0014] Step 2: Analyze storage and access frequency to determine the activity level of the data;

[0015] Step 3: Divide the data into three categories: high-frequency access data, normal data, and low-frequency access data based on the data's future trends and activity levels.

[0016] Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

[0017] Furthermore, when a local data processing node has a low-frequency access data demand and insufficient storage, it sends data shortage information to the central data processing unit. The central data processing unit will search the storage of other local data processing nodes. If the low-frequency access data exists in the storage of other local data processing nodes, the central data processing unit will issue a scheduling suggestion to schedule the low-frequency data from the storage of other local data processing nodes to the local data processing node in need.

[0018] Furthermore, the data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model;

[0019] The weights are assigned based on the prediction accuracy of the three models.

[0020] The weights for each model are:

[0021]

[0022] Among them, k = 1, 2, 3, is the prediction accuracy of each model.

[0023] Furthermore, each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit;

[0024] The central data processing unit calculates the access efficiency of each storage location based on the access time of each storage location, and divides the internal storage locations into fast access area, general access area, and low-speed access area based on the access efficiency; the central data processing unit sends the internal storage location division of each data source to the local data processing node, and the local data processing node gives data storage location recommendations based on the data classification results sent by the central data processing unit; high-frequency access data is placed in the fast access area; ordinary data is placed in the general access area; and low-frequency access data is placed in the low-speed access area.

[0025] Furthermore, the access data of each type of data sent by each local data processing node includes: the number of accesses to each type of data per month;

[0026] The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including:

[0027] The number of accesses to different types of data in different months is recorded using tensors, which serve as training sample sets for the neural network model.

[0028] The number of visits to different types of data in different months is formed into a data frame, which is used as the training sample set for the random forest and GBDT models.

[0029] Furthermore, the random forest model training process includes:

[0030] Determine the range of the number and depth of model parameter trees, where the number and depth of trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of parameter trees through the grid search method, that is, under different combinations of the number and depth of trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of trees when the sum of the mean square errors of various data is minimized as the required random forest model parameter value.

[0031] Furthermore, the neural network model includes a plurality of neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located; the training process of the neural network sub-model of each local data processing node includes:

[0032] Input the visit data of each data type in each data frame in different months into the corresponding neural network sub-model for training;

[0033] A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

[0034] Furthermore, the various data information of the data source stored in the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with the earliest creation time.

[0035] Furthermore, the data source is equipped with multiple sensors for real-time acquisition of environmental monitoring data and uploading the data to a local data processing node; the local data processing node monitors whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the data exceeds the temperature, humidity, and light intensity range suitable for data storage, an alarm is issued to prompt the administrator to take measures to adjust the environment.

[0036] According to the above-mentioned embodiment of the present invention, there are at least the following beneficial effects: the data platform comprehensive analysis and management system described in the present invention can realize efficient processing and analysis of various data sources through the collaborative work of the central data processing unit and the local data processing nodes. The system uses integrated data models, such as random forest models, gradient boosting decision tree models GBDT and neural network models, to conduct in-depth mining and trend prediction of data, thereby providing accurate optimization suggestions for local data processing nodes. In addition, the system divides data into high-frequency access data, ordinary data and low-frequency access data through intelligent data classification, and guides the optimization of data storage and access location based on the activity level and future trends of the data, so as to improve data access efficiency and reduce storage costs. This intelligent data management method enables the data platform to respond to data storage and access needs more flexibly, while ensuring the suitability of the data storage environment, which can improve data security and availability.

[0037] Furthermore, by collecting and analyzing the storage and access frequencies of each local data processing node, the system can promptly identify and schedule low-frequency access data, optimizing the allocation of data storage resources. Regarding environmental monitoring, the system uses sensors to acquire data in real time, monitoring the temperature, humidity, and light intensity of the data storage environment. Upon detecting an anomaly, it immediately issues an alarm and prompts administrators to take action, thereby ensuring the stability of the data storage environment and the integrity of the data. This comprehensive analysis and management mechanism not only enhances the intelligence of data processing but also strengthens the system's adaptability to environmental changes and its ability to ensure data security, providing users with an efficient, reliable, and economical data management solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0039] Figure 1 A schematic diagram of the structure of a comprehensive analysis and management system for a data platform provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0041] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0042] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0043] Reference below Figure 1 , Figure 1 This is a schematic diagram of the structure of a comprehensive analysis and management system for a data platform provided by an embodiment of the present invention. Figure 1 As shown, a data platform comprehensive analysis and management system 100 includes:

[0044] Central data processing unit 101, local data processing node 102;

[0045] Each local data processing node sends the data obtained from its own data source to the central data processing unit;

[0046] The central data processing unit includes a data analysis module and a data mining module; the data analysis module constructs a training sample set of the data model based on various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model;

[0047] The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model;

[0048] The data mining module obtains the data storage and access patterns according to the storage and access frequencies of various types of data at each local data processing node, and classifies various types of data according to the future trends, storage and access patterns of various types of data at each local data processing node to obtain the data classification results of each local data processing node;

[0049] The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions;

[0050] The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node, and each local data processing node provides data storage and access location recommendations based on the data classification results.

[0051] It should be noted that this system comprises a central data processing unit and multiple local data processing nodes. The central data processing unit is responsible for centrally processing and analyzing data collected by the local data processing nodes. Local data processing nodes are data processing units located in different geographical locations or from different data sources. They are responsible for collecting and initially processing data before sending it to the central unit. The central data processing unit is the core of the system, possessing powerful data processing capabilities and capable of in-depth analysis and mining of large amounts of data.

[0052] Specifically, the central data processing unit includes a data analysis module and a data mining module. The data analysis module is responsible for constructing a training sample set for the data model based on the various data sent by the local data processing nodes. The data model here refers to a statistical or machine learning model used to predict future data trends. The training sample set is a collection of historical data used to train the data model. The data mining module derives data storage and access patterns based on data storage and access frequency, and predicts future trends.

[0053] Preferably, the central data processing unit sends the trained data models and model parameters, including but not limited to key parameters such as model weights and thresholds, to the corresponding local data processing nodes. Based on these trained data models, the local data processing nodes predict future data trends for their respective data sources and, based on the predictions, provide optimization recommendations, such as adjusting data collection frequency and optimizing data storage structures.

[0054] Furthermore, the central data processing unit will also send the data classification results output by the data mining module to each local data processing node to guide them to optimize data storage and access locations.

[0055] In some embodiments, the data mining module classifies the data by:

[0056] Step 1: Collect the storage and access frequencies of various types of data at each local data processing node;

[0057] Step 2: Analyze storage and access frequency to determine the activity level of the data;

[0058] Step 3: Divide the data into three categories: high-frequency access data, normal data, and low-frequency access data based on the data's future trends and activity levels.

[0059] Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

[0060] It's important to note that the data mining module classifies data through a series of steps, including collecting data storage and access frequencies, analyzing these frequencies to determine the data's activity level, and then classifying the data into different categories based on future trends and activity levels. Data classification here refers to grouping data into different groups based on its characteristics and usage patterns, allowing for more efficient management and optimization of data storage and access.

[0061] Specifically, the data mining module first collects the storage and access frequencies of various types of data at each local data processing node. These frequencies refer to the number of times data is accessed or modified within a specific time period. The module then analyzes these frequencies to determine the data's activity level, which measures how frequently the data is accessed and can be categorized as high, medium, or low. Next, based on future trends and activity levels, the data is divided into three categories: high-frequency access data, average data, and low-frequency access data. These categorizations guide the optimization of data storage and access locations.

[0062] Preferably, the data mining module can also consider other factors during the classification process, such as the importance of the data and the life cycle of the data, to further improve the efficiency of data management. For example, frequently accessed data can be stored on faster storage media, such as solid-state drives, to increase access speed; while infrequently accessed data can be stored on lower-cost, larger-capacity hard drives.

[0063] Furthermore, the module can also dynamically adjust data classification and storage strategies based on data usage patterns and business needs to adapt to the ever-changing data environment.

[0064] In some embodiments, when a local data processing node has a low-frequency access data demand and insufficient storage, it sends data shortage information to the central data processing unit. The central data processing unit searches the storage of other local data processing nodes. If the low-frequency access data exists in the storage of other local data processing nodes, the central data processing unit issues a scheduling suggestion to schedule the low-frequency data from the storage of other local data processing nodes to the local data processing node in need.

[0065] It's important to note that when a local data processing node faces a need for infrequently accessed data but lacks sufficient storage resources, it sends a data shortage message to the central data processing unit. Infrequently accessed data refers to data that is infrequently accessed or queried, and the data shortage message serves as a request signal to the central unit that the local node needs more storage resources to process this data.

[0066] Specifically, when the local data processing node detects that the storage resources are insufficient to meet the demand for low-frequency access data, it will generate and send a data shortage message containing the required data type and quantity to the central data processing unit.

[0067] More specifically, after receiving this information, the central data processing unit performs a series of searches, querying the storage resources of other local data processing nodes to determine whether the required infrequently accessed data exists. During this process, the central unit may use specific algorithms to optimize search efficiency and ensure rapid response to local node requests.

[0068] Preferably, when the central data processing unit finds that other local data processing nodes have the required low-frequency access data, it will issue a scheduling suggestion. This scheduling suggestion will include detailed information about the data migration, such as the source and target nodes, the amount of data to be migrated, and the recommended migration time.

[0069] Furthermore, the central unit can dynamically adjust the scheduling strategy based on factors such as network conditions, data urgency, and importance to ensure efficient and reliable data migration. Encryption and compression technologies can also be used during data migration to protect data security and improve transmission efficiency.

[0070] In some embodiments, the data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model;

[0071] The weights are assigned based on the prediction accuracy of the three models.

[0072] The weights for each model are:

[0073]

[0074] Among them, k = 1, 2, 3, is the prediction accuracy of each model.

[0075] It should be noted that the data model uses an ensemble of random forests, gradient boosted decision trees (GBDTs), and neural networks. These models are weighted based on their respective prediction accuracy. An ensemble model combines the predictions of multiple different models to improve overall prediction performance. Here, random forests, GBDTs, and neural networks are three different machine learning models, each with unique algorithms and advantages.

[0076] Specifically, the prediction accuracy of these three models is used to determine their weights in the ensemble model. Prediction accuracy refers to the degree of consistency between the model's prediction results and the actual results, which is usually obtained through methods such as cross-validation. The weight distribution is based on the accuracy of each model, according to the formula

[0077]

[0078] Calculation, where i = 1, 2, and 3 represent the random forest, GBDT, and neural network models, respectively. This weight distribution mechanism ensures that the predictive performance of a model is proportional to its influence in the ensemble model.

[0079] Preferably, to further improve the prediction performance of the ensemble model, more machine learning models can be introduced or the parameters of existing models can be adjusted. For example, a support vector machine (SVM) or a deep learning model can be added as an additional prediction model and weighted according to their prediction accuracy.

[0080] Furthermore, feature selection and feature engineering can be used to optimize model inputs and improve the model's prediction accuracy. In practical applications, the weights of each model can be dynamically adjusted based on the characteristics of the data and business needs to achieve the best prediction results.

[0081] In some embodiments, each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit;

[0082] The central data processing unit calculates the access efficiency of each storage location based on the access time of each storage location, and divides the internal storage locations into fast access area, general access area, and low-speed access area based on the access efficiency; the central data processing unit sends the internal storage location division of each data source to the local data processing node, and the local data processing node gives data storage location recommendations based on the data classification results sent by the central data processing unit; high-frequency access data is placed in the fast access area; ordinary data is placed in the general access area; and low-frequency access data is placed in the low-speed access area.

[0083] It's important to note that local data processing nodes upload the access time of each storage location within the stored data source to the central data processing unit. Access time refers to the time it takes to access data, and this metric can be used to measure the performance of a storage location. The central data processing unit uses this data to calculate the access efficiency of each storage location and then classify the storage locations.

[0084] Specifically, after receiving access duration data uploaded by local data processing nodes, the central data processing unit calculates the average access duration for each storage location based on this data. Access efficiency can be determined by comparing the average access times of different storage locations. Locations with shorter access times are considered to have higher access efficiency.

[0085] More specifically, based on this information, the central data processing unit divides internal storage locations into fast access areas, general access areas, and slow access areas. This classification helps optimize data storage and access, ensuring that frequently accessed data is stored in locations with high access efficiency.

[0086] Preferably, the central data processing unit may also consider other factors when classifying storage locations, such as the reliability of the storage location, the importance of the data, and the access pattern, etc. For example, critical data that requires high-reliability storage may be allocated to the fast access area even if its access efficiency is not the highest.

[0087] Furthermore, the central data processing unit can also dynamically adjust the classification of storage locations and data storage strategies based on the actual operation of the system and predicted future data access trends to adapt to changing business needs and technical environments.

[0088] In some embodiments, the access data of each type of data sent by each local data processing node includes: the number of accesses to each type of data per month;

[0089] The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including:

[0090] The number of accesses to different types of data in different months is recorded using tensors, which serve as training sample sets for the neural network model.

[0091] The number of visits to different types of data in different months is formed into a data frame, which is used as the training sample set for the random forest and GBDT models.

[0092] It's important to note that the access data for each type of data sent by local data processing nodes includes the monthly access count for each type of data. Access count refers to the number of times data is requested or queried within a specific time period. This metric is crucial for understanding data usage patterns and optimizing data management. The central data processing unit uses these access counts to build training sample sets for data models, enabling effective data analysis and prediction.

[0093] Specifically, the central data processing unit receives monthly access counts for each type of data from local data processing nodes and uses this data to construct training sample sets. For neural network models, training sample sets are recorded using tensors, a multidimensional array structure used to represent and process data in neural networks. For random forest and GBDT models, training sample sets are stored as data frames, a two-dimensional labeled data structure commonly used to store and manage structured data. These training sample sets are used to train and optimize data models to predict future data access trends.

[0094] Preferably, in addition to the number of visits, other relevant indicators can be incorporated into the construction of the training sample set, such as the size, type, and source of the data. This additional information can provide a more comprehensive perspective and help improve the accuracy of model predictions.

[0095] Furthermore, incremental learning or online learning methods can be used to enable the model to adapt to changes in data access patterns, thereby dynamically updating model parameters. During implementation, it is also possible to consider using a distributed computing framework to process large-scale data sets to improve the efficiency and scalability of data processing.

[0096] In some embodiments, the random forest model training process includes:

[0097] Determine the range of the number and depth of model parameter trees, where the number and depth of trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of parameter trees through the grid search method, that is, under different combinations of the number and depth of trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of trees when the sum of the mean square errors of various data is minimized as the required random forest model parameter value.

[0098] It's important to note that the random forest model training process involves determining the ranges for the number and depth of model parameters, where both the number and depth of trees are integers. Random forest is an ensemble learning method that constructs multiple decision trees for classification or regression prediction. During model training, the number and depth of decision trees must be set to control model complexity and predictive power. The number of trees refers to the number of decision trees in the random forest, while the tree depth refers to the maximum number of layers in the decision tree.

[0099] Specifically, when initializing the random forest model, the initial values ​​for the number and depth of trees and the time period s are input into the model, along with the data frame dataframe. Before model training begins, it is necessary to determine the possible range of values ​​for the number and depth of trees, as the settings of these parameters will affect the performance of the model. The time period s refers to the time limit for model training and is used to control the duration of the training process. The data frame dataframe contains the historical data used to train the model and is the basis for building a random forest. The number and depth of parameter trees are determined through a grid search method. That is, under different combinations of the number and depth of trees, the mean square error between the predicted and true data access values ​​is calculated. The combination of the number and depth of trees that minimizes the sum of the mean square errors of various data is found as the required random forest model parameter value.

[0100] Preferably, in addition to the number and depth of trees, other parameters can also be considered for optimization, such as the splitting criteria of the decision tree, the minimum number of leaf node samples, etc. Adjustment of these parameters can further improve the prediction accuracy and robustness of the model.

[0101] Furthermore, cross-validation can be used to evaluate the performance of the model under different parameter combinations, thereby selecting the optimal parameter configuration. In practice, automated hyperparameter optimization techniques, such as Bayesian optimization, can also be used to find the optimal parameter combination to improve the efficiency and effectiveness of model training.

[0102] In some embodiments, the neural network model includes multiple neural network sub-models, each neural network sub-model is used to predict the access volume of a type of data in the area where the local data processing node is located; the neural network sub-model training process of each local data processing node includes:

[0103] Input the visit data of each data type in each data frame in different months into the corresponding neural network sub-model for training;

[0104] A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

[0105] It should be noted that the neural network model consists of multiple sub-models, each of which is used to predict the access volume of a specific type of data in the area where the local data processing node is located. A neural network is a computational model that mimics the connections between neurons in the human brain. It makes predictions or classifications by learning the complex relationships between input data. Here, the neural network is broken down into multiple sub-models, each of which focuses on predicting the access volume of a specific type of data, to improve the accuracy and efficiency of the prediction.

[0106] Specifically, the neural network sub-model training process for each local data processing node involves inputting the monthly visit count data for each data type in each data frame into the corresponding neural network sub-model for training. A data frame is a two-dimensional data structure used to store and process tabular data. In this case, it contains information about the monthly visit counts for various data types. The neural network sub-model makes predictions by learning patterns from this data. A genetic algorithm, a search algorithm that simulates the process of natural selection, is used to optimize the parameters of the neural network sub-model, including the learning rate and the number of hidden layer neurons. The learning rate determines the magnitude of network weight updates, while the number of hidden layer neurons determines the network's complexity.

[0107] Preferably, the training of the neural network sub-model can adopt a variety of optimization techniques to improve performance. For example, batch normalization can be used to speed up the training process, or dropout technology can be used to prevent overfitting.

[0108] Furthermore, early stopping can be used to avoid overtraining, terminating training when performance on the validation set no longer improves. When setting the parameter search space, a finer granularity can be adopted, or domain knowledge can be incorporated to set a more reasonable range. When setting the number of iterations, an adaptive iteration strategy can be employed, dynamically adjusting the number of iterations based on the model's training progress. These optimization measures can further improve the predictive accuracy and robustness of the neural network sub-model.

[0109] In some embodiments, the various data information of the data source stored by the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing data with early creation time.

[0110] It's important to note that the data source information stored by local data processing nodes also includes the creation time of each piece of data. Local data processing nodes sort the creation time of each piece of data from earliest to latest, prioritizing access to data with the earliest creation time. The creation time here refers to the time when the data was first generated or recorded. It's a fundamental data attribute and is often used for data version control and access optimization.

[0111] Specifically, after receiving the data, the local data processing node extracts the creation time of each data item and sorts the data based on this time information. Various sorting algorithms, such as quick sort and merge sort, can be used to ensure that the data is arranged in the order of creation time.

[0112] More specifically, when a data access request arrives, the node will prioritize processing data that was created earlier. This ensures the timeliness and consistency of the data, which is especially important when processing historical data or performing data retrospective analysis.

[0113] Preferably, in addition to sorting by creation time, data access priority can also be adjusted based on data importance, access frequency, or other business rules. For example, for business-critical data, even if it was created later, its priority can be increased to ensure fast access to critical data.

[0114] Furthermore, intelligent prefetching mechanisms can be introduced to preload likely-to-be-accessed data into fast storage areas based on data access patterns and prediction algorithms, reducing latency and improving overall system performance. In practice, this can be combined with data lifecycle management strategies to regularly clean up or archive old data to optimize storage resource usage.

[0115] In some embodiments, the data source is installed with multiple sensors for acquiring environmental monitoring data in real time and uploading the data to a local data processing node; the local data processing node monitors whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the data exceeds the temperature, humidity, and light intensity range suitable for data storage, an alarm is issued to prompt the administrator to take measures to adjust the environment.

[0116] It should be noted that the data source is equipped with multiple sensors to acquire real-time environmental monitoring data. These sensors are devices that monitor and record various parameters of the data source's environment. These sensors detect environmental factors such as temperature, humidity, and light intensity, and upload this data to local data processing nodes. Environmental monitoring data is crucial to ensuring the security and reliability of data storage, as improper environmental conditions can damage data storage media or affect data integrity.

[0117] Specifically, local data processing nodes monitor the temperature, humidity, and light intensity of the data storage environment based on sensor data to ensure they are within suitable ranges for data storage. These parameters are typically set based on the technical specifications and optimal operating conditions of the data storage device. For example, the temperature may need to be maintained between 20-25 degrees Celsius, the humidity may need to be controlled between 45% and 55%, and the light intensity may need to be kept below a certain standard. Based on these set parameter ranges, local data processing nodes monitor the environmental status in real time and compare it with pre-set standards.

[0118] Ideally, if monitored environmental factors exceed the range suitable for data storage, the local data processing node will issue an alarm, prompting the administrator to take measures to adjust the environment. Alarms can be issued in various ways, such as sending emails, SMS notifications, or displaying warning messages on the system interface. After receiving the alarm, the administrator can take appropriate measures, such as adjusting the air conditioning temperature, controlling the humidity, or blocking excessive light.

[0119] Furthermore, the system can also automatically execute some preset adjustment measures, such as starting the dehumidification system or adjusting the curtains, to quickly respond to environmental changes and ensure the stability of the data storage environment and the security of the data.

[0120] The above-mentioned various embodiments of the present invention have the following beneficial effects: the design of the central data processing unit and the local data processing node of the data platform comprehensive analysis and management system described in the present invention can realize efficient processing and intelligent analysis of data. The system constructs a training sample set of the data model and uses the random forest model, the gradient boosting decision tree model GBDT and the neural network model to conduct comprehensive analysis and trend prediction on the data, which can improve the accuracy of the prediction and the efficiency of data processing. In addition, the data mining module of the system can classify the data into different activity levels according to the storage and access frequency of the data, which can optimize the storage and access location of the data, improve data access efficiency and reduce storage costs. This design can also achieve optimal allocation of data resources through the scheduling suggestions of the central data processing unit when the storage of the local data processing node is insufficient.

[0121] By integrating model weight distribution and uploading data access status to local data processing nodes, the system can further improve the predictive accuracy of data models and personalized management of data access. In addition, the system analyzes the access duration of each storage location within the data source and classifies the access efficiency of storage locations, providing more reasonable storage location recommendations for high-frequency access data, general data, and low-frequency access data. The system can also prioritize access based on the creation time of the data, improving the timeliness of data access. At the same time, sensors monitor the data storage environment in real time to ensure that the temperature, humidity, light intensity and other conditions of the data storage are suitable. It can promptly issue alarms and prompt administrators to take measures, thereby improving data security and reliability. These designs give the entire data platform significant intelligent and efficient advantages in data processing, analysis, storage, and environmental monitoring.

[0122] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0123] The above descriptions are merely some preferred embodiments of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in the embodiments of the present invention.

Claims

1. A comprehensive analysis and management system for a data platform, characterized in that the system include: Central data processing unit, local data processing nodes; Each local data processing node sends the data obtained from its own data source to the central data processing unit; The central data processing unit includes a data analysis module and a data mining module; The data analysis module constructs a training sample set of the data model based on various types of data sent by each local data processing node, and uses the training sample set to train the data model to obtain a trained data model; The central data processing unit predicts the future trends of various types of data of each local data processing node based on the trained data model; The data mining module obtains the data storage and access patterns according to the storage and access frequencies of various types of data at each local data processing node, and classifies various types of data according to the future trends, storage and access patterns of various types of data at each local data processing node to obtain the data classification results of each local data processing node; The central data processing unit sends the trained data model and model parameters to the corresponding local data processing nodes; the local data processing nodes predict the future data trends of their respective data sources based on the trained data model and then make optimization suggestions; The central data processing unit sends the data classification results of each local data processing node output by the data mining module to each local data processing node. Each local data processing node provides data storage and access location recommendations based on the data classification results. Each local data processing node also uploads the access duration of each storage location within the stored data source to the central data processing unit. The central data processing unit calculates the access efficiency of each storage location based on the access time of each storage location, and divides the internal storage locations into fast access areas, general access areas, and low-speed access areas based on the access efficiency. The central data processing unit sends the internal storage location divisions of each data source to the local data processing nodes. The local data processing nodes combine the data classification results sent by the central data processing unit to provide data storage location recommendations. High-frequency access data is placed in the fast access area, and ordinary data is placed in the general access area. Infrequently accessed data is placed in the low-speed access area.

2. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data mining module classifies the data through the following steps: Step 1: Collect the storage and access frequencies of various types of data at each local data processing node; Step 2: Analyze storage and access frequency to determine the activity level of the data; Step 3: Divide the data into three categories: high-frequency access data, normal data, and low-frequency access data based on the data's future trends and activity levels. Step 4: Use the classification results to guide the optimization of data storage and access locations to improve data access efficiency and reduce storage costs.

3. The data platform comprehensive analysis and management system according to claim 1, characterized in that: When a local data processing node has a low-frequency access data demand and insufficient storage, it sends data shortage information to the central data processing unit. The central data processing unit will search the storage of other local data processing nodes. If the low-frequency access data exists in the storage of other local data processing nodes, the central data processing unit will issue a scheduling suggestion to schedule the low-frequency access data from the storage of other local data processing nodes to the local data processing node in need.

4. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data model is an integrated model of a random forest model, a gradient boosted decision tree model GBDT, and a neural network model; The weights are assigned based on the prediction accuracy of the three models. The weights for each model are: Among them, k = 1, 2, 3, is the prediction accuracy of each model.

5. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The access data of various types of data sent by each local data processing node includes: the number of accesses to each type of data per month; The central data processing unit constructs a training sample set for the corresponding data model based on the number of visits to each type of data each month, including: The number of accesses to different types of data in different months is recorded using tensors, which serve as training sample sets for the neural network model. The number of visits to different types of data in different months is formed into a data frame, which is used as the training sample set for the random forest and GBDT models.

6. The data platform comprehensive analysis and management system according to claim 5, characterized in that: The random forest model training process includes: Determine the range of the number and depth of model parameter trees, where the number and depth of trees are both integers; when initializing the random forest model, input the initial values ​​of the number and depth of trees and the time period length s into the model, and also input the data frame dataframe into the model; then call the training function to train the random forest model, and determine the number and depth of parameter trees through the grid search method, that is, under different combinations of the number and depth of trees, calculate the mean square error between the predicted value and the true value of the data access volume, and find the combination of the number and depth of trees when the sum of the mean square errors of various data is minimized as the required random forest model parameter value.

7. The data platform comprehensive analysis and management system according to claim 6, characterized in that: The neural network model includes multiple neural network sub-models, each of which is used to predict the access volume of a type of data in the area where the local data processing node is located. The training process of the neural network sub-model of each local data processing node includes: Input the visit data of each data type in each data frame in different months into the corresponding neural network sub-model for training; A genetic algorithm is used to determine the parameters of each neural network sub-model, and the search space range of the parameter learning rate and the number of hidden layer neurons is defined respectively. Within the set number of iterations, the mean square error between the predicted value and the true value of the data access volume is calculated; the parameters when the mean square error takes the minimum value are taken as the optimal parameter learning rate and the number of hidden layer neurons, and then the trained neural network sub-models are obtained.

8. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The various data information of the data source stored in the local data processing node also includes the creation time of each data. The local data processing node sorts the creation time of each data from early to late, and gives priority to accessing the data with the earliest creation time.

9. The data platform comprehensive analysis and management system according to claim 1, characterized in that: The data source is equipped with multiple sensors to obtain environmental monitoring data in real time and upload the data to the local data processing node; the local data processing node will monitor whether the data exceeds the temperature, humidity, and light intensity range suitable for data storage. If the temperature, humidity, and light intensity range are exceeded, an alarm will be issued to prompt the administrator to take measures to adjust the environment.

Images