Cold and hot data identification and evaluation method based on cost-sensitive learning

Abstract

The invention discloses a cold and hot data identification and evaluation method based on cost-sensitive learning, and the method is characterized in that the method comprises the steps: constructing a power grid data feature sample set; based on the power grid data feature sample set, a cost sensitive learning algorithm is adopted to construct a classification model, and the cost sensitive learning algorithm applies differential penalty to cold and hot data misclassification by introducing an asymmetric cost matrix; collecting power grid operation data in real time, and generating a to-be-identified sample feature vector; inputting the to-be-identified sample feature vector into the classification model, and outputting a cold and hot data classification result and classification confidence; and dynamically adjusting a data storage strategy and a computing resource allocation scheme according to the output classification result and the classification confidence thereof. According to the method, the asymmetric cost matrix is introduced, the probability that hot data is misjudged as cold data is effectively reduced, the accuracy of cold and hot data classification is improved, and therefore a data storage strategy and computing resource allocation are more accurately optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of power data management, and in particular to a method for identifying and evaluating hot and cold data based on cost-sensitive learning. Background Art

[0002] With the rapid development of smart grids, the amount of grid data is exploding. Identifying the hot and cold characteristics of data is crucial for efficient grid management and optimized operation. During grid operation, the hot and cold characteristics of data directly impact data storage strategies and the allocation of computing resources. Hot data typically requires fast access and processing, while cold data can be stored on lower-cost storage media.

[0003] In existing technologies, methods based on access frequency or simple rules are usually used to distinguish between hot and cold data. However, these methods often ignore the importance of data and the cost of misclassification, resulting in inaccurate recognition results. Crucially, existing methods generally adopt a symmetrical misclassification processing mechanism and fail to fully consider the differentiated impacts brought about by misclassification of different types of data. In actual operation, misclassifying real-time control data as cold data may cause serious operational risks, the cost of which is far higher than the waste of resources caused by misclassifying archived data as hot data. Summary of the Invention

[0004] The purpose of the present invention is to provide a hot and cold data identification and evaluation method based on cost-sensitive learning to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying and evaluating hot and cold data based on cost-sensitive learning, characterized by comprising:

[0006] Constructing a power grid data feature sample set, which includes multi-dimensional state features of multiple data objects and preset hot and cold data classification labels;

[0007] Based on the characteristic sample set of power grid data, a classification model is constructed using a cost-sensitive learning algorithm. The cost-sensitive learning algorithm imposes differential penalties on the misclassification of hot and cold data by introducing an asymmetric cost matrix.

[0008] Collect power grid operation data in real time, perform feature extraction and data standardization, and generate feature vectors of samples to be identified;

[0009] Input the feature vector of the sample to be identified into the classification model, and output the classification results of hot and cold data and their classification confidence;

[0010] According to the output classification results and their classification confidence, the data storage strategy and computing resource allocation plan are dynamically adjusted to achieve the optimal configuration of power grid operation resources.

[0011] In a further embodiment, the asymmetric cost matrix satisfies that the cost weight of hot data misclassified as cold data is higher than the cost weight of cold data misclassified as hot data.

[0012] In a further embodiment, the step of constructing a power grid data feature sample set includes:

[0013] Collect historical power grid operation data and extract multi-dimensional state characteristics of the historical power grid operation data;

[0014] Based on preset access frequency thresholds and business importance rules, historical power grid operation data is labeled as cold data or hot data classification labels;

[0015] The SMOTE oversampling algorithm is used to enhance the data of minority class samples and construct a class-balanced power grid data feature sample set.

[0016] In a further embodiment, the multidimensional state features include dynamic features and static features, wherein the dynamic features include data access frequency, most recent access time interval, and associated computing task priority, and the static features include data type, data generation source, and active storage time threshold.

[0017] In a further embodiment, the data storage strategy and computing resource allocation scheme are dynamically adjusted based on the output classification results and their classification confidence levels to achieve optimal configuration of power grid operation resources, including:

[0018] The output classification result and its classification confidence are embedded into the preset strategy generation prompt template to obtain the strategy generation prompt instruction;

[0019] The strategy generation prompt instruction is input into the large language model to control the large language model to generate the corresponding dynamically adjusted data storage strategy and computing resource allocation plan.

[0020] In a further embodiment, the specific implementation of dynamically adjusting the data storage strategy and computing resource allocation scheme includes:

[0021] Migrating data objects classified as hot data and having a confidence level higher than a preset first threshold to high-speed storage media and allocating high-priority computing resources;

[0022] Data objects classified as cold data and having a confidence level higher than a preset second threshold are downgraded to low-cost storage media, and their computing resource allocation weights are reduced.

[0023] In a further embodiment, after outputting the classification result and its classification confidence, the following steps are performed:

[0024] Based on the classification results and classification confidence output by the classification model, a data value assessment index is constructed. Based on the data value assessment index and combined with the real-time operation status data of the power grid, the data access popularity trend within the future preset time period is predicted;

[0025] Based on the prediction results, storage resources are pre-allocated and optimized to pre-load data objects predicted to be hot data into the cache and reserve corresponding computing resources.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] By introducing an asymmetric cost matrix, the accuracy of hot and cold data classification is significantly improved, effectively reducing the probability of hot data being misclassified as cold. Furthermore, the present invention can collect power grid operation data in real time and dynamically adjust data storage strategies and computing resource allocation schemes based on the grid's real-time operating status, ensuring optimal allocation of power grid operating resources and adapting to dynamic changes in grid operation. Furthermore, by migrating hot data to high-speed storage media and allocating high-priority computing resources, and relegating cold data to low-cost storage media and reducing its computing resource allocation weight, the present invention effectively improves the utilization efficiency of storage and computing resources and reduces power grid operating costs. By combining a large language model to generate dynamic adjustment strategies, the present invention enhances the flexibility and adaptability of data management, enabling flexible adjustment of data management strategies based on different power grid operation scenarios. Furthermore, by constructing data value assessment indicators based on classification results and confidence levels, the present invention can predict data access popularity trends within a preset time period and pre-optimize storage resource allocation, further improving the efficiency and reliability of power grid operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0030] Figure 1 Schematic diagram of a typical embodiment of the method for identifying and evaluating hot and cold data based on cost-sensitive learning according to the present invention;

[0031] Figure 2 A schematic diagram of a process for constructing a power grid data feature sample set in an embodiment of the present invention;

[0032] Figure 3A schematic diagram of a process for generating corresponding strategies and solutions based on a large language model in an embodiment of the present invention;

[0033] Figure 4 Schematic diagram of a process for predicting data access popularity trends within a future preset time period in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0035] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] See also Figure 1 As shown, a method for identifying and evaluating hot and cold data based on cost-sensitive learning in an embodiment of the present invention includes:

[0037] Step S100: constructing a power grid data feature sample set, wherein the power grid data feature sample set includes multi-dimensional state features of multiple data objects and preset hot and cold data classification labels;

[0038] Step S200: Based on the power grid data feature sample set, a classification model is constructed using a cost-sensitive learning algorithm, wherein the cost-sensitive learning algorithm imposes differential penalties on misclassification of hot and cold data by introducing an asymmetric cost matrix;

[0039] Step S300: collecting grid operation data in real time, performing feature extraction and data standardization processing, and generating feature vectors of samples to be identified;

[0040] Step S400: Input the feature vector of the sample to be identified into the classification model, and output the classification result of hot and cold data and its classification confidence;

[0041] Step S500: Dynamically adjust the data storage strategy and computing resource allocation plan based on the output classification results and their classification confidence levels to achieve optimal configuration of power grid operation resources.

[0042] In this embodiment, by constructing a grid data feature sample set, using a cost-sensitive learning algorithm to build a classification model, and dynamically adjusting the data storage strategy and computing resource allocation plan based on the real-time collected grid operation data, the optimal configuration of grid operation resources is achieved.

[0043] Specifically, a power grid data feature sample set is first constructed. This sample set contains multidimensional state features of multiple data objects and preset hot and cold data classification labels. These multidimensional state features include data access frequency, recent access time interval, associated computing task priority, data type, data generation source, and storage duration threshold; these features comprehensively reflect the hot and cold characteristics of the data. Then, based on this constructed power grid data feature sample set, a classification model is constructed using a cost-sensitive learning algorithm. The algorithm introduces an asymmetric cost matrix to impose differential penalties on misclassification of hot and cold data, ensuring that the model pays more attention to the correct classification of hot data during training, thereby improving the accuracy and reliability of the classification model. Then, power grid operation data is collected in real time, and feature extraction and data normalization are performed to generate feature vectors of samples to be identified. This step ensures that the data input to the classification model has a unified format and scale, improving the recognition efficiency and accuracy of the model. Subsequently, the generated feature vectors of samples to be identified are input into the classification model, which outputs the classification results of the hot and cold data and their classification confidence. The classification confidence reflects the model's confidence in the classification results and provides an important basis for subsequent resource adjustments. Finally, based on the output classification results and their classification confidence, the data storage strategy and computing resource allocation scheme are dynamically adjusted. Specifically, data objects classified as hot data with high confidence are migrated to high-speed storage media and allocated high-priority computing resources. Data objects classified as cold data with high confidence are downgraded to low-cost storage media and their computing resource allocation weight is reduced. Through this embodiment, the optimal allocation of power grid operation resources is achieved, resource utilization efficiency is improved, and operating costs are reduced.

[0044] In addition, the present invention can further construct a data value assessment index based on the classification results output by the classification model and its classification confidence, and combine the real-time operation status data of the power grid to predict the data access popularity trend in a preset time period in the future. According to the prediction results, the storage resources are pre-allocated and optimized in advance, and the data objects predicted to be hot data are pre-loaded into the cache, and the corresponding computing resources are reserved, thereby further improving the efficiency and reliability of the power grid operation. The specific steps can be referred to the subsequent specific implementation methods and will not be repeated here.

[0045] Furthermore, the asymmetric cost matrix satisfies that the cost weight of hot data misidentified as cold data is higher than the cost weight of cold data misidentified as hot data.

[0046] In this embodiment, the asymmetric cost matrix introduced in the cost-sensitive learning algorithm is designed to ensure that the cost weight of misclassifying hot data as cold data is higher than the cost weight of misclassifying cold data as hot data. This design is based on the actual needs and data characteristics of power grid operations. Because hot data is typically accessed more frequently and has higher business importance, if hot data is misclassified as cold data, users may not be able to obtain critical information in a timely manner, affecting the real-time monitoring and decision-making efficiency of the power grid. Therefore, by setting a higher cost weight, the classification model focuses more on reducing the misclassification of hot data during training, thereby improving the accuracy and practicality of the overall classification.

[0047] Further, see Figure 2 As shown, the steps of constructing a power grid data feature sample set include:

[0048] Step S101: Collect historical power grid operation data and extract multi-dimensional state features of the historical power grid operation data;

[0049] Step S102: Based on a preset access frequency threshold and business importance rule, label the historical operation data of the power grid with a cold data or hot data classification label;

[0050] Step S103: Use the SMOTE oversampling algorithm to perform data enhancement on minority class samples to construct a class-balanced power grid data feature sample set.

[0051] In this embodiment, the historical operation data of the power grid is first collected, and multi-dimensional state features are extracted from this data. These features can comprehensively reflect the hot and cold characteristics of the data, providing a rich information basis for subsequent classification model training; then, based on the preset access frequency threshold and business importance rules, the collected historical operation data of the power grid are labeled with cold / hot data classification labels. This step ensures that each data object in the sample set has a clear classification identifier, providing accurate supervision information for the training of the classification model; finally, considering that there may be category imbalance in the power grid data, that is, the number of one type of data in the cold data or hot data is far greater than the other type, the SMOTE oversampling algorithm is used to enhance the data of the minority class samples; in this way, the number of minority class samples can be effectively increased, and a category-balanced power grid data feature sample set can be constructed, thereby improving the performance and generalization ability of the classification model when processing unbalanced data.

[0052] Furthermore, the multidimensional state features include dynamic features and static features, wherein the dynamic features include data access frequency, the most recent access time interval, and the priority of the associated computing task, and the static features include data type, data generation source, and active storage time threshold.

[0053] In this embodiment, the multidimensional state features include dynamic features and static features. Dynamic features focus on reflecting the activity and real-time nature of the data, specifically including data access frequency, that is, the number of times the data is accessed within a specific time window; the time interval between recent accesses, that is, the time elapsed since the last access; and the priority of associated computing tasks, reflecting the degree of association between the data and high-priority computing tasks. Static features describe the inherent properties of the data, including data types, such as measurement data, event records, etc.; data generation sources, which refer to the specific equipment or business processes where the data is generated; and active storage duration thresholds, that is, the length of time the data is expected to be retained. These features together provide a comprehensive and detailed evaluation basis for the hot and cold classification of data.

[0054] Further, see Figure 3 As shown in the figure, based on the output classification results and their classification confidence, the data storage strategy and computing resource allocation plan are dynamically adjusted to achieve the optimal configuration of power grid operation resources, including:

[0055] Step S501: embed the output classification result and its classification confidence into a preset strategy generation prompt template to obtain a strategy generation prompt instruction;

[0056] Step S502: Input the strategy generation prompt instruction into the large language model to control the large language model to generate a corresponding dynamically adjusted data storage strategy and computing resource allocation plan.

[0057] In this embodiment, the hot and cold data classification results output by the classification model and their corresponding classification confidence information are embedded in a pre-designed policy generation prompt template, thereby generating specific policy generation prompt instructions. The policy generation prompt template can convert the classification results and confidence into a clear instruction format, providing a clear input for subsequent policy generation; the generated policy generation prompt instructions are then input into the large language model. The large language model, with its powerful natural language understanding and generation capabilities, can generate specific dynamic adjustment strategies based on the input prompt instructions. The specific dynamic adjustment strategies include but are not limited to the adjustment of data storage location, the selection of storage media, the allocation weight of computing resources, etc., thereby achieving the optimal configuration of power grid operation resources. Through this embodiment, the present invention can flexibly adjust data management and resource allocation strategies based on real-time classification results and confidence levels to adapt to changes in power grid operation status, improve resource utilization efficiency and the stability of power grid operation.

[0058] Furthermore, the specific implementation of the dynamic adjustment of data storage strategy and computing resource allocation scheme includes:

[0059] Migrating data objects classified as hot data and having a confidence level higher than a preset first threshold to high-speed storage media and allocating high-priority computing resources;

[0060] Data objects classified as cold data and having a confidence level higher than a preset second threshold are downgraded to low-cost storage media, and their computing resource allocation weights are reduced.

[0061] In this embodiment, data objects classified as hot data, with a classification confidence level above a preset first threshold, are migrated from their current storage location to high-speed storage media, such as solid-state drives (SSDs) or memory storage, to ensure rapid access and processing. Furthermore, high-priority computing resources, including but not limited to increased CPU time, higher network bandwidth, and prioritized computing task scheduling, are allocated to these hot data, thereby ensuring the real-time processing requirements of these critical data during power grid operations.

[0062] For data objects classified as cold data and whose classification confidence is higher than the preset second threshold, they are downgraded from their current storage location to low-cost storage media, such as mechanical hard disks (HDDs) or cloud storage services, to reduce storage costs. At the same time, the computing resource allocation weight of these cold data is reduced, reducing the occupation of computing resources, thereby optimizing resource allocation and improving overall resource utilization efficiency.

[0063] Further, see Figure 4 As shown, according to the output classification results and their classification confidence, it includes:

[0064] Step S600: constructing a data value assessment index based on the classification results output by the classification model and its classification confidence;

[0065] Step S700: predicting the data access popularity trend within a future preset time period based on the data value evaluation index and combined with the real-time operation status data of the power grid;

[0066] Step S800: Based on the prediction result, pre-allocation optimization is performed on storage resources to pre-load data objects predicted to be hot data into the cache and reserve corresponding computing resources.

[0067] In this embodiment, first, based on the classification results output by the classification model and their classification confidence, a data value assessment index is constructed. This data value assessment index comprehensively considers the hot and cold classification results and confidence of the data, can quantify the value of each data object, and provide a basis for subsequent resource allocation and optimization; then, combined with the real-time operation status data of the power grid, the constructed data value assessment index is used to predict the data access popularity trend within a preset time period in the future. This prediction process takes into account the current operation status of the power grid and the historical data access pattern, and can determine in advance which data may be frequently accessed in the future. Finally, based on the prediction results, storage resources are pre-allocated and optimized. Specifically, data objects predicted to be hot data are pre-loaded into the cache, and corresponding computing resources are reserved to ensure that these data can be quickly accessed and processed when needed. The implementation of this step can further improve the efficiency of power grid operation and the rationality of resource utilization.

[0068] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0069] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope of the present disclosure is indicated by the following claims.

Claims

1. A method for identifying and evaluating hot and cold data based on cost-sensitive learning, characterized in that: The following steps are involved: Constructing a power grid data feature sample set, wherein the power grid data feature sample set includes multi-dimensional state features of multiple data objects and preset hot and cold data classification labels; Based on the power grid data feature sample set, a classification model is constructed using a cost-sensitive learning algorithm, wherein the cost-sensitive learning algorithm imposes differential penalties on misclassification of hot and cold data by introducing an asymmetric cost matrix; Collect power grid operation data in real time, perform feature extraction and data standardization, and generate feature vectors of samples to be identified; Input the feature vector of the sample to be identified into the classification model, and output the classification results of hot and cold data and their classification confidence; According to the output classification results and their classification confidence, the data storage strategy and computing resource allocation plan are dynamically adjusted to achieve the optimal configuration of power grid operation resources.

2. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to claim 1, characterized in that: The asymmetric cost matrix satisfies the requirement that the cost weight of hot data mistakenly identified as cold data is higher than the cost weight of cold data mistakenly identified as hot data.

3. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to claim 1, characterized in that: The step of constructing a power grid data feature sample set includes: Collecting historical operation data of the power grid and extracting multi-dimensional state characteristics of the historical operation data of the power grid; Based on a preset access frequency threshold and business importance rules, labeling the historical operation data of the power grid with cold data or hot data classification labels; The SMOTE oversampling algorithm is used to enhance the data of minority class samples and construct a class-balanced power grid data feature sample set.

4. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to claim 3, characterized in that: The multi-dimensional state features include dynamic features and static features, wherein the dynamic features include data access frequency, the most recent access time interval, and the priority of the associated computing task, and the static features include data type, data generation source, and active storage time threshold.

5. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to claim 1, characterized in that: Based on the output classification results and their classification confidence, the data storage strategy and computing resource allocation plan are dynamically adjusted to achieve optimal configuration of power grid operation resources, including: The output classification result and its classification confidence are embedded into the preset strategy generation prompt template to obtain the strategy generation prompt instruction; The strategy generation prompt instruction is input into the large language model to control the large language model to generate a corresponding dynamically adjusted data storage strategy and computing resource allocation plan.

6. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to claim 1, characterized in that: The specific implementation of the dynamic adjustment of data storage strategy and computing resource allocation scheme includes: Migrating data objects classified as hot data and having a confidence level higher than a preset first threshold to high-speed storage media and allocating high-priority computing resources; Data objects classified as cold data and having a confidence level higher than a preset second threshold are downgraded to low-cost storage media, and their computing resource allocation weights are reduced.

7. The method for identifying and evaluating hot and cold data based on cost-sensitive learning according to any one of claims 1 to 6, characterized in that: According to the output classification results and their classification confidence, including: Constructing a data value assessment indicator based on the classification results output by the classification model and its classification confidence; Based on the data value assessment indicators and combined with the real-time operation status data of the power grid, predict the data access popularity trend within a preset time period in the future; Based on the prediction results, storage resources are pre-allocated and optimized to pre-load data objects predicted to be hot data into the cache and reserve corresponding computing resources.

Images