A digital service resource optimization storage method and system
By hierarchically clustering and classifying the access frequency of policy data and optimizing storage using an integrated classifier, the problem of mismatch in data access frequency in the existing technology is solved, and efficient data storage and access is achieved.
Patent Information
- Application Number
- CN202410515525.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-04-26
AI Technical Summary
When the prior art stores policy data in sharding, the access frequency of conventional inspections and historical archived data does not match, resulting in wasted system resources and it is difficult to achieve efficient cache hit rate and data access efficiency.
By hierarchically clustering the access frequency of policy data, determining the real layer and the merge layer, using an integrated classifier to classify and store data with similar access frequencies, and updating the weight of the base classifier to improve data storage efficiency.
Improve cache hit rate and data access efficiency, optimize the storage method of policy data, and reduce waste of system resources.
Smart Images

Figure CN118395232B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of classified computer technology, and more specifically, to a method and system for optimizing storage of digital service resources in the field of classified computer technology. Background Art
[0002] As policy data management becomes increasingly important, government departments and related institutions generate a large amount of policy-related data every day, including regulatory documents, policy guidance, implementation reports, research and analysis, etc. These data not only need to be preserved for a long time, but also need to be quickly retrieved and analyzed to support policy formulation, implementation supervision and effect evaluation.
[0003] In related technologies, the Adaboost algorithm is used to optimize data sharding and store a large amount of data in different nodes or storage units, thereby reducing the load pressure on a single node. Data is sharded based on data similarity, but some of the similar data is generated by routine inspections and has a high access frequency, while the other part is special inspection or historical archive data and has a low access frequency. Putting these data in the same slice results in a large waste of system resources. Summary of the invention
[0004] The present application provides a method for optimizing storage of digital service resources. The method classifies the access frequency of policy data and puts policy data with similar access frequency into the same piece, thereby using an integrated classifier to classify and store real-time policy data, which can improve cache hit rate and data access efficiency.
[0005] In a first aspect, an embodiment of the present application provides a method for optimizing storage of digital service resources, the method comprising:
[0006] Frequency of access to obtain policy data;
[0007] hierarchically clustering the policy data based on the access frequency to obtain multi-layer clustering results;
[0008] Determining a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data;
[0009] Based on the true layer and the merged layer, determining the category recognition capabilities of a plurality of base classifiers for classifying the merged layer;
[0010] Based on the category recognition capability and the merging layer, the multiple base classifiers are integrated to obtain an integrated classifier;
[0011] In response to the received real-time policy data, the real-time policy data is classified and stored using the integrated classifier.
[0012] In the above scheme, by performing hierarchical clustering on the access frequency of policy data, the characteristics of policy data with similar access frequencies can be obtained. Afterwards, according to the obtained clustering results and the frequency category labels of the policy data, the multiple layers in the hierarchical clustering are divided into the real layer and the merged layer; and through the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer are calculated; according to the category recognition capabilities and the merged layer, multiple base classifiers are integrated to obtain an integrated classifier; in this way, the weights of the base classifiers are updated by the category recognition capabilities of the base classifiers, and multiple base classifiers are integrated according to the updated weights to obtain an integrated classifier, so that the integrated classification puts the policy data with similar access frequencies into the same piece; thus, when receiving real-time policy data, the integrated classifier is used to classify and store the real-time policy data, which can improve the cache hit rate and data access efficiency.
[0013] In combination with the first aspect, in some possible implementations, determining the real layer and the merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data includes:
[0014] Determining a classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels;
[0015] The true layer and the merged layer are determined among the multiple layers based on the classification error of each layer.
[0016] In the above scheme, by comparing the clustering results of multiple layers with the frequency category labels, the classification error of each layer is obtained, and then the real layer and the merged layer can be quickly and accurately divided from the multiple layers according to the classification error of each layer.
[0017] In combination with the first aspect, in some possible implementations, determining the classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels includes:
[0018] For each node category in each layer, determining a plurality of actual categories corresponding to each node category in the frequency category label;
[0019] Determine the data volume of each actual category in the multiple actual categories to obtain multiple data volumes;
[0020] determining a ratio between a maximum value and a second maximum value among the plurality of data amounts;
[0021] Determining a classification error of each node category based on the ratio and the number of categories of the multiple actual categories;
[0022] The classification errors of multiple node categories in each layer are averaged to obtain the classification error of each layer.
[0023] In the above scheme, by determining the multiple actual categories corresponding to each node category in each layer in the frequency category label, the category data of the multiple actual categories are compared with the data amount in each actual category to obtain the classification error of each node category; thus, the classification error of the layer can be quickly obtained by taking the average of the classification errors of the multiple node categories in the layer.
[0024] In combination with the first aspect, in some possible implementations, determining the true layer and the merged layer in the multiple layers based on the classification error of each layer includes:
[0025] Determining the layer with the smallest classification error among the multiple layers as the true layer;
[0026] The merged layer is determined among the multiple layers based on the true layer.
[0027] In the above scheme, the layer with the smallest classification error is taken as the true layer, so that the classification result of the true layer is closest to the actual category division, and the classification result of the merged layer can better integrate the classification results of multiple nodes in the true layer.
[0028] In combination with the first aspect, in some possible implementations, determining, based on the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer includes:
[0029] Input the policy data into any base classifier among the multiple base classifiers to obtain a first misclassification number of the base classifier and a second misclassification number of each node category in the merged layer by the base classifier;
[0030] Input the policy data corresponding to each node category in the merged layer into any base classifier to obtain the third number of misclassifications and the number of misclassified subdivision categories of any base classifier; wherein the number of misclassified subdivision categories is the number of subdivision categories of each node category misclassified into other subdivision categories by any base classifier; the subdivision category is the node category corresponding to each node category in the merged layer in the real layer;
[0031] Based on a first ratio between the second misclassification number and the first misclassification number, and a second ratio between the number of misclassified subdivision categories and the third misclassification number, category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
[0032] In the above scheme, the ability of the base classifier to identify the merged layer is determined by comparing the misclassification of the base classifier for all policy data and the misclassification of the base classifier for any node category, so that the category recognition ability of the base classifier can be obtained simply and quickly.
[0033] In combination with the first aspect, in some possible implementations, determining the category recognition capabilities of the multiple base classifiers for classifying the merged layer based on a first ratio between the second number of misclassifications and the first number of misclassifications, and a second ratio between the number of misclassified subdivision categories and the third number of misclassifications, includes:
[0034] The first ratio and the second ratio are merged to obtain the candidate category recognition capability of any base classifier for classifying each node category in the merged layer;
[0035] Based on the candidate category recognition capabilities corresponding to the multiple node categories in the merged layer, the category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
[0036] In the above scheme, for any base classifier, the category recognition ability of the base classifier for a merged layer can be obtained by averaging the candidate category recognition abilities of the base classifier for classifying each node category in a merged layer; in this way, the category recognition ability of the base classifier for classification can be accurately analyzed and the computational complexity can be reduced.
[0037] In combination with the first aspect, in some possible implementations, the integrating the multiple base classifiers based on the category recognition capability and the merging layer to obtain an integrated classifier includes:
[0038] Determining a first number of nodes in the real layer and a second number of nodes in the merged layer;
[0039] Determining, based on the first number of nodes and the second number of nodes, a candidate weight of a category recognition capability of any base classifier among the plurality of base classifiers for classifying the merged layer;
[0040] The candidate weights are combined with the category recognition capability of any base classifier to obtain a weight coefficient of any base classifier;
[0041] Based on the weight coefficient, the initial weight of any base classifier is updated to obtain the target weight of any base classifier;
[0042] Based on the target weights respectively corresponding to the multiple base classifiers, the multiple base classifiers are integrated to obtain the integrated classifier.
[0043] In combination with the first aspect above, in some possible implementations, after obtaining the access frequency of the policy data, the method further includes:
[0044] Based on the access frequency, the policy data is segmented by multiple thresholds to obtain multiple segmentation points;
[0045] Determine candidate frequency values of multiple categories based on the frequency values within each two adjacent segmentation points among the multiple segmentation points;
[0046] Based on the difference between the candidate frequency value of each category and the candidate frequency values of other categories, the frequency category label of each category is determined; wherein the other categories are categories other than each category in the multiple categories.
[0047] In the above scheme, the access frequency of policy data is segmented by multi-threshold segmentation processing, so that the access frequency is divided into multiple categories according to the frequency value within the segmentation point, and a frequency category label is set for each category according to the difference between the candidate frequency value of each category and the candidate frequency values of other categories; thereby, the access frequency can be accurately divided into multiple categories, and the frequency category labels of each category can be accurately marked.
[0048] In combination with the first aspect above, in some possible implementations, the hierarchical clustering of the policy data based on the access frequency to obtain multi-layer clustering results includes:
[0049] Performing word vector conversion on the policy data to obtain a converted vector;
[0050] Hierarchical clustering is performed on the access frequencies of the converted vectors to obtain the multi-layer clustering results.
[0051] In the above scheme, by converting the policy data into word vectors, hierarchical clustering can be performed on the converted vectors more accurately, thereby obtaining multi-layer clustering results.
[0052] In a second aspect, a digital service resource optimization storage system is provided, the system comprising:
[0053] An acquisition module, used to obtain the access frequency of policy data;
[0054] A clustering module, used for performing hierarchical clustering on the policy data based on the access frequency to obtain multi-layer clustering results;
[0055] A first determination module, configured to determine a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data;
[0056] A second determination module is used to determine, based on the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer;
[0057] An integration module, configured to integrate the plurality of base classifiers based on the category recognition capability and the merging layer to obtain an integrated classifier;
[0058] The classification module is used to classify and store the real-time policy data in response to the received real-time policy data using the integrated classifier.
[0059] In a third aspect, a server is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the device executes the method in the first aspect or any possible implementation of the first aspect.
[0060] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.
[0061] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a schematic diagram of an implementation environment of a digital service resource optimization storage system provided in an embodiment of the present application;
[0063] Figure 2 It is a schematic diagram of the implementation process of a digital service resource optimization storage method provided in an embodiment of the present application;
[0064] Figure 3 This is another implementation flow diagram of a method for optimizing storage of digital service resources provided in an embodiment of the present application;
[0065] Figure 4 This is another implementation flow diagram of a method for optimizing storage of digital service resources provided in an embodiment of the present application;
[0066] Figure 5 This is another implementation flow diagram of a method for optimizing storage of digital service resources provided in an embodiment of the present application;
[0067] Figure 6This is the composition structure intention of a digital service resource optimization storage system provided by an embodiment of the present application;
[0068] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0069] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0070] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.
[0071] The implementation environment of the embodiment of the present application is introduced as follows. Figure 1 As shown, the implementation environment of the digital service resource optimization storage system provided in the embodiment of the present application includes a data acquisition terminal 101, a server 102, and a policy data storage terminal 103.
[0072] The data acquisition terminal 101 and the policy data storage terminal 103 are connected via a wireless network, and the policy data storage terminal 103 is connected to the server 102 via a wireless or wired network. The data acquisition terminal 101 can be any type of terminal, such as a data acquisition sensor or an image acquisition terminal. The data acquisition terminal 101 collects policy data and uploads it to the server 102; the server 102 determines the access frequency of the policy data and hierarchically clusters the policy data according to the access frequency to obtain multi-layer clustering results; then, based on the multi-layer clustering results and the frequency category labels of the policy data, the real layer and the merged layer in the multi-layer are determined; based on the real layer and the merged layer, the category recognition ability of multiple base classifiers to classify the merged layer is determined; and based on the category recognition ability and the merged layer, multiple base classifiers are integrated to obtain an integrated classifier; and the integrated classifier is uploaded to the policy data storage terminal 103, so that the policy data storage terminal 103 uses the integrated classifier to classify and store the received real-time policy data to improve the cache hit rate and data access efficiency.
[0073] Server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.
[0074] The following is an introduction to the technical solution provided by the embodiment of the present application. The embodiment of the present application provides a method for optimizing storage of digital service resources. Figure 2 , Figure 2 : is a schematic diagram of an implementation flow of a digital service resource optimization storage method provided in an embodiment of the present application, the method comprising:
[0075] 201,Access frequency of obtaining policy data.
[0076] Here, policy data can be any type of policy data generated by any agency, such as regulatory documents, policy guidance documents, implementation report documents, and supervision and inspection documents. The access frequency of policy data indicates the number of times relevant agency staff access the policy data within a unit of time.
[0077] In some possible implementations, the access frequency of policy data is segmented by multi-threshold segmentation processing, so that the access frequency is divided into multiple categories, and a frequency category label is set for each category according to the difference between the candidate frequency value of each category and the candidate frequency value of other categories; thus, the access frequency can be accurately divided into multiple categories, and the frequency category labels of each category can be accurately marked, which can be achieved through the following process:
[0078] Firstly, based on the access frequency, the policy data is segmented using multiple thresholds to obtain multiple segmentation points.
[0079] Here, after obtaining the access frequency of each policy data, the access frequency histogram of all data is obtained by statistics. The access frequency histogram is segmented by a multi-threshold processing method (OSTU) to obtain multiple segmentation points.
[0080] Secondly, based on the frequency values within every two adjacent segmentation points among the plurality of segmentation points, candidate frequency values of a plurality of categories are determined.
[0081] Here, the frequency values within two adjacent segmentation points are taken as candidate frequency values of a category, and the data corresponding to the candidate frequency values are taken as data points of the category, thereby obtaining multiple categories.
[0082] Finally, based on the difference between the candidate frequency value of each category and the candidate frequency values of other categories, the frequency category label of each category is determined.
[0083] The other categories are categories other than each category in the multiple categories. The frequency category label indicates the true value of the frequency category to which each category belongs.
[0084] In some possible implementations, the sum of the absolute values of the differences between each candidate frequency value and other candidate frequency values in each category is calculated, and the frequency value corresponding to the smallest sum result is used as the category frequency value of the category, and the category frequency value is used as the frequency category label of the data points in each category, that is, multiple categories and frequency category labels for each category are obtained.
[0085] 202 , hierarchical clustering is performed on the policy data based on the access frequency to obtain multi-layer clustering results.
[0086] Here, the multi-layer clustering result is the categories included in each layer of the hierarchical clustering tree. By performing hierarchical clustering on the policy data according to the access frequency, a hierarchical clustering tree can be obtained; wherein each layer in the hierarchical clustering tree includes multiple node categories. Since the frequently accessed policy data often corresponds to some common or important data feature categories, the policy data is hierarchically clustered according to the access frequency, so that the obtained multi-layer clustering result can more accurately represent the policy data with similar access frequencies.
[0087] In some possible implementations, the policy data is converted into word vectors so as to facilitate hierarchical clustering of the converted vectors. That is, the above step 202 can be implemented by the following steps 221 and 222 (not shown):
[0088] 221, perform word vector conversion on the policy data to obtain a converted vector.
[0089] Here, in order to avoid the influence of different types of data (such as different formats of text data and image data) on classification, the same operation is performed on each type of data in the embodiment of the present application. Taking text data as an example, each text data is converted into a vector of the same length through word-to-vector conversion (wordvec), which facilitates the subsequent classification process.
[0090] 222, performing hierarchical clustering on the access frequency of the converted vector to obtain the multi-layer clustering result.
[0091] Here, the access frequency of the converted vector is clustered from bottom to top to obtain a hierarchical clustering tree, which is used as the multi-layer clustering result. In this way, by converting the policy data into word vectors, the converted vectors can be clustered more accurately, thereby obtaining a multi-layer clustering result.
[0092] 203 , determining a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data.
[0093] Here, since the multi-layer clustering result includes multiple frequency categories of policy data, the frequency categories obtained through multi-layer clustering have certain errors. Therefore, by comparing the frequency categories in the multi-layer clustering result with the frequency category labels of policy data, the classification error of each layer in the multi-layer clustering can be obtained; thus, according to the classification error of each layer, the multi-layer is divided into a true layer and a merged layer. The true layer is the layer with the smallest classification error in the multi-layer, and the layers above the true layer are used as merged layers in the multi-layer.
[0094] 204 , based on the real layer and the merged layer, determine the category recognition capabilities of multiple base classifiers for classifying the merged layer.
[0095] Here, the classification recognition capability of the base classifier for classifying the merged layer is used to indicate the accuracy of the base classifier in classifying the access frequency of the policy data in the merged layer. The higher the classification recognition capability, the higher the accuracy of the base classifier in classifying the access frequency of the policy data in the merged layer.
[0096] In some possible implementations, an adaptive boosting algorithm is used to classify the policy data in the merged layer, that is, the merged layer is classified by multiple base classifiers to obtain the classification results of each base classifier. The classification results of the base classifier are compared with the category to which the policy data belongs in the real layer to obtain the category recognition ability of the base classifier to classify the merged layer.
[0097] 205 , based on the category recognition capability and the merging layer, the multiple base classifiers are integrated to obtain an integrated classifier.
[0098] Here, after obtaining the true layer and the merged layer, the weight of the recognition ability of the merged layer is calculated by first determining the ratio between the number of nodes included in the merged layer and the number of nodes included in the true layer. For any base classifier among multiple base classifiers, the product corresponding to the merged layer is obtained by multiplying the category recognition ability of the base classifier to classify the merged layer by the weight of the recognition ability of the merged layer; the products corresponding to each merged layer are summed to obtain the weight coefficient of the base classifier. Afterwards, the weight coefficient is multiplied by the original weight of the base classifier to adjust the original weight of the base classifier to obtain the target weight of the base classifier; finally, the base classifiers are integrated into an integrated classifier through the target weights of each base classifier.
[0099] It should be further explained that Adaboost, as an integrated classifier, can classify data better. Since data is not classified based on data similarity, but data is divided according to similar access frequencies, it is necessary to train the classifier to put the real-time data on the appropriate node.
[0100] 206 , in response to the received real-time policy data, classify and store the real-time policy data using the integrated classifier.
[0101] Here, when real-time policy data is received, the real-time policy data is classified by an integrated classifier and stored in different nodes. That is, when new data needs to be placed in a node, it is classified by a trained integrated classifier, and the real-time policy data is placed in different nodes for storage according to the classification results.
[0102] In the embodiment of the present application, by performing hierarchical clustering on the access frequency of policy data, the characteristics of policy data with similar access frequency can be obtained. Afterwards, according to the obtained clustering results and the frequency category labels of the policy data, the multiple layers in the hierarchical clustering are divided into a real layer and a merged layer; and through the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer are calculated; according to the category recognition capabilities and the merged layer, multiple base classifiers are integrated to obtain an integrated classifier; in this way, the weights of the base classifiers are updated through the category recognition capabilities of the base classifiers, and the multiple base classifiers are integrated according to the updated weights, so that when real-time policy data is received, the integrated classifier is used to classify and store the real-time policy data, thereby improving the cache hit rate and data access efficiency.
[0103] In some embodiments, the classification error of hierarchical clustering is analyzed by using the multi-layer clustering results and the frequency category labels, and then the true layer and the merged layer are determined in the multi-layer. That is, the above step 203 can be performed by Figure 3 The steps shown achieve:
[0104] 301 : Determine the classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels.
[0105] Here, the classification error of each layer in the multi-layer indicates the accuracy of clustering in the hierarchical clustering process; the larger the classification error, the lower the accuracy of clustering in the layer; the smaller the classification error, the higher the accuracy of clustering in the layer. Since each layer in the multi-layer includes multiple node categories, the multiple actual categories of each node category in each layer in the frequency category label are first calculated, and then the classification error of the layer is calculated through the number of actual categories and the amount of data included in each actual category.
[0106] In some possible implementations, the classification error of each layer may be determined by comparing the amount of data of each actual category in the multiple actual categories with the number of categories of the multiple actual categories. That is, the above step 301 may be implemented by steps 311 to 315 (not shown in the figure):
[0107] 311 . For each node category in each layer, determine a plurality of actual categories corresponding to each node category in the frequency category label.
[0108] Here, since each layer includes multiple node categories, for each category in the multiple node categories, the actual category corresponding to the node category is searched in the frequency category label, thereby obtaining multiple actual categories. For example, taking layer A (layer A is any layer in the multiple layers) as an example, for each node category in layer A, the node category includes multiple data, and the actual category corresponding to each of the multiple data is searched in the frequency category label, thereby obtaining multiple actual categories a.
[0109] 312, determine the data volume of each actual category in the multiple actual categories to obtain multiple data volumes.
[0110] Here, among multiple actual categories, the amount of data included in each actual category, that is, the number of bytes included in each actual category, is counted to obtain the amount of data of each actual category.
[0111] 313 , determine a ratio between a maximum value and a second maximum value among the multiple data amounts.
[0112] Here, after obtaining multiple data amounts, the multiple data amounts are sorted, and the maximum value and the second maximum value are obtained therefrom to calculate the ratio between the maximum value and the second maximum value. The ratio can indicate the largerness of the quantity value of the actual category with the largest data amount.
[0113] 314 , determining a classification error of each node category based on the ratio and the number of categories of the multiple actual categories.
[0114] Here, the number of categories of multiple actual categories means the total number of actual categories. The classification error of the node category can be obtained by multiplying the ratio by the number of categories of the actual category.
[0115] 315 , performing averaging processing on the classification errors of the multiple node categories in each layer to obtain the classification error of each layer.
[0116] Here, after obtaining the classification errors of each node category in each layer, the classification errors of each node category are averaged, and the average is used as the classification error of the layer. In this way, by determining the multiple actual categories corresponding to each node category in the frequency category label in each layer, the category data of the multiple actual categories are compared with the data volume in each actual category to obtain the classification error of each node category; thus, the classification error of the layer can be quickly obtained by the average of the classification errors of multiple node categories in the layer.
[0117] In some possible implementations, for different types of data, the policy data is classified into different frequency categories; first, the policy data is classified into different frequency categories, and then different types of data of the same frequency category are put into one piece. The embodiment of the present application is only for the processing process of the same type of data. According to the access frequency of the policy data, the policy data is obtained by a bottom-up hierarchical clustering method to obtain a hierarchical clustering tree. Taking any layer of the hierarchical clustering tree, for example, layer A, the classification error f of each node category in layer A is shown in formula (1):
[0118] f=a*e -b (1);
[0119] In formula (1), the more actual categories there are in the node category, the greater the classification error; among the actual categories in the node category, the greater the value of the category corresponding to the maximum number is, the smaller the classification error is.
[0120] In formula (1), a represents the number of actual categories in the node category. The more categories there are, the more the node category fails to divide the actual categories well, and the greater the classification error. e is a natural constant.
[0121] In formula (1), b represents the maximum value of the number of the largest actual category in the actual category of the node category. The specific calculation method is: first, multiple actual categories in the node category can be obtained, and then the number of each actual category can be obtained. All the numbers form a number sequence, and the maximum value of the maximum value in the sequence is calculated. The ratio of the absolute value of the difference between the maximum value and the second largest value to the maximum value is calculated, and the ratio is used as the maximum value. The classification error of each node category in layer A can be obtained by calculation, and the average of the classification errors of all node categories in layer A is used as the classification error of layer A.
[0122] 302. Determine the true layer and the merged layer in the multiple layers based on the classification error of each layer.
[0123] Here, after obtaining the classification error of each layer, the true layer and the merged layer are determined in the multiple layers by comparing the classification errors of each layer. Taking layer A as an example, the classification error of each node category in layer A can be obtained by calculation, and the mean of all classification errors in layer A is taken as the classification error of the layer. The layer corresponding to the minimum classification error is recorded as the true layer, and the classification result of the true layer is the closest to the actual category division. In this way, by comparing the clustering results of multiple layers with the frequency category labels, the classification error of each one can be obtained, and then the true layer and the merged layer can be quickly and accurately divided from the multiple layers according to the classification errors of each layer.
[0124] In some possible implementations, the layer with the smallest classification error is determined as the true layer in the multiple layers; and the merged layer is determined in the multiple layers based on the true layer. For example, the layer above the true layer in the multiple layers is used as the merged layer, and each node of the merged layer is the union of multiple nodes in the true layer. In this way, the layer with the smallest classification error is used as the true layer, so that the classification result of the true layer is closest to the actual category division, and the classification result of the merged layer can better integrate the classification results of multiple nodes in the true layer.
[0125] In an embodiment of the present application, in a multi-layer clustering result, if the classification result of any layer is close to the true classification result, then this layer is used as the true layer, and the layer above the true layer is used as the merged layer. If the base classifier can better classify the data of each category in the merged layer and divide it into the categories of the corresponding sub-nodes, then the base classifier has a better recognition ability for the scale data. When each base classifier recognizes each category of different layers, the closer the recognition ability is to the true layer, the stronger the recognition ability of the base classifier is, and the larger the weight is given to the base classifier, thereby improving the accuracy of the integrated classifier.
[0126] In some embodiments, the classification of the merged layer by the base classifier is compared with the classification of each node category in the real layer to obtain the category recognition ability of the base classifier, that is, the above step 204 can be performed by Figure 4 The steps shown achieve:
[0127] 401 Input the policy data into any base classifier among a plurality of base classifiers to obtain a first number of misclassifications of any base classifier and a second number of misclassifications of any base classifier for each node category in the merged layer.
[0128] Here, the first number of misclassifications is the number of misclassifications of the base classifier for classifying all policy data, and the first number of misclassifications m can be obtained by inputting all policy data carrying access frequencies into the base classifier. The second number of misclassifications m0 is the number of errors in any base classifier for classifying a node category in the merged layer.
[0129] In a specific example, taking any node category c in the merged layer C and any base classifier j as an example, the node category corresponding to the node category c in the real layer is recorded as the subdivision category of the node category c; all data are input into the base classifier j, and the number of misclassifications of the base classifier j is recorded as m, that is, the first number of misclassifications; the number of misclassifications of the base classifier in the node category c is recorded as m0, that is, the second number of misclassifications, and the ratio of m0 to m is calculated; the ratio The smaller it is, the stronger the base classifier j's ability to recognize samples in node category c is, that is, the stronger its ability to recognize subdivided categories of node category c is.
[0130] 402 , input the policy data corresponding to each node category in the merged layer into any base classifier to obtain a third number of misclassifications and a number of misclassified subdivision categories of any base classifier.
[0131] Among them, the number of misclassified subdivision categories is the number of subdivision categories of each node category that are misclassified into other subdivision categories by any base classifier; the subdivision category is the node category corresponding to each node category in the merged layer in the real layer. For any merged layer, the data in any node category in the merged layer is input into the base classifier, then the number of data in the node category that are misclassified by the base classifier can be counted, that is, the third number of misclassifications, and the number of subdivision categories of the data in the node category that are divided into other subdivision categories by the base classifier, that is, the number of misclassified subdivision categories.
[0132] 403. Determine the category recognition capabilities of the multiple base classifiers for classifying the merged layer based on a first ratio between the second misclassification number and the first misclassification number, and a second ratio between the misclassified subdivision category number and the third misclassification number.
[0133] Here, by calculating the first ratio between the first number of misclassifications and the second number of misclassifications, the proportion of the number of misclassifications of each node category in the merged layer by the base classifier in the total number of misclassifications is represented. And by calculating the second ratio between the third number of misclassifications and the number of misclassified subdivision categories, the proportion of the number of misclassifications of the subdivision categories of the node category by the classifier in the total number of node categories is represented for a node category, that is, the proportion of the number of misclassifications of the node category by the classifier in the total number of node categories, so that by multiplying the first ratio and the second ratio, the category recognition ability of the base classifier for classifying the node category is obtained. Finally, by combining the category recognition abilities of the base classifier for classifying multiple node categories, the category recognition ability of the base classifier for a merged layer can be obtained. In this way, by comparing the misclassification of the base classifier for all policy data and the misclassification of the base classifier for any node category, the ability of the base classifier to identify the merged layer is determined, so that the category recognition ability of the base classifier can be obtained simply and quickly.
[0134] In some possible implementations, for any base classifier, the category recognition capability of the base classifier for the merged layer can be obtained by averaging the candidate category recognition capabilities of the base classifier for classifying each node category in a merged layer; in this way, the category recognition capability of the base classifier for classification can be accurately analyzed, and the computational complexity can be reduced, that is, the above step 403 can be implemented by the following process:
[0135] First, the first ratio and the second ratio are fused to obtain the candidate category recognition capability of any base classifier for classifying each node category in the merged layer.
[0136] Here, for any node category in any merged layer, after obtaining the first ratio and the second ratio corresponding to the node category, the first ratio and the second ratio are multiplied to obtain the category recognition ability of the base classifier for classifying the node category, that is, the candidate category recognition ability.
[0137] Then, based on the candidate category recognition capabilities corresponding to the multiple node categories in the merged layer, the category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
[0138] Here, after obtaining the candidate recognition capabilities of the base classifier for each node category of any merged layer, the candidate recognition capabilities corresponding to these node categories are added and averaged to obtain the category recognition capability of the base classifier for the merged layer. Similarly, the category recognition capabilities of each base classifier for classifying the merged layer can be obtained.
[0139] In some possible implementations, if any base classifier can better divide the data in the merged layer into the category divisions of the nodes in the lower layer, then the recognition ability of the base classifier is better, but because the base classifier only classifies the data but does not divide the categories; therefore, in the embodiments of the present application, the error rate of the base classifier on the data is compared to obtain the ability to recognize the boundaries of the categories, and then the category recognition ability of the base classifier to classify the merged layer is obtained.
[0140] It should be further explained that, taking any node category c in the merged layer C as an example, node category c corresponds to category a and category b in the real layer, and all data are input into the base classifier. Since the base classifier is a weak classifier, there are many misclassifications. If the base classifier has a strong ability to identify a and b, the misclassification rate of samples in a and b is low; therefore, the fewer the number of a and b categories included in the misclassified data, the stronger the base classifier's ability to identify a and b.
[0141] In a specific example, taking any node category c and any base classifier j in the merged layer C as an example, the node category corresponding to the node category c in the true layer is recorded as a subdivision category of the node category c.
[0142] Input the data in node category c into base classifier j. If base classifier j can distinguish the subdivided categories in node category c well, then since base classifier j has learned the dividing boundary between any two subdivided categories, the number of mutual misclassifications accounts for a small proportion of the total number of misclassifications. Based on this, we can obtain the base classifier's ability to recognize the subdivided categories in node category c, that is, the category division ability at the data scale corresponding to node category c.
[0143] Input the data in node category c into the base classifier j, and record the number of misclassifications as N (i.e., the third number of misclassifications); the misclassification of subdivision categories in node category c means that a subdivision category in node category c is misclassified into another subdivision category in node category c. The number of misclassified subdivision categories can be obtained by statistics and recorded as n; the ratio of n to N The smaller it is, the stronger the base classifier j's ability to identify the subdivided categories in node category c is, that is, the stronger the base classifier j's ability to classify the data scale corresponding to node category c is.
[0144] Furthermore, the two ratios of the obtained base classifier j can be and The product of is taken as the classification ability of base classifier j for node category c in the merged layer C, and then the classification ability of base classifier j for each node category in the merged layer C can be obtained. Then, the mean of the classification ability of all nodes in the merged layer C is taken as the category recognition ability of base classifier j for classifying the merged layer. The category recognition ability of each base classifier for classifying each layer of the clustering hierarchy tree can be obtained by calculation. It should be noted that the stronger the recognition ability of the base classifier for the upper layer, the stronger the ability of the base classifier to accurately classify different categories.
[0145] In some embodiments, the initial weights of each base classifier are updated by the real layer and the merged layer obtained by classification according to the access frequency, so as to obtain an integrated classifier capable of classifying according to the access frequency. That is, the above step 205 can be performed by Figure 5 The steps shown achieve:
[0146] 501, determine the number of first nodes in the real layer and the number of second nodes in the merged layer.
[0147] Here, for the i-th merged layer among multiple merges (wherein the i-th merged layer is any merged layer among the multiple merged layers), the number of nodes included in the i-th merged layer is counted to obtain the second number of nodes; and the number of nodes included in the real layer, thereby obtaining the first number of nodes.
[0148] 502 : Determine, based on the first number of nodes and the second number of nodes, a candidate weight of a category recognition capability of any base classifier among the multiple base classifiers for classifying the merged layer.
[0149] Here, the first number of nodes is subtracted from the second number of nodes, and the obtained difference is divided by the first number of nodes, and the obtained quotient is used as a candidate weight for the category recognition ability of the base classifier for classification of the i-th merging layer.
[0150] 503 , the candidate weights are integrated with the category recognition capability of any base classifier to obtain a weight coefficient of any base classifier.
[0151] Here, for each base classifier, the candidate weight of the category recognition ability of the base classifier for any merging layer is multiplied by the category recognition ability of the base classifier to obtain the weight coefficient of the base classifier.
[0152] Since the candidate weight is obtained by subtracting the number of first nodes from the number of second nodes and dividing the difference by the number of first nodes, the candidate weight is multiplied by the category recognition ability of the base classifier as the weight coefficient of the base classifier. A higher weight can be assigned to a base classifier with a higher category recognition ability, thereby improving the accuracy of the integrated classifier.
[0153] 504 , updating the initial weight of any base classifier based on the weight coefficient to obtain a target weight of any base classifier.
[0154] Here, for any base classifier, the weight coefficient of the classifier is multiplied by the initial weight of the classifier to obtain the target weight of the base classifier.
[0155] In some possible implementations, if the classification result of a layer obtained through hierarchical clustering is closer to the actual classification of the data, then which base classifier in the ensemble classifier can better identify and classify the categories of this layer, indicating that the base classifier is more capable of identifying some important policy data features and can autonomously classify policy data features of similar frequency data into one piece. Therefore, a larger weight should be given to the base classifier, which helps to improve the classification accuracy of the ensemble classifier.
[0156] 505. Based on the target weights respectively corresponding to the multiple base classifiers, the multiple base classifiers are integrated to obtain the integrated classifier.
[0157] Here, each base classifier is multiplied by the corresponding target weight and concatenated to obtain the integrated classifier. In this way, for any base classifier, the initial weight of the base classifier is updated by the category recognition ability of the base classifier to classify each merged layer, and the difference in the number of nodes in the merged layer and the real layer. In this way, the access frequency of the policy data is fully considered in the process of updating the initial weight, and the base classifier with higher category recognition ability is given a larger target weight, so that the accuracy of the classification of the base classifier can be improved, and then multiple high-accuracy base classifiers are integrated to further improve the accuracy of the integrated classifier.
[0158] In some possible implementations, taking any base classifier j as an example, the weight coefficient q of the base classifier j is calculated as shown in formula (2):
[0159]
[0160] In formula (2), n represents the number of merged layers; L i represents the recognition ability of base classifier j for the i-th merging layer; represents the weight of the recognition ability of the i-th merged layer, that is, the candidate weight; N1 represents the number of nodes in the real layer of the hierarchical clustering tree, N i Represents the number of nodes in the i-th merging layer. Through the above process, the weight coefficient of each base classifier can be obtained, and the weight coefficient is used as the product coefficient of the original weight of each base classifier, multiplied by the original weight, to obtain the updated weight of each base classifier, and then the integrated classifier is obtained.
[0161] In the embodiment of the present application, the access frequency of policy data is classified to obtain the characteristics of data with similar access frequencies, and then the data with similar access frequencies are placed in the same slice to improve the cache hit rate and data access efficiency.
[0162] The present application embodiment provides a digital service resource optimization storage system, such as Figure 6 As shown, the system 600 includes:
[0163] An acquisition module 601 is used to acquire the access frequency of policy data;
[0164] A clustering module 602, configured to perform hierarchical clustering on the policy data based on the access frequency to obtain multi-layer clustering results;
[0165] A first determination module 603, configured to determine a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data;
[0166] A second determination module 604 is used to determine, based on the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer;
[0167] An integration module 605, configured to integrate the plurality of base classifiers based on the category recognition capability and the merging layer to obtain an integrated classifier;
[0168] The classification module 606 is used to classify and store the real-time policy data using the integrated classifier in response to the received real-time policy data.
[0169] In the above system, the first determination module 603 is also used to determine the classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels; based on the classification error of each layer, determine the true layer and the merged layer in the multiple layers.
[0170] In the above system, the first determination module 603 is also used to determine, for each node category in each layer, multiple actual categories corresponding to each node category in the frequency category label; determine the data volume of each actual category in the multiple actual categories to obtain multiple data volumes; determine the ratio between the maximum value and the second maximum value in the multiple data volumes; determine the classification error of each node category based on the ratio and the number of categories of the multiple actual categories; and average the classification errors of the multiple node categories in each layer to obtain the classification error of each layer.
[0171] In the above system, the first determination module 603 is further used to determine the layer with the smallest classification error among the multiple layers as the true layer; and determine the merged layer among the multiple layers based on the true layer.
[0172] In the above system, the second determination module 604 is further used to input the policy data into any base classifier among the multiple base classifiers to obtain a first number of misclassifications of the any base classifier and a second number of misclassifications of each node category in the merged layer by the any base classifier; input the policy data corresponding to each node category in the merged layer into the any base classifier to obtain a third number of misclassifications and a number of misclassified subdivision categories of the any base classifier; wherein the number of misclassified subdivision categories is the number of subdivision categories of each node category misclassified into other subdivision categories by the any base classifier; the subdivision category is the node category corresponding to each node category in the merged layer in the true layer; based on a first ratio between the second number of misclassifications and the first number of misclassifications and a second ratio between the number of misclassified subdivision categories and the third number of misclassifications, the category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
[0173] In the above system, the second determination module 604 is also used to fuse the first ratio and the second ratio to obtain the candidate category recognition capability of any base classifier to classify each node category in the merged layer; based on the candidate category recognition capabilities corresponding to multiple node categories in the merged layer, determine the category recognition capabilities of the multiple base classifiers to classify the merged layer.
[0174] In the above system, the integration module 605 is also used to determine a first node number of nodes in the real layer and a second node number of nodes in the merged layer; based on the first node number and the second node number, determine the candidate weight of the category recognition ability of any base classifier among the multiple base classifiers for classifying the merged layer; fuse the candidate weight with the category recognition ability of any base classifier to obtain the weight coefficient of any base classifier; update the initial weight of any base classifier based on the weight coefficient to obtain the target weight of any base classifier; based on the target weights corresponding to the multiple base classifiers, integrate the multiple base classifiers to obtain the integrated classifier.
[0175] In the above system, the acquisition module 601 is also used to perform multi-threshold segmentation on the policy data based on the access frequency to obtain multiple segmentation points; determine candidate frequency values of multiple categories based on the frequency values within each two adjacent segmentation points among the multiple segmentation points; determine the frequency category label of each category based on the difference between the candidate frequency value of each category and the candidate frequency values of other categories; wherein the other categories are categories among the multiple categories except for each category.
[0176] In the above system, the clustering module 602 is further used to perform word vector conversion on the policy data to obtain converted vectors; and perform hierarchical clustering on the access frequency of the converted vectors to obtain the multi-layer clustering results.
[0177] Optionally, the transmission medium can be a wired link (such as but not limited to coaxial cable, optical fiber and digital subscriber line (DSL)) or a wireless link (such as but not limited to wireless Fidelity (WIFI), Bluetooth and mobile device network).
[0178] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the digital service resource optimization storage system and the digital service resource optimization storage method embodiment provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0179] Figure 7 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. For example, Figure 7As shown, the computer device 700 includes: a memory 701, a processor 702, and a computer program 703 stored in the memory 701 and running on the processor 702, wherein when the processor 702 executes the computer program 703, the computer device can execute any one of the digital service resource optimization storage methods introduced above.
[0180] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to execute a digital service resource optimization storage method provided in an embodiment of the present application.
[0181] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0182] In the case of dividing each module according to each function, the device may also include a signal uploading module, a determining module, an adjusting module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.
[0183] It should be understood that the device provided in this embodiment is used to execute the above-mentioned digital service resource optimization storage method, and thus can achieve the same effect as the above-mentioned implementation method.
[0184] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a device, the processing module may be used to control and manage the actions of the device. The storage module may be used to support the device to execute mutual program codes, etc.
[0185] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits disclosed in conjunction with the present application. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.
[0186] In addition, the device provided in the embodiments of the present application may specifically be a chip, component or module, and the chip may include a connected processor and memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a digital service resource optimization storage method provided in the above embodiments.
[0187] This embodiment also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a digital service resource optimization storage method provided in the above embodiment.
[0188] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a digital service resource optimization storage method provided by the above embodiment.
[0189] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.
[0190] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0191] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0192] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for optimizing storage of digital service resources, It is characterized in that The method comprises: Frequency of access to obtain policy data; Performing hierarchical clustering on the policy data based on the access frequency to obtain multi-layer clustering results; Determining a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data; Based on the true layer and the merged layer, determining the category recognition capabilities of a plurality of base classifiers for classifying the merged layer; Based on the category recognition capability and the merging layer, the multiple base classifiers are integrated to obtain an integrated classifier; In response to the received real-time policy data, classify and store the real-time policy data using the integrated classifier; The method for obtaining the frequency category label specifically includes: Based on the access frequency, the policy data is segmented by multiple thresholds to obtain multiple segmentation points, specifically including: After obtaining the access frequency of each policy data, the access frequency histogram of all data is obtained by statistics; the access frequency histogram is segmented by a multi-threshold processing method to obtain multiple segmentation points; Determining candidate frequency values of multiple categories based on the frequency values of every two adjacent segmentation points among the multiple segmentation points specifically includes: The frequency values between two adjacent segmentation points are taken as candidate frequency values of a category, and the data corresponding to the candidate frequency values are taken as data points of the category, thereby obtaining multiple categories; Determining the frequency category label of each category based on the difference between the candidate frequency value of each category and the candidate frequency values of other categories specifically includes: The other categories are categories other than each of the categories in the multiple categories; Calculate the sum of the absolute values of the differences between each candidate frequency value and other candidate frequency values in each category, use the frequency value corresponding to the minimum sum result as the category frequency value of the category, use the category frequency value as the frequency category label of the data point in each category, and obtain multiple categories and the frequency category label of each category; The determining of the real layer and the merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data comprises: Determining a classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels; Determining the true layer and the merged layer among the multiple layers based on the classification error of each layer; The determining the classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels includes: For each node category in each layer, determining a plurality of actual categories corresponding to each node category in the frequency category label; Determine the data volume of each actual category in the multiple actual categories to obtain multiple data volumes; determining a ratio between a maximum value and a second maximum value among the plurality of data amounts; Determining a classification error of each node category based on the ratio and the number of categories of the multiple actual categories; Performing averaging processing on the classification errors of multiple node categories in each layer to obtain the classification error of each layer; The determining the true layer and the merged layer in the multiple layers based on the classification error of each layer comprises: Determining the layer with the smallest classification error among the multiple layers as the true layer; The merged layer is determined among the multiple layers based on the true layer.
2. A digital service resource optimization storage method according to claim 1, It is characterized in that The step of determining, based on the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer comprises: Input the policy data into any base classifier among the multiple base classifiers to obtain a first misclassification number of the base classifier and a second misclassification number of each node category in the merged layer by the base classifier; Input the policy data corresponding to each node category in the merged layer into any base classifier to obtain the third number of misclassifications and the number of misclassified subdivision categories of any base classifier; wherein the number of misclassified subdivision categories is the number of subdivision categories of each node category misclassified into other subdivision categories by any base classifier; the subdivision category is the node category corresponding to each node category in the merged layer in the real layer; Based on a first ratio between the second misclassification number and the first misclassification number, and a second ratio between the number of misclassified subdivision categories and the third misclassification number, category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
3. A digital service resource optimization storage method according to claim 2, It is characterized in that The determining, based on a first ratio between the second misclassification number and the first misclassification number, and a second ratio between the number of misclassified subdivision categories and the third misclassification number, category recognition capabilities of the multiple base classifiers for classifying the merged layer comprises: The first ratio and the second ratio are merged to obtain the candidate category recognition capability of any base classifier for classifying each node category in the merged layer; Based on the candidate category recognition capabilities corresponding to the multiple node categories in the merged layer, the category recognition capabilities of the multiple base classifiers for classifying the merged layer are determined.
4. A method for optimizing storage of digital service resources according to claim 1, It is characterized in that The step of integrating the multiple base classifiers based on the category recognition capability and the merging layer to obtain an integrated classifier comprises: Determining a first number of nodes in the real layer and a second number of nodes in the merged layer; Determining, based on the first number of nodes and the second number of nodes, a candidate weight of a category recognition capability of any base classifier among the plurality of base classifiers for classifying the merged layer; The candidate weights are combined with the category recognition capability of any base classifier to obtain a weight coefficient of any base classifier; Based on the weight coefficient, the initial weight of any base classifier is updated to obtain the target weight of any base classifier; Based on the target weights respectively corresponding to the multiple base classifiers, the multiple base classifiers are integrated to obtain the integrated classifier.
5. A digital service resource optimization storage method according to claim 1, It is characterized in that The hierarchical clustering of the policy data based on the access frequency to obtain multi-layer clustering results includes: Performing word vector conversion on the policy data to obtain a converted vector; Hierarchical clustering is performed on the access frequencies of the converted vectors to obtain the multi-layer clustering results.
6. A digital service resource optimization storage system, It is characterized in that The system comprises: An acquisition module, used to obtain the access frequency of policy data; A clustering module, used for performing hierarchical clustering on the policy data based on the access frequency to obtain multi-layer clustering results; A first determination module, configured to determine a true layer and a merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data; The method for obtaining the frequency category label specifically includes: Based on the access frequency, the policy data is segmented by multiple thresholds to obtain multiple segmentation points, specifically including: after obtaining the access frequency of each policy data, obtaining the access frequency histogram of all data by statistics; segmenting the access frequency histogram by a multiple threshold processing method to obtain multiple segmentation points; Determining candidate frequency values of multiple categories based on the frequency values of each two adjacent segmentation points among the multiple segmentation points specifically includes: taking the frequency values of the two adjacent segmentation points as candidate frequency values of one category, taking the data corresponding to the candidate frequency values as data points of the category, and obtaining multiple categories; Based on the difference between the candidate frequency value of each category and the candidate frequency values of other categories, the frequency category label of each category is determined, specifically including: wherein the other categories are categories other than the category in the multiple categories; calculating the sum of the absolute values of the differences between each candidate frequency value and other candidate frequency values in each category, taking the frequency value corresponding to the minimum summation result as the category frequency value of the category, taking the category frequency value as the frequency category label of the data point in each category, and obtaining multiple categories and the frequency category label of each category; The determining of the real layer and the merged layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels of the policy data comprises: Determining a classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels; Determining the true layer and the merged layer among the multiple layers based on the classification error of each layer; The determining the classification error of each layer in the multiple layers based on the clustering results of the multiple layers and the frequency category labels includes: For each node category in each layer, determining a plurality of actual categories corresponding to each node category in the frequency category label; Determine the data volume of each actual category in the multiple actual categories to obtain multiple data volumes; determining a ratio between a maximum value and a second maximum value among the plurality of data amounts; Determining a classification error of each node category based on the ratio and the number of categories of the multiple actual categories; Performing averaging processing on the classification errors of multiple node categories in each layer to obtain the classification error of each layer; The determining the true layer and the merged layer in the multiple layers based on the classification error of each layer comprises: Determining the layer with the smallest classification error among the multiple layers as the true layer; determining the merged layer among the multiple layers based on the real layer; A second determination module is used to determine, based on the real layer and the merged layer, the category recognition capabilities of multiple base classifiers for classifying the merged layer; An integration module, configured to integrate the plurality of base classifiers based on the category recognition capability and the merging layer to obtain an integrated classifier; The classification module is used to classify and store the real-time policy data in response to the received real-time policy data using the integrated classifier.
