A data asset management platform
Through the prediction and strategy adjustment model of the data asset management platform, the problem that traditional storage platforms cannot cope with massive data is solved, and efficient utilization of storage resources and real-time data analysis is achieved.
Patent Information
- Application Number
- CN202411154156.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-22
AI Technical Summary
Traditional single data storage platforms cannot carry massive data, resulting in storage bottlenecks, reduced storage efficiency and increased costs. In high-visit scenarios, current storage resources are difficult to expand in time, which is easy to cause waste.
A data asset management platform is designed, including data prejudgment module, data processing module and policy output module. By predicting memory occupancy, dynamically adjusting memory resources, building and training strategy adjustment models, the allocation and use of storage resources are optimized.
It realizes efficient management of data storage, reduces waste of storage capacity, improves the utilization rate of storage resources, and provides real-time and accurate analysis of data storage.
Smart Images

Figure CN118656225B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data storage management, and more specifically, to a data asset management platform. Background Art
[0002] In data asset management, there are problems with data storage and processing capabilities. The main reason is that the advancement of the digitalization process has led to a rapid increase in data volume. The traditional single storage platform cannot carry massive data, which easily leads to storage bottlenecks. Therefore, the amount of data is growing exponentially, and storage capacity is difficult to meet. In addition, the data is generated quickly, especially in application scenarios such as the Internet of Things, financial transactions, and social media. The demand for real-time data processing is extremely high, which leads to reduced storage efficiency and increased storage costs, reducing the economic benefits of the asset management platform. In addition, during the high-frequency period of platform access, the amount of data access increases rapidly, and the platform's operating computing power requirements expand rapidly. If the cloud platform is applied for the memory resources required for expansion to establish a new node, it will take a long time and delay, and it will not respond in time, which will easily increase the platform access congestion and cause waste of existing storage resources. Therefore, how to reasonably allocate and design resources among the existing storage resources is now an urgent problem to be solved. Summary of the invention
[0003] In view of the problems existing in the prior art, the purpose of the present invention is to provide a data asset management platform to achieve efficient management of asset data.
[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a data asset management platform, the data asset management platform comprising:
[0005] Data pre-judgment module: used to collect the amount of data of users on the platform in the current time period, predict the memory usage rate in the next time period based on the data amount, and judge the memory usage rate;
[0006] Data processing module: used to determine whether to dynamically adjust the memory resources that need to be expanded, and to perform in-depth analysis on the data that needs to be dynamically adjusted to obtain an information mapping set;
[0007] Strategy output module: used to build and train the strategy adjustment model, import the information mapping set in the current time period into the trained strategy adjustment model, and obtain the adjustment strategy set at each node in each data category;
[0008] The data pre-judgment module, the data processing module and the strategy output module are connected via wired and / or wireless means.
[0009] Preferably, the data volume of the user includes access frequency and transaction frequency, and the memory occupation contribution rate of each user is calculated according to the data volume;
[0010] Calculate the access frequency in the current time period ;in, Indicates the number of logins. Indicates the number of query clicks. A time indicating a time period;
[0011] Calculate the transaction frequency in the current time period ;in, represents the number of asset expenditures, Indicates the number of times assets are received;
[0012] Calculate the memory usage contribution rate in the current time period ;in, and Represent the average contribution coefficients of access frequency and transaction frequency, and They represent the access frequency and transaction frequency of user i respectively, Indicates the number of users on the platform during the current time period;
[0013] Indicates the platform in the current time period Total memory usage Compared with the previous time period Total memory usage The difference.
[0014] Preferably, the predicting of the memory usage rate in the next time period includes:
[0015] Perform exponential smoothing calculation on the memory usage contribution rate and set the smoothing coefficient , get the smoothed memory usage contribution rate , Indicates the previous time period Memory usage contribution rate;
[0016] Predict the next time period Memory usage of internal users ;in, Indicates the hotspot index. Hotspot The density aggregation value, Hotspot The influence diffusion factor, The heat affects the conversion parameters.
[0017] Preferably, the determining of the memory occupancy rate includes:
[0018] Set the threshold EA_t;
[0019] like ≤EA_t, indicating that the platform's memory resources are sufficient to meet demand in the next time period and do not need to be expanded;
[0020] like >EA_t, indicating that the platform's memory resources cannot meet the demand in the next time period and need to be expanded, and the memory resources required to be expanded are = ;
[0021] in, Indicates the prediction for the next time period The operating memory of components in the platform, and ,in, Indicates the growth rate of component running memory. Indicates the component running memory occupied by the platform in the current time period.
[0022] The determining whether to dynamically adjust the memory resources that need to be expanded includes:
[0023] Extract the data categories corresponding to the storage blocks on the platform, as well as the remaining storage capacity of the data nodes in each storage block, and identify the need to expand memory resources The data categories in the table are selected, and the memory resource size that needs to be expanded for each data category is extracted. If the remaining storage capacity at the node satisfies the memory resource size that needs to be expanded for the corresponding data category, no dynamic adjustment is performed. If the remaining storage capacity at the node does not satisfy the memory resource size that needs to be expanded for the corresponding data category, dynamic adjustment is performed.
[0024] Preferably, the in-depth analysis of the data that needs to be dynamically adjusted includes:
[0025] Construct an information gain matrix. In the information gain matrix, the data categories are rows, the remaining storage capacity at each node is columns, and the storage capacity is the same standard, specifically ,in, Indicates data category In the The remaining storage capacity at the nodes, and , ;
[0026] Initialize the data categories whose storage capacity meets the memory resource expansion requirements and set them uniformly to 0 to obtain the information similarity matrix ;
[0027] Set the memory resource size that needs to be expanded in the data category whose storage capacity meets the memory resource expansion to 0, and build the central information data set ,in To predict the next time period Internal data category The value corresponding to the memory resource that needs to be expanded;
[0028] Extracting central information dataset Each data category in the data set needs to expand the value of memory resources to build the information similarity matrix The cluster analysis used Dimensional cluster center set ;
[0029] Data Category Centralized information data Numerical construction Dimensional cluster center set , set the number of cluster centers in the set to , and The value of the cluster center is calculated as ;
[0030] in , An adaptive parameter for the remaining storage capacity in each data class.
[0031] Preferably, the information similarity matrix The cluster analysis includes:
[0032] extract Dimensional cluster center set The cluster center corresponding to each data category in , and the information similarity matrix is extracted based on the cluster center Cluster the values of the corresponding data categories in ;
[0033] According to the distance between each value and the cluster center, the value with the smallest distance from the cluster center is grouped into a cluster, and then the corresponding Clusters;
[0034] Extracting central information dataset Each of Corresponding clusters are clustered to build information mapping sets;
[0035] The information mapping set construction includes: one-hot encoding the data categories that need to expand memory resources and the nodes in the storage blocks corresponding to each data category to form a binary string, and setting the binary string as a label, mapping the size of memory resources that need to be expanded in the same data category and the storage capacity of each node in the storage block into a combination, and counting the combinations of all data categories to form an information mapping set.
[0036] Preferably, the strategy adjustment model is constructed in the following manner:
[0037] The strategy adjustment model is based on the LSTM basic framework, and sets an input layer, an LSTM layer, a fully connected layer, and an output layer. The input of the input layer is set to an information mapping set, the LSTM layer is set to two layers, each layer has 50 units, LSTM layer 1 is set to return a sequence, and LSTM layer 2 is set to not return a sequence;
[0038] The fully connected layer is set to 1 layer, which is used for regression tasks. It is set with 10 nodes and the activation function is ReLU. The output layer includes the regression task output layer. The output is set to the adjustment strategy set. The activation function is set to the softmax function. The regression task output layer is used to predict the optimization method of each node and is set to 1 node.
[0039] The Adam optimization algorithm is used, the learning rate is set to 0.001, and the loss function is:
[0040] ;
[0041] in, is the number of nodes, is the node index, and is the weight coefficient, is the compression index at the true marked node, is the predicted compression index at the node, is the elastic expansion index of the node at the real mark, is the predicted elastic expansion index at the node;
[0042] And the constraint condition is set to minimize the comprehensive delay .
[0043] Preferably, the compression index is used to indicate the compression ratio of the used storage capacity at the node, and the elastic expansion index is used to indicate the ratio of the multiples of the remaining storage capacity at the node that needs to be temporarily expanded.
[0044] Preferably, the training and use of the strategy adjustment model includes:
[0045] S11, collect information mapping sets and corresponding adjustment strategy sets to form a sample set, divide it into a training set and a validation set in a ratio of 8:2, divide the training set into batches and input them into the strategy adjustment model in turn, and perform forward propagation in the strategy adjustment model;
[0046] S12, obtaining an adjustment strategy set output by the strategy adjustment model prediction, using the loss function to calculate the loss value, using the back propagation algorithm to calculate each parameter in the model, and using the gradient descent algorithm to update the parameters;
[0047] S13, repeat S11 and S12 until the loss function value of the policy adjustment model no longer changes and satisfies the constraint conditions, import the verification set for verification, and when the verification is successful, obtain the trained policy adjustment model. When the verification is unsuccessful, repeat S13;
[0048] S14. Input the information mapping set in the current time period into the trained strategy adjustment model to obtain the adjustment strategy set of the predicted output, and dynamically adjust and optimize the design of each node in the platform according to the adjustment strategy set.
[0049] Compared with the prior art, the present invention has the following beneficial effects:
[0050] In the present invention, by predicting and judging the memory occupancy rate in the current time period, the data growth demand of the platform is explored. And by transforming the information gain matrix and the information similarity matrix, the amount of data analysis is further reduced, and high-quality input for subsequent data depth analysis is achieved. By improving the cluster analysis of the similarity matrix, the remaining storage capacity of the node closest to the memory resource to be expanded is obtained, the patchwork of multiple node storage capacities, and the wasteful use of the remaining high storage capacity are reduced, and the optimization use of the remaining storage capacity at the node is improved, and the high integration of data storage on the node is achieved. The strategy adjustment model implemented by this platform outputs personalized design for the dynamic adjustment and optimization design of each node; the adaptive control optimization mechanism continuously improves the overall efficiency of intelligent analysis, and multiple modules cooperate with each other to achieve comprehensive data collection, processing and analysis, and provide real-time and accurate analysis of data storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 This is a structural diagram of a data asset management platform proposed by the present invention.
[0052] Figure 2 A schematic diagram of the steps of a data asset management method proposed in the present invention. DETAILED DESCRIPTION
[0053] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0054] Embodiment 1
[0055] Reference Figure 1 , Example 1 further illustrates a data asset management platform proposed by the present invention.
[0056] In data asset management, there are problems with data storage and processing capabilities. The main reason is that the advancement of the digitalization process has led to a rapid increase in data volume. The traditional single storage platform cannot carry massive data, which easily leads to storage bottlenecks. Therefore, the amount of data is growing exponentially, and storage capacity is difficult to meet. In addition, the data is generated quickly, especially in application scenarios such as the Internet of Things, financial transactions, and social media. The demand for real-time data processing is extremely high, which leads to reduced storage efficiency and increased storage costs, reducing the economic benefits of the asset management platform. In addition, during the high-frequency period of platform access, the amount of data access increases rapidly, and the platform's operating computing power requirements expand rapidly. If the cloud platform is applied for the memory resources required for expansion to establish a new node, it will take a long time and delay, and it will not respond in time, which will easily increase the platform access congestion and cause waste of existing storage resources. Therefore, how to reasonably allocate and design resources among the existing storage resources is now an urgent problem to be solved.
[0057] This data asset management platform includes: data pre-judgment module, data processing module and strategy output module.
[0058] Data pre-judgment module: used to collect the data volume of users on the platform in the current time period, predict the memory occupancy rate in the next time period based on the data volume, and judge the memory occupancy rate. The data volume of users in the present invention can be used to represent the active behavior of users, that is, the memory resources occupied by users' activities on the platform; the data volume of users includes access frequency and transaction frequency, and the memory occupancy contribution rate of each user is calculated based on the data volume.
[0059] Calculate the access frequency in the current time period ;in, Indicates the number of logins. Indicates the number of query clicks. Indicates the time of a time period; the number of logins and query clicks can be obtained through login logs, user activity logs (such as Hadoop, Spark), etc.
[0060] Calculate the transaction frequency in the current time period ;in, represents the number of asset expenditures, Indicates the number of times an asset is received. The historical data and new data of asset expenditures and receipts on the platform can be obtained through the statistical tools of the storage platform (such as HDFS, AWSS3);
[0061] Calculate the memory usage contribution rate in the current time period ;in, and They represent the average contribution coefficients of access frequency and transaction frequency respectively. The average contribution coefficient can be set by the operator based on the experimental data analysis of the memory usage of unit transaction frequency and access frequency. and They represent the access frequency and transaction frequency of user i respectively, Indicates the number of users on the platform during the current time period.
[0062] Indicates the platform in the current time period Total memory usage Compared with the previous time period Total memory usage The real-time memory usage, including used memory and available memory, can be obtained through platform monitoring tools (such as Prometheus and Grafana).
[0063] Perform exponential smoothing calculation on the memory usage contribution rate and set the smoothing coefficient , get the smoothed memory usage contribution rate , Indicates the previous time period The memory usage contribution rate.
[0064] Predict the next time period Memory usage of internal users ;in, Indicates the hotspot index. Hotspot The density aggregation value, Hotspot The influence diffusion factor, It is the conversion parameter affected by heat, which is used to indicate the conversion probability from hot spots to users on the platform.
[0065] Because hot news will affect the access data and active data of users on the platform, which in turn affects the platform's memory. Therefore, paying attention to the impact of hot news on the platform can better predict the platform's memory usage in the next time period.
[0066] For example: Assuming the smoothing coefficient =0.5;
[0067] Total memory usage in the current time period =1000MB;
[0068] Total memory usage in the previous period =900MB;
[0069] Difference =1000-900=100MB;
[0070] The number of users on the platform in the current time period is 100;
[0071] Assuming that the access frequency and transaction frequency of 100 users take the same value, the settings are =10 and =5;
[0072] And set the average contribution coefficient of access frequency and transaction frequency to a uniform value, then set =0.1, =0.05;
[0073] Assume that the memory usage contribution rate in the previous time period =0.2;
[0074] Then calculate the memory usage contribution rate of the current time period:
[0075] ;
[0076] Smoothed memory usage contribution rate: ;
[0077] Assuming that there are no hot spots in the current time period, that is, the number of hot spots is 0, the memory usage rate in the next time period is predicted:
[0078] MB;
[0079] Through the above steps and formulas, the memory usage rate in the next time period can be effectively calculated and predicted, so as to determine whether memory resources need to be expanded or allocated.
[0080] Get the next time period for prediction Memory usage within Make a judgment and set the threshold EA_t;
[0081] like ≤EA_t, indicating that the platform's memory resources are sufficient to meet demand in the next time period and do not need to be expanded;
[0082] like >EA_t, indicating that the platform's memory resources cannot meet the demand in the next time period and need to be expanded, and the memory resources required to be expanded are = ;
[0083] in, Indicates the prediction for the next time period The operating memory of components in the platform, and ,in, Indicates the growth rate of component running memory. Indicates the component running memory occupied by the platform in the current time period. This information can be obtained through the platform monitoring tool or memory manager.
[0084] For example, suppose the component running memory occupied by the platform in the current time period is =500MB, there is no hotspot in the current time period, that is, the number of hotspots is 0, then =0, the growth rate of component running memory =0.1, then MB;
[0085] Set the threshold EA_t=1000MB, =1050>1000, the platform's memory resources need to be expanded, and the memory resources need to be expanded:
[0086] = =1050-1000=50MB.
[0087] By collecting user retention information and monitoring platform memory usage, memory resources can be expanded and allocated based on set thresholds and formulas to meet platform needs and maintain platform stability and performance.
[0088] The data asset management platform adopts distributed storage. Distributed storage is to divide the storage resources of the platform according to the data category. The division here can be set by manual intervention of the management personnel, or it can be set adaptively through big data. Each storage block only stores data corresponding to one data category. Assuming that the data categories stored in the present invention include component running memory and user data volume, identity information, and platform security protection information, there are 4 storage blocks set. Assuming that the storage capacity on each data node is the same, when the memory resource needs to be expanded, the remaining (free and available) storage capacity at each node is read. Many distributed storage databases (such as MySQL, PostgreSQL) provide built-in statistical functions. For example, distributed storage platforms such as Hadoop's HDFS and AWS's S3 provide statistical tools, which can be used to query the usage and performance indicators of each storage block in the database. According to the memory resources in the next time period, Determine whether expansion is needed, design nodes for the data categories that need to be expanded, and expand storage resources reasonably and dynamically.
[0089] Data processing module: used to determine whether to dynamically adjust the memory resources that need to be expanded, and to perform in-depth analysis on the data that needs to be dynamically adjusted to obtain an information mapping set.
[0090] Determining whether to dynamically adjust the memory resources that need to be expanded includes:
[0091] Extract the data categories corresponding to the storage blocks on the platform, as well as the remaining storage capacity of the data nodes in each storage block, and identify the need to expand memory resources The data categories in the table are selected, and the memory resource size that needs to be expanded for each data category is extracted. If the remaining storage capacity at the node satisfies the memory resource size that needs to be expanded for the corresponding data category, no dynamic adjustment is performed. If the remaining storage capacity at the node does not satisfy the memory resource size that needs to be expanded for the corresponding data category, dynamic adjustment is performed.
[0092] In-depth analysis of data that needs to be dynamically adjusted includes:
[0093] Construct an information gain matrix. In the information gain matrix, the data categories are rows, the remaining storage capacity at each node is columns, and the storage capacity is the same standard, specifically ,in, Indicates data category In the The remaining storage capacity at the nodes, and , ;
[0094] Initialize the data categories whose storage capacity meets the memory resource expansion requirements and set them uniformly to 0 to obtain the information similarity matrix ;
[0095] Set the memory resource size that needs to be expanded in the data category whose storage capacity meets the memory resource expansion to 0, and build the central information data set ,in To predict the next time period Internal data category The value corresponding to the memory resource that needs to be expanded;
[0096] Extracting central information dataset Each data category in the data set needs to expand the value of memory resources to build the information similarity matrix The cluster analysis used Dimensional cluster center set ;
[0097] Data Category Centralized information data Numerical construction Dimensional cluster center set , set the number of cluster centers in the set to , and The value of the cluster center is calculated as ;
[0098] in , The adaptive parameter is an adaptive parameter of the remaining storage capacity in each data category, and the adaptive parameter can be set by the administrator.
[0099] For example, there are four data categories corresponding to the storage blocks on the platform. Assume that the data categories are component running memory, user data volume, identity information, and platform security protection information, and the number of nodes in each data category is as follows:
[0100] Node number Component running memory Data volume Identity information Safety information 1 0 2MB 100MB 800MB 2 2000MB 160MB 567MB 0 3 100MB 1000MB 0 0 4 0 8MB 0 0
[0101] Assume that you need to expand memory resources The data categories in the data are the user's data volume, identity information, and platform security protection information, and are specifically:
[0102] Data volume Identity information Safety information 370MB 2000MB 500MB
[0103] The unmarked table indicates that there is no node here, so it is represented as 0. In this case, the data volume and security protection information do not need to be dynamically adjusted because the storage capacity at the node meets the memory resources that need to be expanded, while the identity information needs to be dynamically adjusted;
[0104] That is, the information gain matrix ;
[0105] Information Similarity Matrix ;
[0106] Constructed central information dataset , assuming that the number of cluster centers in the user's data volume is =2, then the value of the cluster center in the user's data volume is calculated as ; ;
[0107] Then get Dimensional cluster center set:
[0108] ;
[0109] Information Similarity Matrix The cluster analysis includes:
[0110] extract Dimensional cluster center set The cluster center corresponding to each data category in , and the information similarity matrix is extracted based on the cluster center Cluster the values of the corresponding data categories in ;
[0111] According to the distance between each value and the cluster center, the value with the smallest distance from the cluster center is grouped into a cluster, and then the corresponding Clusters;
[0112] Extracting central information dataset Each of Corresponding clusters are constructed to build information mapping sets.
[0113] The information mapping set construction includes: one-hot encoding the data categories that need to expand memory resources and the nodes in the storage blocks corresponding to each data category to form a binary string, and setting the binary string as a label, mapping the size of memory resources that need to be expanded in the same data category and the storage capacity of each node in the storage block into a combination, and counting the combinations of all data categories to form an information mapping set.
[0114] Strategy output module: used to build and train the strategy adjustment model, import the information mapping set in the current time period into the trained strategy adjustment model, and obtain the adjustment strategy set at each node in each data category.
[0115] The strategy adjustment model is constructed in the following ways:
[0116] The strategy adjustment model is based on the LSTM basic framework, and sets an input layer, an LSTM layer, a fully connected layer, and an output layer. The input of the input layer is set to an information mapping set, the LSTM layer is set to two layers, each layer has 50 units, LSTM layer 1 is set to return a sequence, and LSTM layer 2 is set to not return a sequence;
[0117] The fully connected layer is set to 1 layer, which is used for regression tasks. It is set with 10 nodes and the activation function is ReLU. The output layer includes the regression task output layer. The output is set to the adjustment strategy set. The activation function is set to the softmax function. The regression task output layer is used to predict the optimization method of each node and is set to 1 node.
[0118] The Adam optimization algorithm is used, the learning rate is set to 0.001, and the loss function is:
[0119] ;
[0120] in, is the number of nodes, is the node index, and is the weight coefficient, is the compression index at the true marked node, is the predicted compression index at the node, is the elastic expansion index of the node at the real mark, is the predicted elastic expansion index at the node;
[0121] And the constraint condition is set to minimize the comprehensive delay .
[0122] The compression index is used to indicate the compression ratio of the used storage capacity at the node, and the elastic expansion index is used to indicate the ratio of the multiples of the remaining storage capacity at the node that needs to be temporarily expanded.
[0123] The training and use of the policy adjustment model includes:
[0124] S11, collect information mapping sets and corresponding adjustment strategy sets to form a sample set, divide it into a training set and a validation set in a ratio of 8:2, divide the training set into batches and input them into the strategy adjustment model in turn, and perform forward propagation in the strategy adjustment model;
[0125] Among them, the adjustment strategy set corresponding to the information mapping set can use historical data, or the adjustment strategy set can be manually annotated;
[0126] S12, obtaining an adjustment strategy set output by the strategy adjustment model prediction, using the loss function to calculate the loss value, using the back propagation algorithm to calculate each parameter in the model, and using the gradient descent algorithm to update the parameters;
[0127] S13, repeat S11 and S12 until the loss function value of the policy adjustment model no longer changes and satisfies the constraint conditions, import the verification set for verification, and when the verification is successful, obtain the trained policy adjustment model. When the verification is unsuccessful, repeat S13;
[0128] S14. Input the information mapping set in the current time period into the trained strategy adjustment model to obtain the adjustment strategy set of the predicted output, and dynamically adjust and optimize the design of each node in the platform according to the adjustment strategy set.
[0129] In this embodiment, the present invention explores the data growth demand of the platform by predicting and judging the memory occupancy rate in the current time period. And through the transformation of the information gain matrix and the information similarity matrix, the amount of data analysis is further reduced, and the high-quality input of the subsequent data depth analysis is realized. Through the improved cluster analysis of the similarity matrix, the remaining storage capacity of the node closest to the memory resource to be expanded is obtained, the patchwork of storage capacity of multiple nodes and the wasteful use of the remaining high storage capacity are reduced, the optimization use of the remaining storage capacity at the node is improved, and the high integration of data storage on the node is realized. The strategy adjustment model realized by this platform outputs personalized design for the dynamic adjustment and optimization design of each node; the adaptive control optimization mechanism continuously improves the overall efficiency of intelligent analysis, and multiple modules cooperate with each other to realize the comprehensive collection, processing and analysis of data, and provide real-time and accurate analysis of data storage.
[0130] Embodiment 2
[0131] This embodiment provides a data asset management method, which uses the above-mentioned data asset management platform and includes the following steps:
[0132] Step S1: Collect the amount of data of users on the platform in the current time period, predict the memory usage rate in the next time period, and make a judgment;
[0133] Step S2: Perform in-depth analysis on the memory resources that need to be dynamically adjusted to obtain an information mapping set;
[0134] Step S3: Build and train a policy adjustment model;
[0135] Step S4: Import the information mapping set in the current time period into the trained strategy adjustment model, obtain the adjustment strategy set at each node in each data category, and adjust each node according to the adjustment strategy set.
[0136] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A data asset management platform, characterized in that: The data asset management platform includes: Data pre-judgment module: used to collect the amount of data of users on the platform in the current time period, predict the memory usage rate in the next time period based on the data amount, and judge the memory usage rate; The data volume of the user includes access frequency and transaction frequency, and the memory occupation contribution rate of each user is calculated according to the data volume; Calculate the access frequency in the current time period Among them, q1 represents the number of logins, q2 represents the number of query clicks, and T represents the time of the time period; Calculate the transaction frequency in the current time period Among them, w1 represents the number of asset expenditures, and w2 represents the number of asset receipts; Calculate the memory usage contribution rate in the current time period Among them, e1 and e2 represent the average contribution coefficients of access frequency and transaction frequency respectively, and Fw i (t) and Jy i (t) represents the access frequency and transaction frequency of user i, respectively, and N represents the number of users on the platform in the current time period; ΔM(t)=M(t)-M(t-1) represents the difference between the total memory usage M(t) of the platform at the current time period t and the total memory usage M(t-1) at the previous time period t-1; Data processing module: used to determine whether to dynamically adjust the memory resources that need to be expanded, and to perform in-depth analysis on the data that needs to be dynamically adjusted to obtain an information mapping set; The determining whether to dynamically adjust the memory resources that need to be expanded includes: Extract the data categories corresponding to the storage blocks on the platform and the remaining storage capacity of the data nodes in each storage block, identify the data categories in the memory resources ΔU(t+1) that need to be expanded, and extract the size of the memory resources that need to be expanded for each data category. If the remaining storage capacity at the node satisfies the size of the memory resources that need to be expanded for the corresponding data category, no dynamic adjustment is performed. If the remaining storage capacity at the node does not satisfy the size of the memory resources that need to be expanded for the corresponding data category, dynamic adjustment is performed. The in-depth analysis of the data that needs to be dynamically adjusted includes: Construct an information gain matrix. In the information gain matrix, the data categories are rows, the remaining storage capacity at each node is columns, and the storage capacity is the same standard, specifically Among them, x jl represents the remaining storage capacity of data category l at the jth node, where l = 1, ..., b, j = 1, ..., a; Initialize the data categories whose storage capacity meets the memory resource expansion requirements and set them uniformly to 0, then obtain the information similarity matrix X 1 ; The memory resource size that needs to be expanded in the data category whose storage capacity meets the memory resource expansion is set to 0, and a central information data set F = {u1_ut1, ..., ul_ut1, ..., ub_ut1} is constructed, where ul_ut1 is the value corresponding to the memory resource expansion required for data category l in the next time period t+1; Extract the value of memory resources required to expand each data category in the central information data set F = {u1_ut1,...,ul_ut1,...,ub_ut1} and construct the information similarity matrix X 1 The K-dimensional cluster center set Q used in cluster analysis K ={q_tew1,...,q_tew l ,...,q_tew b }; Data category l constructs K based on the ul_ut1 value in the central information data set l Dimensional cluster center set q_tew l ={jz_1,...,jz_c,...,jz_K l }, set the number of cluster centers in the set to K l The value of the cth cluster center is calculated as Where c=1,......,K l ,σ is the adaptive parameter of the remaining storage capacity in each data category; Strategy output module: used to build and train the strategy adjustment model, import the information mapping set in the current time period into the trained strategy adjustment model, and obtain the adjustment strategy set at each node in each data category; The data pre-judgment module, the data processing module and the strategy output module are connected via wired and / or wireless means.
2. A data asset management platform according to claim 1, characterized in that: The predicted memory usage rate in the next time period includes: Perform exponential smoothing calculation on the memory usage contribution rate, set the smoothing coefficient α, and obtain the smoothed memory usage contribution rate Ng s (t) = α × Ng(t) + (1-α) × Ng(t-1), Ng(t-1) represents the memory usage contribution rate at the previous time period t-1; Predict the user's memory usage in the next time period t+1 Among them, u represents the hotspot index, o u _tre is the density clustering value of hotspot u, s u _wre is the influence diffusion factor of hotspot u, β u The heat affects the conversion parameters.
3. A data asset management platform according to claim 2, characterized in that: The determining of the memory usage includes: Set the threshold EA_t; If U y (t+1)+U other (t+1)≤EA_t, indicating that the platform's memory resources are sufficient to meet demand in the next time period and do not need to be expanded; If U y (t+1)+U other (t+1)>EA_t, indicating that the platform's memory resources cannot meet the demand in the next time period and need to be expanded, and the memory resources required to be expanded are ΔU(t+1)=U y (t+1)+U other (t+1); Among them, U other (t+1) indicates the predicted component running memory occupied by the platform in the next time period t-1, and Where η represents the growth rate of component running memory, U other (t) represents the component running memory occupied by the platform in the current time period.
4. A data asset management platform according to claim 3, characterized in that: The information similarity matrix X 1 The cluster analysis includes: Extract K-dimensional cluster center set Q K ={q_tew1,...,q_tew l ,...,q_tew b The cluster center corresponding to each data category in}, and the information similarity matrix X is extracted based on the cluster center 1 Cluster the values of the corresponding data categories in ; According to the distance between each value and the cluster center, the value with the smallest distance from the cluster center is grouped into a cluster, and then the K corresponding to each data category is obtained. l Clusters; Extract the cluster corresponding to each ul_ut1 in the central information data set F = {u1_ut1,…,ul_ut1,…,ub_ut1}, and construct an information mapping set; The information mapping set construction includes: one-hot encoding the data categories that need to expand memory resources and the nodes in the storage blocks corresponding to each data category to form a binary string, and setting the binary string as a label, mapping the size of memory resources that need to be expanded in the same data category and the storage capacity of each node in the storage block into a combination, and counting the combinations of all data categories to form an information mapping set.
5. A data asset management platform according to claim 4, characterized in that: The construction method of the strategy adjustment model includes: The strategy adjustment model is based on the LSTM basic framework, and sets an input layer, an LSTM layer, a fully connected layer, and an output layer. The input of the input layer is set to an information mapping set, the LSTM layer is set to two layers, each layer has 50 units, LSTM layer 1 is set to return a sequence, and LSTM layer 2 is set to not return a sequence; The fully connected layer is set to 1 layer, which is used for regression tasks. It is set with 10 nodes and the activation function is ReLU. The output layer includes the regression task output layer. The output is set to the adjustment strategy set. The activation function is set to the softmax function. The regression task output layer is used to predict the optimization method of each node and is set to 1 node. The Adam optimization algorithm is used, the learning rate is set to 0.001, and the loss function is: Among them, g is the number of nodes, h is the node index, θ1 and θ2 are weight coefficients, z_y1 h is the compression index at the true marked node, is the predicted compression index at the node, z_y2 h is the elastic expansion index of the node at the real mark, is the predicted elastic expansion index at the node; And the constraint condition is set to minimize the comprehensive delay Among them, ts h _v represents the delay at the hth node.
6. A data asset management platform according to claim 5, characterized in that: The compression index is used to indicate the compression ratio of the used storage capacity at the node, and the elastic expansion index is used to indicate the ratio of the multiples of the remaining storage capacity at the node that needs to be temporarily expanded.
7. A data asset management platform according to claim 6, characterized in that: The training and use of the policy adjustment model includes: S11, collect information mapping sets and corresponding adjustment strategy sets to form a sample set, divide it into a training set and a validation set in a ratio of 8:2, divide the training set into batches and input them into the strategy adjustment model in turn, and perform forward propagation in the strategy adjustment model; S12, obtaining an adjustment strategy set output by the strategy adjustment model prediction, using the loss function to calculate the loss value, using the back propagation algorithm to calculate each parameter in the model, and using the gradient descent algorithm to update the parameters; S13, repeat S11 and S12 until the loss function value of the policy adjustment model no longer changes and satisfies the constraint conditions, import the verification set for verification, and when the verification is successful, obtain the trained policy adjustment model. When the verification is unsuccessful, repeat S13; S14. Input the information mapping set in the current time period into the trained strategy adjustment model to obtain the adjustment strategy set of the predicted output, and dynamically adjust and optimize the design of each node in the platform according to the adjustment strategy set.
