Data classified storage system and method based on artificial intelligence
Through the data classification storage system based on artificial intelligence, data processing is automatically processed, the problem of low data classification integration efficiency is solved, the accuracy of data storage and system performance is improved, the risk of data loss is reduced, and the load and important data of the storage system are optimized. Save.
Patent Information
- Application Number
- CN202410896568.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, data classification and integration efficiency is low, resulting in confusion in data operation and large load on storage systems, which may affect the storage of important data and increase the risk of data loss.
Using an artificial intelligence-based data classification and storage system, through data acquisition, feature extraction, clustering, activity calculation and storage weight determination modules, data classification and storage are automatically implemented to ensure that important data is stored in high-performance memory and storage resources are reasonably allocated.
It improves the accuracy and efficiency of data classification, reduces the risk of data loss, optimizes the workload of the storage system, improves data access speed and system stability, and supports data-driven decision-making and enterprise operations.
Smart Images

Figure CN119025728A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to an artificial intelligence-based data classification storage system, method, electronic device and non-transitory computer-readable storage medium. Background Art
[0002] Nowadays, with the continuous development of information technology, more and more companies record, store and transmit production data through the Internet and other means. Whether it is in the process of communication within the enterprise or between enterprises, a large amount of production data will be generated. Some of these data are very important and need to be used frequently and stored for a long time, but some are just ordinary data with low frequency of use and importance, and only need to be stored.
[0003] However, the efficiency of manually classifying and integrating these production data is low. If these production data are not properly classified and processed, it will lead to data operation confusion, heavy workload of the storage system, slow operation, and may also affect the preservation of important data, which may easily lead to the loss of important data. Summary of the invention
[0004] In response to the technical problems existing in the prior art, the present invention provides an artificial intelligence-based data classification storage system, method, electronic device and non-transitory computer-readable storage medium that can improve the accuracy and rationality of data classification storage.
[0005] The technical solution of the present invention to solve the above technical problems is as follows:
[0006] The present invention provides a data classification storage system based on artificial intelligence, the system comprising:
[0007] A data acquisition module is used to acquire original data and pre-process the original data to obtain corresponding multiple data to be stored;
[0008] A feature extraction module, used for extracting features from the plurality of data to be stored, and obtaining a feature vector for each of the data to be stored;
[0009] A clustering module, used for performing data clustering on the plurality of data to be stored by using an objective function according to a feature vector of each data to be stored, so as to obtain a category label of each data to be stored;
[0010] An activity module, used to calculate the user activity of each of the data to be stored;
[0011] A storage weight module, used to determine the storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0012] The classification storage module is used to classify and store the multiple data to be stored according to the storage weight of each data to be stored.
[0013] Optionally, the feature extraction module is specifically used for:
[0014] Obtaining a feature category of each of the data to be stored;
[0015] Obtaining the value of each of the data to be stored in each of the feature categories;
[0016] Obtaining the standard deviation and the average value of the initial value of each feature in the data set of the data to be stored;
[0017] A feature vector of each of the data to be stored is obtained according to the value of each of the feature categories of the data to be stored, the standard deviation and the average value.
[0018] Optionally, each feature vector of the data to be stored is expressed as:
[0019] F i =(f1,f2,...,f m );
[0020] Among them, F i is the feature vector of the i-th data to be stored, f is the feature, and m is the total number of features;
[0021] Wherein, each of the features is expressed as:
[0022]
[0023] Among them, f j is the eigenvalue of the jth feature, x ij is the value of the i-th data to be stored in the j-th feature category, m is the total number of features, μ j is the average value of the initial value of the jth feature in the data set to be stored, σ j is the standard deviation of the initial value of the jth feature in the data set to be stored.
[0024] Optionally, the clustering module is specifically used for:
[0025] Calculating the distance between each of the data to be stored and each cluster center;
[0026] Constructing a clustering indicator matrix according to the data to be processed and the clustering categories, and calculating each element of the clustering indicator matrix;
[0027] The function value of the objective function is calculated according to the distance between each of the to-be-stored data and each cluster center and each element of the cluster indication matrix.
[0028] Optionally, the objective function is expressed as:
[0029]
[0030] Among them, J(X,M) is the objective function, X is the cluster indicator matrix, M is the cluster center matrix, x ij is an element of the cluster indicator matrix, indicating the value of the i-th data to be stored in the j-th feature category, i and j are serial values, F i is the feature vector of the i-th data to be stored, M j is the element of the cluster center matrix, representing the jth cluster center.
[0031] Optionally, the activity module is specifically used to:
[0032] Obtaining an observation period during which each user uses the data to be stored;
[0033] Obtaining a time point at which each user uses the data to be stored;
[0034] Obtaining the number of times each of the users uses the stored data at multiple time points during the observation period;
[0035] For each of the data to be stored, the user activity of each of the data to be stored is determined according to the number of times each of the users uses the data.
[0036] Optionally, the user activity of each data to be stored is expressed as:
[0037] A=(a1,a2,…,a n );
[0038] Where A is the vector of user activity of each data to be stored;
[0039] Wherein, for multiple users who use each of the data to be stored, the user activity of each of the users is expressed as:
[0040]
[0041] Among them, a i is the user activity of the i-th user who uses the data to be stored, T is the observation period, t is the time point, α is the attenuation coefficient, and u it is the number of times the i-th user uses the stored data at time point t, and ∈ is the first constant.
[0042] Optionally, the storage weight module is specifically used for:
[0043] For each of the data to be stored, obtaining the value of the data to be stored in each of the feature categories;
[0044] Obtaining a feature value for each of the features;
[0045] Obtaining user activity of each user who uses the data to be stored;
[0046] The storage weight of the data to be stored is determined according to the value of the data to be stored in each of the feature categories, the feature value of each of the features, and the user activity of each of the users who use the data to be stored.
[0047] Optionally, the storage weight of each of the data to be stored is expressed as:
[0048]
[0049] Among them, ω i is the storage weight of the i-th data to be stored, N is the total number of features used to calculate the storage weight, x ij is the value of the i-th data to be stored in the j-th feature category, f j is the eigenvalue of the jth feature, a i is the user activity of the i-th user who uses the data to be stored, i and j are serial numbers
[0050] The present invention also provides a data classification storage method based on artificial intelligence, the method comprising:
[0051] Acquire original data, and preprocess the original data to obtain corresponding multiple data to be stored;
[0052] Performing feature extraction on the plurality of data to be stored to obtain a feature vector of each data to be stored;
[0053] According to the feature vector of each of the data to be stored, clustering the plurality of data to be stored is performed by using an objective function to obtain a category label of each of the data to be stored;
[0054] Calculating the user activity of each of the data to be stored;
[0055] Determining a storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0056] According to the storage weight of each of the data to be stored, the multiple data to be stored are classified and stored.
[0057] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, comprising: a memory for storing computer software programs; a processor for reading and executing the computer software programs, thereby realizing an artificial intelligence-based data classification and storage method as described above.
[0058] In addition, to achieve the above-mentioned purpose, the present invention also proposes a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, it implements an artificial intelligence-based data classification storage method as described above.
[0059] The beneficial effects of the present invention are:
[0060] (1) The present invention can automatically classify data through feature extraction and clustering algorithms, avoiding the inefficiency and error-proneness of manual classification. The automated classification process can quickly adapt to massive data and improve the overall processing speed.
[0061] (2) The present invention designs storage strategies based on the importance, usage frequency and user activity of the data to ensure that important data is stored in high-performance memory, thereby improving access speed and data reliability. By calculating storage weights, storage resources can be reasonably allocated, reducing the storage space occupied by unimportant data and optimizing the workload of the storage system.
[0062] (3) The present invention ensures that important data is preserved for a long time through intelligent classification and priority storage, reduces the risk of data loss, improves the access efficiency of important data, meets business needs, and enhances the operational capabilities of enterprises.
[0063] (4) The present invention adjusts the data storage strategy according to user activity to ensure that the data of highly active users is stored in high-performance storage, thereby improving their data access experience, dynamically adjusting the storage strategy, and promptly responding to changes in user behavior, thereby improving the intelligence level of the system and user satisfaction.
[0064] (5) The present invention reduces the ineffective workload of the storage system through reasonable classification and storage strategy, avoids slow system operation due to data confusion, improves the stability and operation efficiency of the storage system, extends the service life of the storage device, and reduces operation and maintenance costs.
[0065] (6) The classified and stored data of the present invention can be analyzed and mined more conveniently, supporting enterprises to make data-driven decisions, providing visual analysis of data classification and storage strategies, helping managers understand data distribution and usage, and optimizing business processes.
[0066] This invention uses artificial intelligence technology to achieve automatic classification and optimized storage of production data, significantly improves the efficiency and security of data management, enhances user experience and overall system performance, and provides strong support for the digital transformation and data-driven decision-making of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 A scene diagram of a data classification and storage method based on artificial intelligence provided by the present invention;
[0068] Figure 2 A schematic diagram of the structure of a data classification storage system based on artificial intelligence provided by the present invention;
[0069] Figure 3 A flowchart of a data classification and storage method based on artificial intelligence provided by the present invention;
[0070] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;
[0071] Figure 5 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION
[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0073] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0074] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.
[0075] See also Figure 1 , Figure 1 A scene diagram of a data classification and storage method based on artificial intelligence provided by the present invention. Figure 1 As shown, the terminal and the server are connected via a network, such as a wired or wireless network connection. The terminal may include but is not limited to portable terminals such as mobile phones and tablets installed with various network platform applications, as well as fixed terminals such as computers, query machines, and advertising machines. The server provides users with various business services, including service push servers, user recommendation servers, etc.
[0076] It should be noted that Figure 1 The scenario diagram of a data classification and storage method based on artificial intelligence shown is only an example. The terminal, server and application scenario described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present invention, and do not generate limitations on the technical solution provided by the embodiment of the present invention. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.
[0077] Among them, the terminal can be used for:
[0078] Acquire original data, and preprocess the original data to obtain corresponding multiple data to be stored;
[0079] Performing feature extraction on the plurality of data to be stored to obtain a feature vector of each data to be stored;
[0080] According to the feature vector of each of the data to be stored, clustering the plurality of data to be stored is performed by using an objective function to obtain a category label of each of the data to be stored;
[0081] Calculating the user activity of each of the data to be stored;
[0082] Determining a storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0083] According to the storage weight of each of the data to be stored, the multiple data to be stored are classified and stored.
[0084] See also Figure 2 , Figure 2 A schematic diagram of the structure of an artificial intelligence-based data classification storage system provided by the present invention.
[0085] like Figure 2 As shown, an artificial intelligence-based data classification storage system proposed in an embodiment of the present invention includes:
[0086] The data acquisition module 201 is used to acquire original data and pre-process the original data to obtain corresponding multiple data to be stored.
[0087] In some embodiments, raw data refers to various data generated and collected by an enterprise during production and operation. These data may come from multiple different systems and devices, such as sensor data on a production line, transaction records in an enterprise resource planning (ERP) system, customer data in a customer relationship management (CRM) system, environmental data collected by Internet of Things (IoT) devices, and log data generated in business processes. Raw data is usually unprocessed and contains various noises and outliers, and needs to be preprocessed before it can be used for subsequent classification and storage.
[0088] In some embodiments, data preprocessing is the process of converting raw data into high-quality, uniformly formatted data to be stored, and mainly includes the following methods, such as noise and outlier removal: Identify and remove noise and outliers in the data, such as sensor false alarms, system error records, etc. For example, if a sensor data point deviates significantly from other data points, it can be regarded as an outlier and removed.
[0089] Handling missing values: Fill or delete missing data to ensure data integrity. For example, for missing data, you can use mean filling method, interpolation method, etc.
[0090] Standardize data range: Convert data of different dimensions into a unified dimension range to facilitate subsequent processing and analysis.
[0091] Format conversion: Convert data in different formats into a unified format for subsequent processing. For example, unify the date format into "YYYY-MM-DD".
[0092] Data merging: Merging data from different sources to form a complete data set. For example, merging production line data with transaction records in the ERP system to form a comprehensive data set.
[0093] Deduplication: Identify and remove duplicate items in the data to ensure the uniqueness and accuracy of the data set. For example, in customer data, remove duplicate customer records.
[0094] After the above steps, the present invention cleans, normalizes, converts and integrates the raw data to form high-quality, structured data to be stored. These high-quality data to be stored provide a solid foundation for subsequent feature extraction, clustering, classification labeling and storage strategy design. Through preprocessing, the raw data is converted into a data set that can be directly used in artificial intelligence algorithms, thereby improving the efficiency and effect of the entire data classification and storage process.
[0095] The feature extraction module 202 is used to extract features from the plurality of data to be stored, and obtain a feature vector of each data to be stored.
[0096] In some embodiments, the feature extraction module 202 is specifically used to:
[0097] Obtaining a feature category of each of the data to be stored;
[0098] Obtaining the value of each of the data to be stored in each of the feature categories;
[0099] Obtaining the standard deviation and the average value of the initial value of each feature in the data set of the data to be stored;
[0100] A feature vector of each of the data to be stored is obtained according to the value of each of the feature categories of the data to be stored, the standard deviation and the average value.
[0101] In some embodiments, the feature vector of each of the data to be stored is expressed as:
[0102] F i =(f1, f2, ..., f m );
[0103] Among them, F i is the feature vector of the i-th data to be stored, f is the feature, and m is the total number of features;
[0104] Wherein, each of the features is expressed as:
[0105]
[0106] Among them, f j is the eigenvalue of the jth feature, x ijis the value of the i-th data to be stored in the j-th feature category, m is the total number of features, μ j is the average value of the initial value of the jth feature in the data set to be stored, σ j is the standard deviation of the initial value of the jth feature in the data set to be stored.
[0107] In the specific implementation, f j is the jth feature, which may be, for example, a product attribute feature of the data to be stored;
[0108] x ij is the value of the i-th data in the preprocessed data to be stored on the j-th feature, which means the data value related to the j-th feature in the i-th data;
[0109] m is the number of features, that is, how many data there are in total;
[0110] μ j is the average of the initial values of the jth feature in the data set to be stored, and is the average of the jth eigenvalues of all data points, used to standardize the eigenvalues;
[0111] σ j It is the standard deviation of the initial value of the j-th feature in the data set to be stored, which indicates the degree of discreteness of the j-th eigenvalue and is used to standardize the eigenvalue.
[0112] In some embodiments, feature f may be an important attribute used to describe and distinguish data points. Features may be various attributes or dimensions in the data, depending on the specific application field and data set. For example, if the data to be stored is production equipment data, then f may be the operating temperature, working pressure, vibration frequency, and cumulative operating time of the production equipment. For another example, when the data to be stored is sales data, feature f may be sales amount, sales quantity, product type, and product rating.
[0113] In the specific calculation process, for the jth feature, the average value of the feature value in all data points can be calculated: Then for the jth feature, calculate the standard deviation of the feature value in all data points: For each data point x ij , minus the mean μ j , and then divided by the standard deviation σ j The purpose of this step is to convert the eigenvalues into a standard normal distribution, that is, the mean is 0 and the standard deviation is 1, and we get Then square the standardized eigenvalues. The purpose of squaring is to amplify the impact of values far from the mean on the sum, and we get Finally, the normalized and squared eigenvalues of all data points are summed up, and the final f jIt is the sum of these values.
[0114] Through the above method, the present invention can obtain the extraction results of each feature, which will be used in the subsequent clustering and classification steps. The key to this process is standardization, which can eliminate the dimensional differences between different features, so that each feature has equal importance in the subsequent analysis.
[0115] The clustering module 203 is used to perform data clustering on the plurality of data to be stored according to the feature vector of each data to be stored by using an objective function to obtain a category label of each data to be stored.
[0116] In some embodiments, the clustering module 203 is specifically used to:
[0117] Calculating the distance between each of the data to be stored and each cluster center;
[0118] Constructing a clustering indicator matrix according to the data to be processed and the clustering categories, and calculating each element of the clustering indicator matrix;
[0119] The function value of the objective function is calculated according to the distance between each of the to-be-stored data and each cluster center and each element of the cluster indication matrix.
[0120] In some embodiments, the objective function is expressed as:
[0121]
[0122] Among them, J(X,M) is the objective function, X is the cluster indicator matrix, M is the cluster center matrix, x ij is an element of the cluster indicator matrix, indicating the value of the i-th data to be stored in the j-th feature category, i and j are serial values, F i is the feature vector of the i-th data to be stored, M j is the element of the cluster center matrix, representing the jth cluster center.
[0123] In the specific implementation, J(X,M) is the objective function, which represents the total error or total cost of clustering. By minimizing this value, the best clustering result can be obtained;
[0124] X is the cluster indicator matrix, which is n×k in size, where n is the number of data points to be processed and k is the number of clusters. ij Indicates whether the i-th data point belongs to the j-th cluster;
[0125] M is the cluster center matrix, size is k×m, where k is the number of clusters, m is the dimension of each feature vector, and the element M in the matrix j represents the center of the jth cluster;
[0126] F i is the feature vector of the i-th data to be stored;
[0127] ||F i -M j || 2 is the square of the Euclidean distance from the i-th data point to the j-th cluster center.
[0128] In the specific implementation, the distance from each data point to be stored to each cluster center can be calculated first. For each data point F i And each cluster center M j , we can calculate the Euclidean distance between them and square the distance, specifically:
[0129] In some embodiments, an element of an indicator matrix may be calculated, based on each data point F i , belongs to which cluster M j , determine the value of the corresponding position in the indicator matrix X. For each i and j, if the i-th data point belongs to the j-th cluster, then x ij =1; otherwise x ij =0.
[0130] In some embodiments, the objective function value can be calculated by multiplying the square of the distance from each data point to its corresponding cluster center by the value of the indicator matrix, and summing all data points and all clusters to obtain the value of the objective function J(X,M).
[0131] Through the above method, the present invention finds the optimal cluster center and cluster indicator matrix by minimizing the objective function, so that the total square distance between the data point and its corresponding cluster center is minimized, and the distribution of the data points in each cluster is more compact, thereby achieving the best clustering effect.
[0132] In some embodiments, the vector of class labels is L = (l1, l2, ..., l k ).
[0133] Among them, L is a vector of category labels, containing k category labels;
[0134] l k is the category label of the k-th category, indicating the category name or label of the k-th cluster;
[0135] k is the total number of cluster labels.
[0136] In the specific implementation, clustering tags are used to give meaningful labels to clustering results so that each cluster can be clearly identified and processed in the subsequent data management and storage process. The assignment of category tags can be based on specific business needs and data characteristics.
[0137] Now suppose that a cluster analysis is performed on the data of a batch of production equipment, and two clusters (k=2) are finally obtained. Appropriate category labels are assigned to these two clusters so that they can be distinguished in subsequent storage and management.
[0138] In some embodiments, the device data can be divided into two clusters: Cluster 1 and Cluster 2 by using a clustering algorithm. A category label can be assigned to each cluster based on the clustering results and business requirements. For example, assume that the devices in Cluster 1 usually exhibit high temperature and high pressure characteristics and are high-risk devices; while the devices in Cluster 2 exhibit low temperature and low pressure characteristics and are low-risk devices.
[0139] Category label: L = (High Risk, Low Risk);
[0140] Among them: l1=High Risk means that the devices in cluster 1 belong to the high risk category, and l2=Low Risk means that the devices in cluster 2 belong to the low risk category.
[0141] The final clustering results and category labels correspond to the following:
[0142] High-risk devices (High Risk): Device 1, Device 3, Device 5; Low-risk devices (Low Risk): Device 2, Device 4.
[0143] In summary, the present invention can achieve higher efficiency and better decision support in subsequent data storage and management by assigning meaningful labels to each cluster.
[0144] The activity module 204 is used to calculate the user activity of each data to be stored.
[0145] In some embodiments, the activity module 204 is specifically used to:
[0146] Obtaining an observation period during which each user uses the data to be stored;
[0147] Obtaining a time point at which each user uses the data to be stored;
[0148] Obtaining the number of times each of the users uses the stored data at multiple time points during the observation period;
[0149] For each of the data to be stored, the user activity of each of the data to be stored is determined according to the number of times each of the users uses the data.
[0150] In some embodiments, the user activity of each of the data to be stored is expressed as:
[0151] A=(a1,a2,…,a n );
[0152] Where A is the vector of user activity of each data to be stored;
[0153] Wherein, for multiple users who use each of the data to be stored, the user activity of each of the users is expressed as:
[0154]
[0155] Among them, a i is the user activity of the i-th user who uses the data to be stored, T is the observation period, t is the time point, α is the attenuation coefficient, and u it is the number of times the i-th user uses the stored data at time point t, and ∈ is the first constant.
[0156] In the specific implementation, A is the user activity corresponding to each data to be processed;
[0157] a i is the user activity of the i-th user who uses the data to be stored;
[0158] T is the observation period;
[0159] t is a time point, distributed from 1 to T;
[0160] α is the decay coefficient, which is used to control the time decay speed;
[0161] u it is the number of times the i-th user uses the stored data at time point t;
[0162] ∈ is the first constant;
[0163] In the specific implementation, e -(α·t) It represents the time decay weight. The larger the time t is, the smaller the weight is. The decay coefficient α controls the decay speed. The larger the value is, the faster the decay is. It represents the weighted usage times of user i in time period T, where the most recent usage times have a larger weight and the earlier usage times have a smaller weight. represents the sum of the time decay weights, plus a small constant ∈ to prevent the denominator from being zero. Through the above calculation, the activity of the i-th user can be obtained, which reflects the user's activeness in using the data to be stored during the observation period T.
[0164] The storage weight module 205 is used to determine the storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored.
[0165] In some embodiments, the storage weight module 205 is specifically used to:
[0166] For each of the data to be stored, obtaining the value of the data to be stored in each of the feature categories;
[0167] Obtaining a feature value for each of the features;
[0168] Obtaining user activity of each user who uses the data to be stored;
[0169] The storage weight of the data to be stored is determined according to the value of the data to be stored in each of the feature categories, the feature value of each of the features, and the user activity of each of the users who use the data to be stored.
[0170] In some embodiments, the storage weight of each of the data to be stored is expressed as:
[0171]
[0172] Among them, ω i is the storage weight of the i-th data to be stored, N is the total number of features used to calculate the storage weight, x ij is the value of the i-th data to be stored in the j-th feature category, f j is the eigenvalue of the jth feature, a i is the user activity of the i-th user who uses the data to be stored, and i and j are serial numbers.
[0173] In the specific implementation, ω is a vector of storage weights, which contains the storage weights of n data;
[0174] ω i is the storage weight of the i-th data to be stored,
[0175] N is the total number of features used to calculate the storage weights,
[0176] j is the number value of the feature, for example, the jth feature,
[0177] x ij is the value of the i-th data to be stored on the j-th feature,
[0178] f j is the eigenvalue of the jth feature,
[0179] a iis the user activity of the i-th user who uses the data to be stored,
[0180] ∈ is the first constant.
[0181] In the specific implementation, It represents the ratio of the value of the i-th data on the j-th feature to the eigenvalue of the j-th feature, which is used to standardize the eigenvalue. represents the weighted sum of all features of the i-th data, where the weight is the activity a of user i i . is the sum of the weighted sums of the characteristic ratios, plus a small first constant To prevent the denominator from being zero. Through the above calculation, the storage weight of the i-th data is obtained, which reflects the importance and priority of the data to be stored.
[0182] The classification storage module 206 is used to classify and store the multiple data to be stored according to the storage weight of each data to be stored.
[0183] In some embodiments, data can be divided into different storage categories based on storage weights. Generally, they can be divided into three categories: high priority, medium priority, and low priority, including high priority, medium priority, and low priority. Specifically:
[0184] High priority: Stores data with high storage weight. This type of data is usually important and frequently used, and needs to be stored in high-performance storage media to ensure fast access and high reliability.
[0185] Medium priority: Stores data with medium weight. This type of data is used infrequently and is stored in storage media with moderate performance.
[0186] Low priority: Stores data with low storage weight. This type of data is rarely used or is not important and is stored in storage media with lower costs.
[0187] In some embodiments, corresponding storage strategies can be generated based on storage weights and storage categories to allocate data to different storage media. For example: Storage media for high-priority data storage: high-speed SSD or NVMe storage devices, storage strategy: use high-performance, high-reliability storage solutions such as RAID 10 to ensure fast data read and write and high availability. Examples: real-time monitoring data of production lines, key business transaction data, etc.
[0188] Storage media for medium-priority data storage: traditional hard disk (HDD) or hybrid storage devices (such as hybrid SSD), storage strategy: use storage solutions such as RAID 5 or RAID 6 to balance storage performance and data security. Example: regularly generated report data, general business record data, etc.
[0189] Storage media for low-priority data storage: tape storage, cloud storage or cold storage devices, storage strategy: adopt low-cost, high-capacity storage solutions suitable for long-term storage and archiving. Examples: historical data backup, archived data, etc.
[0190] In some embodiments, the storage weight can be calculated, and the storage weight ω of each data can be obtained by preprocessing, feature extraction and user activity calculation. i . Determine the storage category and classify the data into three categories: high priority, medium priority, and low priority according to the storage weight. Select the storage medium and select the appropriate storage medium and storage policy according to the storage category of the data. Perform storage operations to store the data in the specified storage medium to ensure that the data is reasonably allocated according to priority. Assume that there are the following three types of data to be stored: Data 1: storage weight ω1 = 0.95, Data 2: storage weight ω2 = 0.6, Data 3: storage weight ω3 = 0.2. According to the storage weight, these data are classified as follows:
[0191] High priority data: Data 1, stored in high-speed SSD.
[0192] Medium priority data: Data 2, stored in traditional HDDs.
[0193] Low priority data: Data 3, stored in cloud storage or tapes.
[0194] The classified storage strategy of the present invention can ensure that important data is stored preferentially and accessed quickly, while optimizing the utilization of storage resources and improving the efficiency and reliability of overall data management.
[0195] See also Figure 3 , provides a flow chart of a data classification and storage method based on artificial intelligence of the present invention, comprising the following steps:
[0196] Step 301: acquiring original data, and preprocessing the original data to obtain corresponding multiple data to be stored;
[0197] Step 302: extracting features from the plurality of data to be stored to obtain a feature vector for each of the data to be stored;
[0198] Step 303: clustering the plurality of data to be stored by using an objective function according to the feature vector of each of the data to be stored, to obtain a category label of each of the data to be stored;
[0199] Step 304: Calculate the user activity of each of the data to be stored;
[0200] Step 305: Determine the storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0201] Step 306: Classify and store the plurality of data to be stored according to the storage weight of each data to be stored.
[0202] See also Figure 4 , Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, the following steps are implemented:
[0203] Acquire original data, and preprocess the original data to obtain corresponding multiple data to be stored;
[0204] Performing feature extraction on the plurality of data to be stored to obtain a feature vector of each data to be stored;
[0205] According to the feature vector of each of the data to be stored, clustering the plurality of data to be stored is performed by using an objective function to obtain a category label of each of the data to be stored;
[0206] Calculating the user activity of each of the data to be stored;
[0207] Determining a storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0208] According to the storage weight of each of the data to be stored, the multiple data to be stored are classified and stored.
[0209] See also Figure 5 , Figure 5 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 5 As shown, this embodiment provides a computer-readable storage medium 500, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the following steps are implemented:
[0210] Acquire original data, and preprocess the original data to obtain corresponding multiple data to be stored;
[0211] Performing feature extraction on the plurality of data to be stored to obtain a feature vector of each data to be stored;
[0212] According to the feature vector of each of the data to be stored, clustering the plurality of data to be stored is performed by using an objective function to obtain a category label of each of the data to be stored;
[0213] Calculating the user activity of each of the data to be stored;
[0214] Determining a storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored;
[0215] According to the storage weight of each of the data to be stored, the multiple data to be stored are classified and stored.
[0216] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0217] Those skilled in the art will appreciate that embodiments of the present invention may be provided as systems, methods, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0218] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.
[0219] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0220] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0221] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0222] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A data classification storage system based on artificial intelligence, characterized in that: The system comprises: A data acquisition module is used to acquire original data and pre-process the original data to obtain corresponding multiple data to be stored; A feature extraction module, used for extracting features from the plurality of data to be stored, and obtaining a feature vector for each of the data to be stored; A clustering module, used for performing data clustering on the plurality of data to be stored by using an objective function according to a feature vector of each data to be stored, so as to obtain a category label of each data to be stored; An activity module, used to calculate the user activity of each of the data to be stored; A storage weight module, used to determine the storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored; The classification storage module is used to classify and store the multiple data to be stored according to the storage weight of each data to be stored.
2. The data classification storage system based on artificial intelligence according to claim 1 is characterized in that: The feature extraction module is specifically used for: Obtaining a feature category of each of the data to be stored; Obtaining the value of each of the data to be stored in each of the feature categories; Obtaining the standard deviation and the average value of the initial value of each feature in the data set of the data to be stored; A feature vector of each of the data to be stored is obtained according to the value of each of the feature categories of the data to be stored, the standard deviation and the average value.
3. The data classification storage system based on artificial intelligence according to claim 2 is characterized in that: Each feature vector of the data to be stored is expressed as: F i =(f1,f2,…,f m ); Among them, F i is the feature vector of the i-th data to be stored, f is the feature, and m is the total number of features; Wherein, each of the features is expressed as: Among them, f i is the eigenvalue of the jth feature, x ij is the value of the i-th data to be stored in the j-th feature category, m is the total number of features, μ j is the average value of the initial value of the jth feature in the data set to be stored, σ j is the standard deviation of the initial value of the jth feature in the data set to be stored.
4. The data classification storage system based on artificial intelligence according to claim 3 is characterized in that: The clustering module is specifically used for: Calculating the distance between each of the data to be stored and each cluster center; Constructing a clustering indicator matrix according to the data to be processed and the clustering categories, and calculating each element of the clustering indicator matrix; The function value of the objective function is calculated according to the distance between each of the to-be-stored data and each cluster center and each element of the cluster indication matrix.
5. The artificial intelligence-based data classification storage system according to claim 4 is characterized in that: The objective function is expressed as: Among them, J(X,M) is the objective function, X is the cluster indicator matrix, M is the cluster center matrix, x ij is an element of the cluster indicator matrix, indicating the value of the i-th data to be stored in the j-th feature category, i and j are serial values, F i is the feature vector of the i-th data to be stored, M j is the element of the cluster center matrix, representing the jth cluster center.
6. The artificial intelligence-based data classification storage system according to claim 5 is characterized in that: The activity module is specifically used for: Obtaining an observation period during which each user uses the data to be stored; Obtaining a time point at which each user uses the data to be stored; Obtaining the number of times each of the users uses the stored data at multiple time points during the observation period; For each of the data to be stored, the user activity of each of the data to be stored is determined according to the number of times each of the users uses the data.
7. The artificial intelligence-based data classification storage system according to claim 6, characterized in that: The user activity of each of the data to be stored is expressed as: <h2 style=";text-align:left;direction:ltr">A = (a1, a2,…, a)<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ); Where A is the vector of user activity of each data to be stored; Wherein, for multiple users who use each of the data to be stored, the user activity of each of the users is expressed as: Among them, a i is the user activity of the i-th user who uses the data to be stored, T is the observation period, t is the time point, α is the attenuation coefficient, and u it is the number of times the i-th user uses the stored data at time point t, and ∈ is the first constant.
8. The artificial intelligence-based data classification storage system according to claim 7, characterized in that: The storage weight module is specifically used for: For each of the data to be stored, obtaining the value of the data to be stored in each of the feature categories; Obtaining a feature value for each of the features; Obtaining user activity of each user who uses the data to be stored; The storage weight of the data to be stored is determined according to the value of the data to be stored in each of the feature categories, the feature value of each of the features, and the user activity of each of the users who use the data to be stored.
9. The artificial intelligence-based data classification storage system according to claim 8, characterized in that: The storage weight of each of the data to be stored is expressed as: Among them, ω i is the storage weight of the i-th data to be stored, N is the total number of features used to calculate the storage weight, x ij is the value of the i-th data to be stored in the j-th feature category, f j is the eigenvalue of the jth feature, a i is the user activity of the i-th user who uses the data to be stored, and i and j are serial numbers.
10. A data classification storage method based on artificial intelligence, characterized in that: The method comprises: Acquire original data, and preprocess the original data to obtain corresponding multiple data to be stored; Performing feature extraction on the plurality of data to be stored to obtain a feature vector of each data to be stored; According to the feature vector of each of the data to be stored, clustering the plurality of data to be stored is performed by using an objective function to obtain a category label of each of the data to be stored; Calculating the user activity of each of the data to be stored; Determining a storage weight of each of the data to be stored according to the user activity, feature vector and category label of each of the data to be stored; According to the storage weight of each of the data to be stored, the multiple data to be stored are classified and stored.
