Enterprise data intelligent storage optimization method and system

Abstract

The invention discloses an enterprise data intelligent storage optimization method and system. The method comprises the steps of enterprise data collection, data access mode preliminary screening, data access mode clustering, alternating refinement and enterprise data storage optimization. The invention belongs to the field of data processing, and particularly relates to an enterprise data intelligent storage optimization method and system.According to the scheme, a dispersion popularity membership degree is introduced, a weakening coefficient is added into a target function, historical access influences are attenuated, the weight of burst high-frequency access is reduced, and then misclassification caused by accidental access is reduced; business association driven local self-adaptive approaching clustering is performed on enterprise data, so that a high-benefit storage scheme can be formulated in a unified manner; a common precision objective function is constructed, and bidirectional matching of the mode and enterprise data association degree is ensured through an alternate precision framework; and the enterprise data intelligent storage optimization effect is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for optimizing intelligent storage of enterprise data. Background Technology

[0002] Enterprise data intelligent storage optimization methods leverage data access characteristics and business-related information to automatically allocate data to storage media with varying performance and cost profiles through pattern recognition, clustering, and strategy matching, thereby achieving efficient utilization of storage resources and cost reduction. However, typical enterprise data intelligent storage optimization methods suffer from several drawbacks. Firstly, they ignore access uncertainties, leading to unnecessary high-cost storage allocations. Secondly, sudden access can disrupt the stability of pattern recognition. Furthermore, these methods often fail to connect pattern recognition with the overall enterprise data's relevance, disrupting the tightness of business connections and causing data from the same business flow to cross different storage layers, increasing access latency and costs. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a method and system for optimizing intelligent enterprise data storage. Addressing the problems of general intelligent enterprise data storage optimization methods ignoring access uncertainties, leading to unnecessary high-cost storage allocation, and the instability of pattern recognition due to sudden access interference, this solution introduces a diffused membership degree and adds a weakening coefficient to the objective function to attenuate the impact of historical accesses and reduce the weight of sudden high-frequency accesses, thereby reducing misclassification caused by occasional accesses. Furthermore, it performs business-related driven local adaptive convergent clustering of enterprise data, explicitly incorporating business lifecycle changes into the membership degree calculation, ensuring that enterprise data is not only optimized based on access patterns... The patterns are similar and have strong business relevance, which facilitates the unified formulation of efficient storage solutions. Addressing the common problem in general enterprise data intelligent storage optimization methods—a disconnect between pattern recognition and enterprise data relevance, resulting in the disruption of business relevance and causing data from the same business flow to cross different storage layers, increasing access latency and costs—this solution constructs a common refinement objective function. Through a rotating refinement framework, it ensures bidirectional matching between pattern and enterprise data relevance. By combining alternating anchor variables, analytical solutions, and feature decomposition, the final result achieves global consistency in both pattern coordinates and enterprise data relevance, providing a reliable foundation for subsequent storage strategy allocation; thereby improving the effectiveness of enterprise data intelligent storage optimization.

[0004] The technical solution adopted by this invention is as follows: This invention provides an intelligent storage optimization method for enterprise data, which includes the following steps:

[0005] Step S1: Enterprise data collection;

[0006] Step S2: Initial screening of data access patterns;

[0007] Step S3: Data access pattern clustering;

[0008] Step S4: Rotate and improve;

[0009] Step S5: Enterprise data storage optimization.

[0010] Furthermore, in step S1, the enterprise data collection involves acquiring multi-source enterprise data, including access frequency, recent access time, data size, and number of related businesses; the collected data is preprocessed to ultimately construct an enterprise data feature set.

[0011] Furthermore, in step S2, the initial screening of data access patterns involves introducing a heat dissipation factor and a weakening coefficient into the enterprise data feature set, and identifying the potential state of the data through soft partitioning.

[0012] Furthermore, in step S3, the data access pattern clustering specifically includes:

[0013] Construct a local adaptive convergence value matrix, where the matrix elements represent the correlation between enterprise data, measured by diffuse distance, and also incorporate business correlation strength;

[0014] The number of modes is limited by adding a rank constraint to the Laplace matrix of the local adaptive approximation matrix;

[0015] Construct a pattern feature matrix, where the pattern feature vector represents the coordinate vector of enterprise data in the pattern space.

[0016] Furthermore, in step S4, the rotation and refinement is to avoid mismatch between data patterns and correlation, by synchronously adjusting the data correlation and pattern feature matrix through the rotation and refinement framework.

[0017] Furthermore, in step S5, the enterprise data storage optimization utilizes the enterprise-related clusters naturally formed by the locally adaptive convergent value matrix after rotation and refinement, and assigns the optimal storage strategy to each enterprise-related cluster.

[0018] This invention provides an intelligent storage optimization system for enterprise data, comprising an enterprise data acquisition module, a data access pattern initial screening module, a data access pattern clustering module, a rotation and refinement module, and an enterprise data storage optimization module;

[0019] The enterprise data acquisition module acquires enterprise data from multiple sources and constructs an enterprise data feature set after preprocessing.

[0020] The data access pattern screening module uses a soft partitioning objective function to optimize and identify the access patterns of enterprise data for the enterprise data feature set.

[0021] The data access pattern clustering module constructs a local adaptive convergence value matrix and a pattern feature matrix based on the access pattern.

[0022] The rotation and refinement module constructs a joint objective function that integrates correlation and pattern features, and performs rotation and refinement.

[0023] The enterprise data storage optimization module optimizes enterprise data storage based on the clusters formed by the refined local adaptive convergence value matrix.

[0024] The beneficial effects achieved by the present invention using the above solution are as follows:

[0025] (1) In view of the problems of general enterprise data intelligent storage optimization methods ignoring the uncertainty of access, causing unnecessary high-cost storage allocation, and the stability of sudden access interference pattern recognition, this solution introduces heat dissipation membership degree and adds a weakening coefficient to the objective function to attenuate the impact of historical access, reduce the weight of sudden high-frequency access, and thus reduce misclassification caused by occasional access; perform business-related driven local adaptive convergence clustering on enterprise data, and explicitly incorporate business life cycle changes into the correlation calculation to ensure that enterprise data not only have similar access patterns, but also have strong business correlation, which is conducive to the unified formulation of high-efficiency storage solutions.

[0026] (2) In view of the problem that general enterprise data intelligent storage optimization methods are disconnected from the correlation between pattern recognition and enterprise data, and the closeness of business correlation is destroyed, resulting in data of the same business flow crossing different storage layers, increasing access latency and cost, this solution constructs a common refinement objective function, and ensures bidirectional matching between pattern and enterprise data correlation through a rotating refinement framework; by combining alternating anchor variables, analytical solution and feature decomposition, the final result achieves global consistency in both pattern coordinates and enterprise data correlation, providing a reliable basis for subsequent storage strategy allocation; thereby improving the optimization effect of enterprise data intelligent storage. Attached Figure Description

[0027] Figure 1 A flowchart illustrating an intelligent storage optimization method for enterprise data provided by the present invention;

[0028] Figure 2 This is a schematic diagram of an enterprise data intelligent storage optimization system provided by the present invention.

[0029] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0032] Example 1, see Figure 1 The present invention provides an intelligent storage optimization method for enterprise data, which includes the following steps:

[0033] Step S1: Enterprise data collection, acquiring multi-source enterprise data, and constructing an enterprise data feature set after preprocessing;

[0034] Step S2: Initial screening of data access patterns. For the enterprise data feature set, the access patterns of enterprise data are identified by optimizing the soft partitioning objective function.

[0035] Step S3: Cluster data access patterns, construct a local adaptive convergent value matrix, and construct a pattern feature matrix based on the access patterns;

[0036] Step S4: Rotation and refinement, construct a joint objective function that integrates correlation and pattern features, and perform rotation and refinement;

[0037] Step S5: Enterprise data storage optimization. Based on the clusters formed by the refined local adaptive convergence value matrix, enterprise data storage is optimized.

[0038] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, enterprise data collection involves acquiring multi-source enterprise data, including access frequency (number of accesses per unit time, reflecting data popularity), recent access time (interval from the current time, reflecting data timeliness), data size (unit: MB, affecting storage costs), and the number of related businesses; the collected data is preprocessed (Min-Max normalization processing) to finally construct an enterprise data feature set.

[0039] Example 3, see Figure 1This embodiment is based on the above embodiment. In step S2, the initial screening of data access patterns indicates that enterprise data has access uncertainty (data is occasionally accessed frequently, or it has not been accessed for a long time but is suddenly called). To avoid misjudgment due to hard partitioning, a heat dissipation factor and a weakening coefficient are introduced for the enterprise data feature set. The potential hot / warm / cold state of the data is identified through soft partitioning, which improves the robustness of pattern recognition. At the same time, historical access is gradually weakened to reduce sudden interference. The specific operation is as follows:

[0040] Define enterprise data d i heat dissipation membership degree u ik (k=1,2,3 represent hot / warm / cold models respectively). To avoid interference from sudden high-frequency accesses that often occur in enterprise data (month-end reports centrally retrieve cold data from the previous month), a weakening coefficient is introduced to optimize the objective function, expressed as: ; Limited to: ; m is the diffusion factor, which controls the degree of soft partitioning; c k It is the feature center of the k-th access mode; u ik It is data d i The membership degree to pattern k; the larger the value, the more likely the data belongs to this pattern. It is the objective function for reducing heat intensity, used to produce preliminary pattern centers; It is the feature vector of the i-th enterprise's data; This is the decay rate, with a value ranging from 0.5 to 8; It is the most recent access time of the enterprise data; It is the current time; It is the weakening coefficient; is the maximum time span; n is the total number of enterprise data.

[0041] Example 4, see Figure 1 This embodiment is based on the above embodiment. In step S3, data access pattern clustering indicates that there are local business relationships between enterprise data. Data with similar access patterns and strong business relationships need to be clustered into one category to design a targeted storage strategy. Therefore, a local adaptive proximity value matrix is ​​constructed using local adaptive proximity values. Combined with a pattern number constraint, this ensures that the clustering results match actual business needs. Simultaneously, the strength of business relationships is explicitly included in the correlation calculation, making it more sensitive to changes in the business lifecycle. Specifically, this includes:

[0042] Construct a locally adaptive convergent value matrix S, with matrix elements s ij Represents enterprise data d i and enterprise data d j The degree of correlation is measured by diffuse distance, while also incorporating business correlation strength. , represented as: ; Limited to: ;d ij It is the Euclidean distance of data features, reflecting local differences; It is the objective function that approaches the target value and is used to construct a local adaptive convergence value matrix; Within the window at time t, the enterprise data d i and enterprise data d j The number of times something appears in the same business process is obtained from the enterprise's business process logs; It is enterprise data d i and enterprise data d k Statistics on the number of times they co-occur in the same business process; This is the sharpness adjustment factor, with a value ranging from 0.1 to 6;

[0043] Mode number constraint, Laplace matrix of locally adaptive approaching value matrix Add a rank constraint to ensure the number of clusters conforms to the business preset, represented as: ; is the rank of the matrix; c is the number of clusters, corresponding to the number of storage levels determined by business requirements;

[0044] To further characterize the distribution of each data point in the pattern space, a pattern feature matrix F and a pattern feature vector f are constructed. i This represents the coordinate vector of the i-th enterprise data in the schema space. The schema space is derived from the schema feature centers obtained in step S2. The positional relationship of enterprise data in the schema space directly reflects their proximity in access patterns, rather than simply their distance.

[0045] pass Diffusion of correlation degree reduces sensitivity to initial distance metric; The forced local adaptive convergence value matrix forms c enterprise association clusters. Each enterprise association cluster corresponds to a data access mode and consists of all enterprise data contained in the connected components.

[0046] By performing the above operations, this solution addresses the issues of general enterprise data intelligent storage optimization methods, such as ignoring access uncertainties, leading to unnecessary high-cost storage allocation, and the instability of pattern recognition due to sudden access interference. It introduces a diffused membership degree and adds a weakening coefficient to the objective function to attenuate the impact of historical accesses and reduce the weight of sudden high-frequency accesses, thereby reducing misclassification caused by occasional accesses. Furthermore, it performs business-related driven local adaptive convergent clustering of enterprise data, explicitly incorporating business lifecycle changes into the correlation calculation. This ensures that enterprise data not only has similar access patterns but also strong business correlations, facilitating the unified development of efficient storage solutions.

[0047] Example 5, see Figure 1 This embodiment is based on the above embodiment. In step S4, the rotation and refinement is to avoid mismatch between data patterns and correlation. The data correlation and pattern feature matrix F (the core feature vector of each pattern) are adjusted synchronously through the rotation and refinement framework. Specifically, it includes:

[0048] Construct a common refinement objective function J, and restrict the rank that cannot be directly refined to a differentiable trace refinement term. The process of refinement will promote The rank of the cluster approximates n−c, thus ensuring the final formation of c enterprise-related clusters, as shown below: Limited to: F is the pattern feature matrix, and each column represents the feature center of the pattern. It is a regularization parameter used to balance the distance term and the feature smoothing term; T is the trace, T is the transpose operation, and I is the identity matrix;

[0049] Anchoring the local adaptive approximation matrix S, refining the feature matrix F of the pattern, eigenvalue decomposition, taking... The eigenvectors corresponding to the first c smallest eigenvalues ​​are represented as follows: Find c mutually orthogonal eigenvectors that have the lowest energy under the influence of the Laplace matrix. These correspond precisely to the c main pattern directions of the enterprise data. These eigenvectors constitute the pattern feature matrix, providing accurate pattern coordinates for the next step of refining S.

[0050] The anchoring mode feature matrix F and the refined local adaptive approximation matrix S are solved in closed form using the Lagrange multiplier method, and are expressed as: ; where, normalization factor Ultimately simplified to: ;d ik It is enterprise data d i and enterprise data d k The Euclidean distance; , and These are enterprise data d i Enterprise data d j and enterprise data d k The pattern feature vector (elements of F).

[0051] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S5, enterprise data storage optimization utilizes the c enterprise-related clusters naturally formed by the locally adaptive convergent value matrix after rotation and refinement, and assigns the optimal storage strategy to each enterprise-related cluster; specifically:

[0052] Extract c enterprise-related clusters from the local adaptive approximation matrix, and assign storage strategies to the enterprise data corresponding to the enterprise-related clusters. The strategy parameters are determined by the average characteristics of the data within the cluster.

[0053] like and ,distribute (High-speed solid-state drive, high IOPS);

[0054] like and ,distribute (Medium-speed mechanical hard drive, balancing cost and performance);

[0055] like and ,distribute (Cloud archiving, low access cost);

[0056] in, , , and These are thresholds based on the enterprise's historical storage costs, corresponding to the hot data threshold (values ​​from 0.5 to 1.0), the recent access threshold (values ​​from 0 to 0.5), the lower limit threshold for warm data access frequency (values ​​from 0.2 to 0.5), and the upper limit threshold for cold data timeliness (values ​​from 0.5 to 1.0). and It is the mean of the pattern feature vector of all enterprise data in the k-th enterprise association cluster block on the first dimension (heat projection) and the mean of the second dimension (timeliness projection), which respectively reflect the average heat and average timeliness of the enterprise association cluster block.

[0057] By performing the above operations, this solution addresses the common problems of pattern recognition and enterprise data correlation in general enterprise data intelligent storage optimization methods. These problems include the disconnect between pattern recognition and enterprise data correlation, which disrupts the tightness of business correlations and causes data from the same business flow to cross different storage layers, increasing access latency and costs. This solution constructs a common refinement objective function and ensures bidirectional matching between pattern and enterprise data correlation through a rotating refinement framework. By combining alternating anchor variables, analytical solutions, and feature decomposition, the final result achieves global consistency in both pattern coordinates and enterprise data correlation, providing a reliable foundation for subsequent storage strategy allocation. This, in turn, improves the optimization effect of enterprise data intelligent storage.

[0058] Example 7, see Figure 2 Based on the above embodiments, this embodiment provides an enterprise data intelligent storage optimization system, including an enterprise data acquisition module, a data access pattern initial screening module, a data access pattern clustering module, a rotation and refinement module, and an enterprise data storage optimization module.

[0059] The enterprise data acquisition module acquires enterprise data from multiple sources and constructs an enterprise data feature set after preprocessing.

[0060] The data access pattern screening module uses a soft partitioning objective function to optimize and identify the access patterns of enterprise data for the enterprise data feature set.

[0061] The data access pattern clustering module constructs a local adaptive convergence value matrix and a pattern feature matrix based on the access pattern.

[0062] The rotation and refinement module constructs a joint objective function that integrates correlation and pattern features, and performs rotation and refinement.

[0063] The enterprise data storage optimization module optimizes enterprise data storage based on the clusters formed by the refined local adaptive convergence value matrix.

[0064] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0065] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0066] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for optimizing intelligent storage of enterprise data, characterized in that: The method includes the following steps: Step S1: Enterprise data collection, acquiring multi-source enterprise data, and constructing an enterprise data feature set after preprocessing; Step S2: Initial screening of data access patterns. For the enterprise data feature set, the access patterns of enterprise data are identified by optimizing the soft partitioning objective function. Step S3: Cluster data access patterns, construct a local adaptive convergent value matrix, and construct a pattern feature matrix based on the access patterns; Step S4: Rotation and refinement, construct a joint objective function that integrates correlation and pattern features, and perform rotation and refinement; Step S5: Enterprise data storage optimization. Based on the clusters formed by the refined local adaptive convergence value matrix, enterprise data storage is optimized.

2. The enterprise data intelligent storage optimization method according to claim 1, characterized in that: In step S2, the initial screening of data access patterns involves introducing a heat dissipation factor and a weakening coefficient into the enterprise data feature set, and identifying the potential state of the data through soft partitioning.

3. The enterprise data intelligent storage optimization method according to claim 1, characterized in that: In step S3, the data access pattern clustering specifically includes: Construct a local adaptive convergence value matrix, where the matrix elements represent the correlation between enterprise data, measured by diffuse distance, and also incorporate business correlation strength; The number of modes is limited by adding a rank constraint to the Laplace matrix of the local adaptive approximation matrix; Construct a pattern feature matrix, where the pattern feature vector represents the coordinate vector of enterprise data in the pattern space.

4. The enterprise data intelligent storage optimization method according to claim 1, characterized in that: In step S4, the rotation and refinement is to avoid mismatch between data patterns and correlation, and to synchronously adjust the data correlation and pattern feature matrix through the rotation and refinement framework.

5. The enterprise data intelligent storage optimization method according to claim 1, characterized in that: In step S5, the enterprise data storage optimization utilizes the enterprise-related clusters naturally formed by the locally adaptive convergent value matrix after rotation and refinement, and assigns the optimal storage strategy to each enterprise-related cluster.

6. The enterprise data intelligent storage optimization method according to claim 1, characterized in that: In step S1, the enterprise data collection involves acquiring multi-source enterprise data, including access frequency, recent access time, data size, and number of related business transactions; the collected data is preprocessed to ultimately construct an enterprise data feature set.

7. An enterprise data intelligent storage optimization system, used to implement the enterprise data intelligent storage optimization method as described in any one of claims 1-6, characterized in that: It includes an enterprise data acquisition module, a data access pattern initial screening module, a data access pattern clustering module, a rotation and refinement module, and an enterprise data storage optimization module; The enterprise data acquisition module acquires enterprise data from multiple sources and constructs an enterprise data feature set after preprocessing. The data access pattern screening module uses a soft partitioning objective function to optimize and identify the access patterns of enterprise data for the enterprise data feature set. The data access pattern clustering module constructs a local adaptive convergence value matrix and a pattern feature matrix based on the access pattern. The rotation and refinement module constructs a joint objective function that integrates correlation and pattern features, and performs rotation and refinement. The enterprise data storage optimization module optimizes enterprise data storage based on the clusters formed by the refined local adaptive convergence value matrix.

Images