A distributed parallel data management method and system based on hierarchical storage

Abstract

The application relates to the technical field of data storage, in particular to a distributed parallel data management method and system based on hierarchical storage, which comprises the following steps: obtaining a distributed data set, numbering each data object in the distributed data set to obtain a first data object to an Nth data object, obtaining a data type sequence, a data size sequence and an access frequency sequence of the first data object to the Nth data object, obtaining a first data density to an Nth data density through data density analysis, obtaining a first weight matrix through reference-dependence joint analysis of the first data object to the Nth data object, obtaining a first corrected density to an Nth corrected density through density correction according to the first weight matrix, and obtaining a data storage level allocation result through hierarchical storage according to the first corrected density to the Nth corrected density, so that more accurate and efficient storage management of distributed parallel data is realized.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a distributed parallel data management method and system based on hierarchical storage. Background Technology

[0002] As businesses expand and IT architectures evolve into distributed systems, data is characterized by massive volume, multiple sources, and distributed storage. Traditional centralized data storage and management methods, and storage strategies based on single dimensions such as data type and access frequency, are no longer sufficient to meet the demands of parallel access, efficient storage, and management in distributed environments. For example, in a distributed system, a configuration file on an edge node may be accessed infrequently, but it contains critical parameters for cluster scheduling and has extremely high information value. Conversely, a log file on a central node, while containing important data types and being accessed frequently, often contains repetitive information with limited actual value. This extensive management approach based on superficial characteristics fails to accurately match the actual value of the data, leading to inappropriate allocation of storage resources and impacting the overall system's parallel processing performance and data access efficiency.

[0003] While some research teams have begun to focus on the informational value of data itself, their analysis is often limited to individual data objects. In a distributed parallel computing environment, the value of a data object depends not only on its inherent attributes but also on its position and relationships within the data network. For example, an index file may not have high information density, but it is frequently referenced by multiple computing nodes in a distributed system, serving as a key entry point for accessing multiple important data shards. Its actual value far exceeds the value reflected by its information density. Current technologies, lacking consideration of the complex references and dependencies between distributed data when assessing data value, often lead to the incorrect allocation of such high-network-value data to low-performance storage tiers. This becomes a bottleneck for the system's parallel processing capabilities, impacting data access performance and overall system management efficiency. Therefore, how to more accurately assess the informational value of data in a distributed parallel environment and, based on this, achieve efficient and reliable hierarchical storage management has become a pressing technical problem. Summary of the Invention

[0004] This invention provides a distributed parallel data management method and system based on hierarchical storage to solve the problem that in distributed data storage management, the neglect of the correlation between data leads to inaccurate information value assessment, which in turn causes unreasonable storage allocation and affects the parallel performance of the system.

[0005] To achieve the above objectives, in one aspect, the present invention provides a distributed parallel data management method based on hierarchical storage, the method comprising:

[0006] Obtain the distributed dataset, number each data object in the distributed dataset to obtain the first data object to the Nth data object, where N represents the number of data objects; read the data type, data size, and access frequency of the first data object to the Nth data object to obtain the data type sequence, data size sequence, and access frequency sequence.

[0007] Based on the data type sequence, data size sequence, and access frequency sequence, the first data density to the Nth data density are obtained through data density analysis.

[0008] The first weight matrix is ​​obtained by performing reference-dependency joint analysis on the first data object to the Nth data object. The first corrected density to the Nth corrected density is obtained by density correction based on the first data density to the Nth data density and the first weight matrix.

[0009] Based on the first corrected density to the Nth corrected density and the number of storage levels, a hierarchical storage is performed to obtain the data storage hierarchy allocation result. Based on the data storage hierarchy allocation result, the first data object to the Nth data object are configured for hierarchical storage.

[0010] Furthermore, the method for obtaining a data type sequence, a data size sequence, and an access frequency sequence by reading the data type, data size, and access frequency of the first data object to the Nth data object includes:

[0011] The data type sequence is obtained by reading the data type identifiers of the first data object to the Nth data object according to the preset sampling interval.

[0012] The data size sequence is obtained by reading the storage space usage of the first data object to the Nth data object.

[0013] The access frequency sequence is obtained by reading the access counts of the first data object to the Nth data object within a preset statistical period.

[0014] Furthermore, the method for obtaining the first to Nth data densities through data density analysis based on the data type sequence, data size sequence, and access frequency sequence includes:

[0015] Based on the data type sequence, data size sequence, and access frequency sequence, the information content representation data is calculated as follows:

[0016] .

[0017] in It is the first in the data size sequence One element, It is the maximum value in the data size sequence. It is the first in the access frequency sequence One element, It is the maximum value in the access frequency sequence. It is the first in the data type sequence One element, It is the maximum value in the data type sequence. It is a preset smoothing factor. It is the first Each piece of information represents the data.

[0018] Each information representation data point is normalized to obtain a normalized information representation data sequence.

[0019] The first to Nth data densities are calculated by quotienting each normalized information content representation data in the data sequence with its corresponding data size.

[0020] Further, the method of obtaining a first weight matrix by performing reference-dependency joint analysis on the first data object to the Nth data object, and obtaining a first corrected density to the Nth corrected density based on the first data density to the Nth data density and the first weight matrix through density correction includes:

[0021] By scanning the file header, metadata, and content of data objects, we can identify data objects containing file path references and identifier references, and obtain data object pairs with dependency and reference relationships.

[0022] The strength of the dependency relationship between data object pairs is calculated as follows:

[0023] ;

[0024] in It is the number of times it is depended on. It is the total number of dependencies. It is the strength of the dependency relationship;

[0025] The strength of the reference relationship between data object pairs is calculated as follows:

[0026] ;

[0027] in It refers to the number of citations. It is the total number of citations. It refers to the strength of the reference relationship;

[0028] The initial fusion matrix is ​​constructed by superimposing the dependency strength and the reference strength, and the first weight matrix is ​​obtained by performing transitive correlation analysis on the initial fusion matrix.

[0029] The first corrected density to the Nth corrected density are obtained by performing density correction on the first data density to the Nth data density according to the first weight matrix.

[0030] Furthermore, the method of superimposing dependency strength and reference strength to construct an initial fusion matrix, and performing transitive correlation analysis on the initial fusion matrix to obtain the first weight matrix includes:

[0031] Dimensions are constructed based on dependency strength and reference strength. The initial fusion matrix, where This refers to the total number of data objects, specifically:

[0032] When the The data object and the first When multiple data objects have dependency or reference relationships, matrix elements These are the strength values ​​for the corresponding dependency and reference strengths;

[0033] When the The data object and the first When there are no dependencies or references between data objects, the matrix element values =0;

[0034] Calculate the first The connectivity of each data object is:

[0035] ;

[0036] in It is the total number of data objects. This is an indicator function; its value is true when the condition is true. Otherwise , The threshold value for connection determination is within a certain range. , It is the first weight matrix. Line number Column elements; It is the first weight matrix. Line number Column elements; It is the index variable for traversal. It is the first The degree of connectivity of each data object;

[0037] Set the initial condition for iteration as follows: The initial fusion matrix is ​​subjected to transitive correlation iterative optimization according to the iterative formula, which is:

[0038] ;

[0039] in It is the first After the nth iteration The data object and the first The strength value of each data object. It is a preset transmission attenuation coefficient, with a value range of... , It is the intermediate node traversal index. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object;

[0040] The iteration process is terminated when the change in the maximum intensity value between two consecutive iterations is less than the preset convergence threshold, and the first weight matrix is ​​obtained.

[0041] Further, the method for obtaining the first corrected density to the Nth corrected density by density correction of the first data density to the Nth data density according to the first weight matrix includes:

[0042] The fusion enhancement factor for each data object is calculated based on the first data density to the Nth data density and the first weight matrix as follows:

[0043] .

[0044] in It is the first weight matrix. Line number Column elements, It is the first Data density of a data object It is the enhancement coefficient, and its value range is... , It is the first Fusion enhancement factor for each data object.

[0045] The centrality index of each data object is calculated based on the first weight matrix as follows:

[0046] .

[0047] in It is the first weight matrix. Line number Column elements, It is the first weight matrix. Line number Column elements, It is the index variable for traversal. It is the total number of data objects. It is the first The centrality index of a data object.

[0048] Based on the first to Nth data densities, the fusion enhancement factor for each data object, and the centrality index of each data object, the first to Nth corrected densities are obtained through weighted fusion calculation as follows:

[0049] .

[0050] in It is the first Data density of a data object , , These are preset fusion weight coefficients. And the range of values ​​is , It is the first Corrected density of each data object.

[0051] Furthermore, the method for obtaining the data storage hierarchy allocation result by performing hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels includes:

[0052] A distributed data association graph is constructed by using data objects as a vertex set and the weight values ​​of the first weight matrix as edge weights to form an edge set.

[0053] Distributed data subgraphs are obtained by dividing the distributed data association graph into subgraphs.

[0054] Data objects belonging to the same distributed data subgraph are assigned to the same storage level as the data storage level allocation result.

[0055] Furthermore, the method for obtaining distributed data subgraphs by partitioning the distributed data association graph into subgraphs includes:

[0056] The weight values ​​for each storage tier are obtained by weighting the storage tiers according to their performance levels, and the constraint is that the total size of the data objects allocated in each storage tier does not exceed the preset capacity limit.

[0057] The data objects are sorted in descending order of their modified density and then allocated to the highest available level using a greedy algorithm to form a distributed data subgraph.

[0058] Furthermore, the method also includes:

[0059] Before allocating data objects belonging to the same distributed data subgraph to the same storage level as the data storage level allocation result, the optimal storage level allocation adjustment is performed on the data objects belonging to the same distributed data subgraph. Specifically:

[0060] The optimization goal is set as follows: maximize the cumulative corrected density of all data objects within each storage tier.

[0061] The vertex adjustment gain is obtained by performing hierarchical migration calculations on the vertices of the association graph:

[0062] .

[0063] in It is the first Vertices in the association graph Storage hierarchy before adjustment It is the first Vertices in the association graph Adjusted storage hierarchy It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is the first The vertices of the association graph are from the storage level. to storage tier Adjust the vertex gain.

[0064] Vertex migration optimization is performed when the boundary vertex adjustment gain is greater than zero and the constraints are met after migration. Optimization is terminated when the rate of change of the optimization target is less than a preset threshold for two consecutive optimizations, and the optimal storage level allocation is obtained.

[0065] Based on the same inventive concept, this invention provides a distributed parallel data management system based on hierarchical storage. The system includes: a data reading module, a density analysis module, a reference-dependency analysis module, and a hierarchical allocation module, which are connected in sequence.

[0066] The data reading module is used to acquire a distributed dataset, number each data object in the distributed dataset to obtain the first data object to the Nth data object, where N represents the number of data objects; and read the data type, data size, and access frequency of the first data object to the Nth data object to obtain the data type sequence, data size sequence, and access frequency sequence.

[0067] The density analysis module is used to obtain the first data density to the Nth data density through data density analysis based on the data type sequence, data size sequence, and access frequency sequence.

[0068] The reference-dependency analysis module is used to perform joint reference-dependency analysis on the first data object to the Nth data object to obtain the first weight matrix, and obtain the first corrected density to the Nth corrected density by density correction based on the first data density to the Nth data density and the first weight matrix.

[0069] The hierarchical allocation module is used to perform hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels to obtain the data storage hierarchical allocation result, and to configure hierarchical storage for the first data object to the Nth data object based on the data storage hierarchical allocation result.

[0070] Compared with existing technologies, the beneficial effects of this invention are: it enables preliminary value assessment of distributed data through density analysis, and achieves more accurate measurement of the value of data in the distributed network by correcting the density of distributed data through reference-dependency joint analysis. Furthermore, it performs hierarchical storage allocation based on the corrected data value, thereby realizing efficient and parallel management of data in a distributed environment, and thus improving the overall system's storage resource utilization and data access performance. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart of a distributed parallel data management method based on hierarchical storage according to Embodiment 1 of the present invention;

[0073] Figure 2 This is a schematic diagram of the module composition of a distributed parallel data management system based on hierarchical storage according to Embodiment 2 of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Example 1: As Figure 1As shown, this embodiment provides a distributed parallel data management method based on hierarchical storage, the method comprising:

[0076] The distributed dataset is acquired, and each data object in the dataset is numbered from the first data object to the Nth data object, where N represents the number of data objects. The data type, size, and access frequency of the first to Nth data objects are then read to obtain the data type sequence, data size sequence, and access frequency sequence, respectively. First, the distributed dataset to be processed is uniformly numbered to provide an identification basis for subsequent analysis and storage allocation. The numbering method uses a continuous integer sequence to facilitate matrix operations and index lookups. Attributes such as data type, size, and access frequency reflect the basic characteristics of the data and provide the raw input for calculating data density.

[0077] For example, the distributed dataset obtained from a certain platform includes: Data Object 1: User Profile Master File, Data Object 2: User Behavior Log, Data Object 3: Product Core Information Table, Data Object 4: Product Image Index, Data Object 5: Order Transaction Records, Data Object 6: Database Configuration File, Data Object 7: Recommendation Algorithm Model, and Data Object 8: Pricing Strategy Rules. These 8 data objects are numbered 1-8 in sequence.

[0078] Based on the data type sequence, data size sequence, and access frequency sequence, data density analysis is used to obtain the first to Nth data densities. This step transforms the multidimensional attributes of the data into a single density index. A higher density value indicates that the data contains more effective information within a unit of storage space, thus possessing higher storage value.

[0079] A first weight matrix is ​​obtained by performing a joint reference-dependency analysis on the first data object to the Nth data object. Based on the density of the first data object to the Nth data object and the first weight matrix, density corrections are applied to obtain the first to Nth corrected densities. Some data, although not inherently dense, may have high data value as references or dependencies of other important data. By constructing a first weight matrix between the data and analyzing the dependencies and references between them to correct the original density, a more accurate data value assessment can be obtained.

[0080] Based on the first to Nth corrected densities and the number of storage levels, a hierarchical storage allocation result is obtained. Then, hierarchical storage configuration is applied to the first to Nth data objects according to this allocation result. Data objects are assigned to different storage levels based on their data value, thereby achieving differentiated storage management.

[0081] It should be noted that the method for obtaining the data type sequence, data size sequence, and access frequency sequence by reading the data type, data size, and access frequency of the first data object to the Nth data object includes:

[0082] The data type sequence is obtained by reading the data type identifiers of the first data object to the Nth data object according to a preset sampling interval. Reading the data type identifiers according to the preset sampling interval avoids a full scan of all data and improves processing efficiency. The preset sampling interval is dynamically adjusted according to the size of the dataset and processing capacity.

[0083] For example: Data object 1: JSON format, type identifier 3; Data object 2: Log format, type identifier 1; Data object 3: Database format, type identifier 5; Data object 4: XML format, type identifier 4; Data object 5: CSV format, type identifier 2; Data object 6: Configuration file format, type identifier 6; Data object 7: Binary model, type identifier 7; Data object 8: JSON format, type identifier 3, resulting in the data type sequence. .

[0084] The data size sequence is obtained by reading the storage space usage of the first data object to the Nth data object. The data size directly reflects the storage cost of the data.

[0085] For example: Data object 1: 2.5GB, Data object 2: 15.8GB, Data object 3: 8.2GB, Data object 4: 0.3GB, Data object 5: 12.1GB, Data object 6: 0.5GB, Data object 7: 1.2GB, Data object 8: 0.1GB, then the data size sequence is as follows: .

[0086] The access frequency sequence is obtained by reading the access counts of the first data object to the Nth data object within a preset statistical period. Access frequency reflects the popularity of data and is an important indicator of its value.

[0087] For example: If the preset statistical period is one month, and data object 1 is accessed 1200 times, data object 2 450 times, data object 3 2800 times, data object 4 3200 times, data object 5 800 times, data object 6 50 times, data object 7 150 times, and data object 8 600 times within one month, then the access frequency sequence is: .

[0088] It should be noted that the method for obtaining the first to Nth data densities through data density analysis based on the data type sequence, data size sequence, and access frequency sequence includes:

[0089] Based on the data type sequence, data size sequence, and access frequency sequence, the information content representation data is calculated as follows:

[0090] .

[0091] in It is the first in the data size sequence One element, It is the maximum value in the data size sequence. It is the first in the access frequency sequence One element, It is the maximum value in the access frequency sequence. It is the first in the data type sequence One element, It is the maximum value in the data type sequence. It is a preset smoothing factor. It is the first The information content of the data is represented by a logarithmic function, and a smoothing factor is introduced. This avoids division by zero errors and ensures computational stability based on the theoretical framework of information entropy.

[0092] For example: setting a smoothing factor Maximum value of data type Maximum data size Maximum access frequency Then the information content of data object 1 is calculated to represent the data. Similarly, the information content of the other seven data points can be calculated to represent the data. , , , , , , .

[0093] Each information representation data point is normalized to obtain a normalized information representation data sequence.

[0094] For example: performing max-min normalization on each information content representation data yields the information content representation data sequence. .

[0095] The first to Nth data densities are calculated by quotienting the normalized information content of each data point in the data sequence with its corresponding data size. Dividing the normalized information content by the data size yields the information density per unit storage space, reflecting the fundamental value density of the data.

[0096] For example: the quotient of each normalized information content representation data in the normalized information content representation data sequence and its corresponding data size is used to calculate the data density from the first to the Nth data density. .

[0097] It should be noted that the method of obtaining the first weight matrix by performing reference-dependency joint analysis on the first data object to the Nth data object, and obtaining the first corrected density to the Nth corrected density based on the first data density to the Nth data density and the first weight matrix through density correction includes:

[0098] By scanning the file header, metadata, and content of data objects, we can identify data objects containing file path references and identifier references, thus obtaining pairs of data objects with dependency and reference relationships.

[0099] The strength of the dependency relationship between data object pairs is calculated as follows:

[0100] .

[0101] in It is the number of times it is depended on. It is the total number of dependencies. It is the strength of the dependency relationship.

[0102] The more times an object is depended on, the more important it is in the data network. The strength of the dependency relationship reflects the degree to which a data object is depended on by other objects.

[0103] For example, by scanning the file header, metadata, and content of data objects, we can identify data objects containing file path references and identifier references. This yields data such as: product image index references core product information (i.e., data object 4 references data object 3); order records reference user profiles (i.e., data object 5 references data object 1); the recommendation model references user profiles and product information (i.e., data object 7 references data objects 1 and 3); pricing strategies reference product information (i.e., data object 8 references data object 3); data object 1 references data objects 2 and 4; data object 2 references data object 8; and data object 3 references data object 6. Taking data object 1 as an example, its reference relationships are: referenced by data object 5 and data object 7. Assuming it is referenced 5 times by data object 5 and 2 times by data object 7, the total number of dependencies Nt = 7. This further allows us to determine the dependency strength between data object 1 and data object 5. The strength of the dependency between data object 1 and data object 7 .

[0104] The strength of the reference relationship between data object pairs is calculated as follows:

[0105] .

[0106] in It refers to the number of citations. It is the total number of citations. It refers to the strength of the reference relationship.

[0107] The strength of a reference relationship reflects how frequently a data object actively references other objects; objects with high reference frequency are the core nodes in data processing.

[0108] For example, taking data object 1 as an example, assuming it references data object 2 8 times and data object 4 4 times, with a total reference frequency of 12 times, then the strength of the reference relationship between data object 1 and data object 2 is... The strength of the reference relationship between data object 1 and data object 4 .

[0109] The initial fusion matrix is ​​constructed by superimposing the strengths of dependency and reference relationships. Transitive correlation analysis is then performed on this initial fusion matrix to obtain the first weight matrix. Further analysis using transitive correlation analysis reveals indirect relationships between data. Some data, although not directly referenced or dependent, possess transitive value through intermediate nodes, thus influencing the data's value assessment.

[0110] For example, when the strength of dependency relationship and the strength of reference relationship are superimposed to obtain the initial fusion matrix, the strength value between data object 1 and data object 2 is 0.67, the strength value between data object 1 and data object 4 is 0.33, the strength value between data object 1 and data object 5 is 0.71, and the strength value between data object 1 and data object 7 is 0.29. Similarly, the strength values ​​between other data object pairs can be obtained to form the initial fusion matrix.

[0111] The first to Nth data densities are corrected using the first weight matrix to obtain the first to Nth corrected densities. This correction of the original data densities enhances the value of the data objects.

[0112] It should be noted that the method of constructing an initial fusion matrix by superimposing dependency strength and reference strength, and then performing transitive correlation analysis on the initial fusion matrix to obtain the first weight matrix includes:

[0113] Dimensions are constructed based on dependency strength and reference strength. The initial fusion matrix, where This refers to the total number of data objects, specifically:

[0114] When the The data object and the first When multiple data objects have dependency or reference relationships, matrix elements These are the strength values ​​for the corresponding dependency and reference strengths;

[0115] When the The data object and the first When there are no dependencies or references between data objects, the matrix element values =0;

[0116] For example: the dependency and reference strength of the above 8 data objects are combined to form The initial fusion matrix is:

[0117] .

[0118] Calculate the first The connectivity of each data object is:

[0119] ;

[0120] in It is the total number of data objects. This is an indicator function; its value is true when the condition is true. Otherwise , The threshold value for connection determination is within a certain range. , It is the first weight matrix. Line number Column elements; It is the first weight matrix. Line number Column elements; It is the index variable for traversal. It is the first The degree of connectivity of each data object;

[0121] Connectivity reflects the degree of connectivity of each data object in the overall data network. Objects with high connectivity are key nodes in the network and have a significant impact on other objects.

[0122] For example: setting a connection determination threshold The value is 0.2. Taking data object 1 as an example, its connectivity is calculated as its out-degree: =0.67>0.2, =0.33>0.2, =0.71>0.2, =0.29>0.2, a total of 4. In-degree: =0.67>0.2, =0.33>0.2, =0.71>0.2, =0.29>0.2, there are 4 in total, then =4+4=8, the connectivity of data object 2 is calculated as its out-degree: =0.67, =1. Degree: =0.67, =1 =2, and similarly, the connectivity of all other data objects can be calculated.

[0123] Set the initial condition for iteration as follows: The initial fusion matrix is ​​subjected to transitive correlation iterative optimization according to the iterative formula, which is:

[0124] ;

[0125] in It is the first After the nth iteration The data object and the first The strength value of each data object. It is a preset transmission attenuation coefficient, with a value range of... , It is the intermediate node traversal index. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object;

[0126] Indirect effects can be identified through iterative formulas, and a transmission attenuation coefficient is introduced during the iteration process to ensure that the effects over long distances gradually weaken.

[0127] For example: setting the transmission attenuation coefficient λ=0.05, taking the first iteration of data object 1 and data object 2 as an example, .

[0128] The iteration process is terminated when the change in the maximum intensity value between two consecutive iterations is less than the preset convergence threshold, and the first weight matrix is ​​obtained.

[0129] The preset convergence threshold is set to 0.025. After iteration until the termination of the iteration, the first weight matrix is ​​obtained as follows:

[0130] .

[0131] It should be noted that the method for obtaining the first corrected density to the Nth corrected density by performing density correction on the first data density to the Nth data density according to the first weight matrix includes:

[0132] The fusion enhancement factor for each data object is calculated based on the first data density to the Nth data density and the first weight matrix as follows:

[0133] .

[0134] in It is the first weight matrix. Line number Column elements, It is the first Data density of a data object It is the enhancement coefficient, and its value range is... , It is the first Fusion enhancement factor for each data object.

[0135] The fusion enhancement factor for each object is calculated using the first weight matrix and data density, reflecting the value enhancement gained by the object due to mutual references and dependencies.

[0136] For example, taking the fusion enhancement factor of data object 1 as an example, if the enhancement coefficient is set to 1, then... .

[0137] The centrality index of each data object is calculated based on the first weight matrix as follows:

[0138] .

[0139] in It is the first weight matrix. Line number Column elements, It is the first weight matrix. Line number Column elements, It is the index variable for traversal. It is the total number of data objects. It is the first The centrality index of a data object.

[0140] The centrality index of a data object represents the influence of that data object on other data objects and the sum of the dependencies of other data objects on it, reflecting the importance of that data object in the entire data.

[0141] For example: the centrality index of data object 1 is .

[0142] Based on the first to Nth data densities, the fusion enhancement factor for each data object, and the centrality index of each data object, the first to Nth corrected densities are obtained through weighted fusion calculation as follows:

[0143] .

[0144] in It is the first Data density of a data object , , These are preset fusion weight coefficients. And the range of values ​​is , It is the first Corrected density of each data object.

[0145] For example: when the fusion weight coefficients are set as follows: , , At that time, the first Corrected density of data objects That is, although the original data density of the first data object, the user profile master file, is low, its data value is improved after analysis and correction due to its strong references or dependencies with other objects. Similarly, the corrected density of other data objects can be obtained.

[0146] It should be noted that the method for obtaining the data storage hierarchy allocation result by performing hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels includes:

[0147] A distributed data association graph is constructed by using data objects as a vertex set and the weight values ​​of the first weight matrix as edge weights to form an edge set.

[0148] For example: using 8 data objects as vertices arrive Forming a vertex set The edge set contains all The edge.

[0149] Distributed data subgraphs are obtained by dividing the distributed data association graph into subgraphs.

[0150] Assigning data objects belonging to the same distributed data subgraph to the same storage level is the result of data storage level allocation. This ensures that closely referenced and dependent data can achieve consistent access performance within the same storage environment.

[0151] It should be noted that the method for obtaining distributed data subgraphs by partitioning a distributed data association graph into subgraphs includes:

[0152] The weight values ​​for each storage tier are obtained by weighting the storage tiers according to their performance levels, and the constraint is that the total size of the data objects allocated in each storage tier does not exceed the preset capacity limit.

[0153] The data objects are sorted in descending order of their modified density and then allocated to the highest available level using a greedy algorithm to form a distributed data subgraph.

[0154] Capacity constraints ensure that the storage load at each tier does not exceed hardware limits, and a greedy algorithm is used for initial data allocation so that high-density data is allocated to high-performance tiers.

[0155] It should be noted that the method further includes:

[0156] Before allocating data objects belonging to the same distributed data subgraph to the same storage level as the data storage level allocation result, the optimal storage level allocation adjustment is performed on the data objects belonging to the same distributed data subgraph. Specifically:

[0157] The optimization goal is set as follows: maximize the cumulative corrected density of all data objects within each storage tier.

[0158] The specific solution expression for objective 1 is as follows: ,in It is the first A vertex in the relational graph, It is the first Layered storage hierarchy, It is the first Corrected density of each data object This is the weight value of the first level. It represents the total number of storage levels.

[0159] The vertex adjustment gain is obtained by performing hierarchical migration calculations on the vertices of the association graph:

[0160] .

[0161] in It is the first Vertices in the association graph Storage hierarchy before adjustment It is the first Vertices in the association graph Adjusted storage hierarchy It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is the first The vertices of the association graph are from the storage level. to storage tier Adjust the vertex gain.

[0162] Vertex migration optimization is performed when the boundary vertex adjustment gain is greater than zero and the constraints are met after migration. Optimization is terminated when the rate of change of the optimization target is less than a preset threshold for two consecutive optimizations, and the optimal storage level allocation is obtained.

[0163] Example 2: Based on the same inventive concept, such as Figure 2 As shown in the figure, this embodiment provides a distributed parallel data management system based on hierarchical storage. The system includes: a data reading module, a density analysis module, a reference-dependency analysis module, and a hierarchical allocation module, which are connected in sequence.

[0164] The data reading module is used to acquire a distributed dataset, number each data object in the distributed dataset to obtain the first data object to the Nth data object, where N represents the number of data objects; and read the data type, data size, and access frequency of the first data object to the Nth data object to obtain the data type sequence, data size sequence, and access frequency sequence.

[0165] The density analysis module is used to obtain the first data density to the Nth data density through data density analysis based on the data type sequence, data size sequence, and access frequency sequence.

[0166] The reference-dependency analysis module is used to perform joint reference-dependency analysis on the first data object to the Nth data object to obtain the first weight matrix, and obtain the first corrected density to the Nth corrected density by density correction based on the first data density to the Nth data density and the first weight matrix.

[0167] The hierarchical allocation module is used to perform hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels to obtain the data storage hierarchical allocation result, and to configure hierarchical storage for the first data object to the Nth data object based on the data storage hierarchical allocation result.

[0168] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in Embodiment 1 of the method, and will not be elaborated here.

[0169] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed parallel data management method based on hierarchical storage, characterized in that, The method includes: Obtain a distributed dataset, number each data object in the distributed dataset to obtain the first data object to the Nth data object, where N represents the number of data objects; read the data type, data size, and access frequency of the first data object to the Nth data object to obtain the data type sequence, data size sequence, and access frequency sequence. Based on the data type sequence, data size sequence, and access frequency sequence, the first data density to the Nth data density are obtained through data density analysis. The first weight matrix is ​​obtained by performing reference-dependency joint analysis on the first data object to the Nth data object. The first corrected density to the Nth corrected density is obtained by density correction based on the first data density to the Nth data density and the first weight matrix. Based on the first corrected density to the Nth corrected density and the number of storage levels, the data storage hierarchy allocation result is obtained by performing hierarchical storage. Based on the data storage hierarchy allocation result, the first data object to the Nth data object are configured for hierarchical storage. The method for obtaining the first to Nth data densities through data density analysis based on the data type sequence, data size sequence, and access frequency sequence includes: Information content representation data is calculated based on the data type sequence, data size sequence, and access frequency sequence. Each information content representation data is normalized to obtain a normalized information content representation data sequence. The first data density to the Nth data density is calculated by quotienting each normalized information content representation data in the normalized information content representation data sequence with its corresponding data size. The reference-dependency joint analysis includes: identifying data objects containing file path references and identifier references by scanning the file header, metadata, and content of data objects to obtain data object pairs with dependency and reference relationships; calculating the dependency strength and reference strength of the data object pairs; superimposing the dependency strength and reference strength to construct an initial fusion matrix; setting initial iteration conditions, and performing transitive correlation iterative optimization on the initial fusion matrix according to the iterative formula, wherein the iterative formula includes a preset transitive attenuation coefficient, and terminating the iteration process when the maximum strength value change between two adjacent iterations is less than a preset convergence threshold to obtain the first weight matrix; The density correction includes: calculating the fusion enhancement factor of each data object based on the first data density to the Nth data density and the first weight matrix, wherein the enhancement coefficient ranges from [0,1]; calculating the centrality index of each data object based on the first weight matrix; and calculating the first corrected density to the Nth corrected density through weighted fusion based on the first data density to the Nth data density, the fusion enhancement factor of each data object, and the centrality index of each data object. The hierarchical storage includes: constructing a distributed data association graph by using data objects as a vertex set and the weight values ​​of the first weight matrix as edge weights to form an edge set; dividing the distributed data association graph into subgraphs to obtain distributed data subgraphs; and allocating data objects belonging to the same distributed data subgraph to the same storage level as the data storage level allocation result.

2. The distributed parallel data management method based on hierarchical storage according to claim 1, characterized in that, The method for obtaining a data type sequence, a data size sequence, and an access frequency sequence by reading the data type, data size, and access frequency of the first data object to the Nth data object includes: The data type sequence is obtained by reading the data type identifiers of the first data object to the Nth data object according to the preset sampling interval; The data size sequence is obtained by reading the storage space usage of the first data object to the Nth data object; The access frequency sequence is obtained by reading the access counts of the first data object to the Nth data object within a preset statistical period.

3. The distributed parallel data management method based on hierarchical storage according to claim 1, characterized in that, The method of constructing an initial fusion matrix by superimposing dependency strength and reference strength, and obtaining the first weight matrix by performing transitive correlation analysis on the initial fusion matrix includes: Dimensions are constructed based on dependency strength and reference strength. The initial fusion matrix, where This refers to the total number of data objects, specifically: When the The data object and the first When multiple data objects have dependency or reference relationships, matrix elements These are the strength values ​​for the corresponding dependency and reference strengths; When the The data object and the first When there are no dependencies or references between data objects, the matrix element values =0; Calculate the first The connectivity of each data object is: ； in It is the total number of data objects. This is an indicator function; its value is true when the condition is true. Otherwise , The threshold value for connection determination is within a certain range. , It is the first weight matrix. Line number Column elements; It is the first weight matrix. Line number Column elements; It is the index variable for traversal. It is the first The degree of connectivity of each data object; Set the initial condition for iteration as follows: The initial fusion matrix is ​​subjected to transitive correlation iterative optimization according to the iterative formula, which is: ； in It is the first After the nth iteration The data object and the first The strength value of each data object. It is a preset transmission attenuation coefficient, with a value range of... , It is an intermediate node traversal index. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object. It is the first After the nth iteration The data object and the first The strength value of each data object; The iteration process is terminated when the change in the maximum intensity value between two consecutive iterations is less than the preset convergence threshold, and the first weight matrix is ​​obtained.

4. The distributed parallel data management method based on hierarchical storage according to claim 1, characterized in that, The method for obtaining the first corrected density to the Nth corrected density by performing density correction on the first data density to the Nth data density according to the first weight matrix includes: The fusion enhancement factor for each data object is calculated based on the first data density to the Nth data density and the first weight matrix as follows: ； in It is the first weight matrix. Line number Column elements, It is the first Data density of a data object It is the enhancement coefficient, and its value range is... , It is the first Fusion enhancement factor for individual data objects; The centrality index of each data object is calculated based on the first weight matrix as follows: ； in It is the first weight matrix. Line number Column elements, It is the first weight matrix. Line number Column elements, It is the index variable for traversal. It is the total number of data objects. It is the first Centrality indicators of individual data objects; Based on the first to Nth data densities, the fusion enhancement factor for each data object, and the centrality index of each data object, the first to Nth corrected densities are obtained through weighted fusion calculation as follows: ； in It is the first Data density of a data object , , These are preset fusion weight coefficients. And the range of values ​​is , It is the first Corrected density of each data object.

5. The distributed parallel data management method based on hierarchical storage according to claim 1, characterized in that, The method for obtaining the data storage hierarchy allocation result by performing hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels includes: A distributed data association graph is constructed by using data objects as a set of vertices and the weight values ​​of the first weight matrix as edge weights to form an edge set. Distributed data subgraphs are obtained by dividing the distributed data association graph into subgraphs. Data objects belonging to the same distributed data subgraph are assigned to the same storage level as the data storage level allocation result.

6. The distributed parallel data management method based on hierarchical storage according to claim 4, characterized in that, The method for obtaining distributed data subgraphs by partitioning a distributed data association graph into subgraphs includes: The weight values ​​for each storage tier are obtained by weighting the storage tier performance levels, and the constraint is that the total size of the data objects allocated in each storage tier does not exceed the preset capacity limit. The data objects are sorted in descending order of their modified density and then allocated to the highest available level using a greedy algorithm to form a distributed data subgraph.

7. A distributed parallel data management method based on hierarchical storage according to claim 6, characterized in that, The method further includes: Before allocating data objects belonging to the same distributed data subgraph to the same storage level as the data storage level allocation result, the optimal storage level allocation adjustment is performed on the data objects belonging to the same distributed data subgraph. Specifically: The optimization objective is set as follows: maximize the cumulative corrected density value of all data objects in each storage tier; The vertex adjustment gain is obtained by performing hierarchical migration calculations on the vertices of the association graph: ； in It is the first Vertices in the association graph Storage hierarchy before adjustment It is the first Vertices in the association graph Adjusted storage hierarchy It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is a vertex of the association graph With storage tier Other data objects The sum of their intensity values, It is the first The vertices of the association graph are from the storage level. to storage tier Adjust the vertex gain; Vertex migration optimization is performed when the boundary vertex adjustment gain is greater than zero and the constraints are met after migration. Optimization is terminated when the rate of change of the optimization target is less than a preset threshold for two consecutive optimizations, and the optimal storage level allocation is obtained.

8. A distributed parallel data management system based on hierarchical storage, used to execute the method according to any one of claims 1-7, characterized in that, The system includes: a data reading module, a density analysis module, a reference-dependency analysis module, and a hierarchical allocation module, which are connected in sequence. The data reading module is used to acquire a distributed dataset, number each data object in the distributed dataset to obtain the first data object to the Nth data object, where N represents the number of data objects; and read the data type, data size, and access frequency of the first data object to the Nth data object to obtain the data type sequence, data size sequence, and access frequency sequence. The density analysis module is used to obtain the first data density to the Nth data density through data density analysis based on the data type sequence, data size sequence, and access frequency sequence. The reference-dependency analysis module is used to perform joint reference-dependency analysis on the first data object to the Nth data object to obtain the first weight matrix, and to obtain the first corrected density to the Nth corrected density based on the first data density to the Nth data density and the first weight matrix through density correction. The hierarchical allocation module is used to perform hierarchical storage based on the first corrected density to the Nth corrected density and the number of storage levels to obtain the data storage hierarchical allocation result, and to configure hierarchical storage for the first data object to the Nth data object based on the data storage hierarchical allocation result; The method for obtaining the first to Nth data densities through data density analysis based on the data type sequence, data size sequence, and access frequency sequence includes: Information content representation data is calculated based on the data type sequence, data size sequence, and access frequency sequence. Each information content representation data is normalized to obtain a normalized information content representation data sequence. The first data density to the Nth data density is calculated by quotienting each normalized information content representation data in the normalized information content representation data sequence with its corresponding data size. The reference-dependency joint analysis includes: identifying data objects containing file path references and identifier references by scanning the file header, metadata, and content of data objects to obtain data object pairs with dependency and reference relationships; calculating the dependency strength and reference strength of the data object pairs; superimposing the dependency strength and reference strength to construct an initial fusion matrix; setting initial iteration conditions, and performing transitive correlation iterative optimization on the initial fusion matrix according to the iterative formula, wherein the iterative formula includes a preset transitive attenuation coefficient, and terminating the iteration process when the maximum strength value change between two adjacent iterations is less than a preset convergence threshold to obtain the first weight matrix; The density correction includes: calculating the fusion enhancement factor of each data object based on the first data density to the Nth data density and the first weight matrix, wherein the enhancement coefficient ranges from [0,1]; calculating the centrality index of each data object based on the first weight matrix; and calculating the first corrected density to the Nth corrected density through weighted fusion based on the first data density to the Nth data density, the fusion enhancement factor of each data object, and the centrality index of each data object. The hierarchical storage includes: constructing a distributed data association graph by using data objects as a vertex set and the weight values ​​of the first weight matrix as edge weights to form an edge set; dividing the distributed data association graph into subgraphs to obtain distributed data subgraphs; and allocating data objects belonging to the same distributed data subgraph to the same storage level as the data storage level allocation result.

Images