A Traceability Data Management Method and System Based on Clustering and Online Prediction

Abstract

The present invention discloses a method and system for tracing data management based on clustering and online prediction. The method comprises: traversing the tracing nodes in the tracing graph, calculating the node importance of each tracing node according to the dependency relationship between the tracing nodes, and sorting them; setting at least two nodes whose node importance is greater than a threshold as central nodes, and assigning initial labels, and then propagating the initial labels to other nodes according to the order of dependency and node importance, to determine multiple tracing classes; obtaining access records of each tracing class, and calculating the access correlation between the tracing classes; online prediction of the access probability of the tracing class at the target moment, and pre-fetching the tracing class with an access probability greater than the access threshold as hot data to a solid-state hard disk; compressing and storing the tracing class that has not been accessed within a preset time interval as cold data to a mechanical hard disk. The method achieves the effect of accelerating the access speed of tracing data, reducing the query time of tracing data, and reducing the storage overhead of tracing data.

Description

Technical Field

[0001] The present invention belongs to the field of computer data processing, and more specifically, relates to a traceability data management method and system based on clustering and online prediction. Background Art

[0002] Academic institutions at home and abroad have designed a variety of provenance systems tailored to different needs. These include the provenance-aware storage system PASS, implemented by Harvard University's Systems Research Group, and the provenance collection system SPADE, published by Ashish Gehani et al. However, these general-purpose provenance storage systems store provenance information in chronological order, without clustering the data. Furthermore, they lack further analysis of provenance data access characteristics, making them inefficient for subsequent application analysis of provenance data.

[0003] Research on traceability management systems for specific fields includes: Shiqing Ma et al. proposed a lightweight traceability tracking system, ProTracer, which uses lightweight kernel modules and complex concurrent user space daemons for event processing, filtering redundant or irrelevant events in traceability analysis online, reducing space consumption without affecting the effectiveness of the analysis. Kyu Hyung Lee et al. proposed a new selective fine-grained recording method, BEEP, which implements a high-precision attack traceability solution and avoids the dependency explosion problem in conventional audit logs. However, traceability management systems for specific fields are mostly tailored to different application requirements, with each application corresponding to a specific management method. If multiple traceability applications are run on a server, and each application uses its own traceability management method, repeated traceability collection will result in a significant waste of system resources and reduce the access speed of traceability data. Summary of the Invention

[0004] In response to the defects of related technologies, the purpose of the present invention is to provide a traceability data management method and system based on clustering and online prediction, aiming to solve the problems of system resource waste and reduced traceability data access speed caused by each application managing traceability data independently.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a traceability data management method based on clustering and online prediction, comprising:

[0006] Traverse the traceability nodes in the traceability graph, calculate the sum of the initial importance of the traceability node and the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes, and use this as the node importance of the traceability node. Then, sort the traceability nodes according to the node importance.

[0007] At least two nodes whose node importance is greater than a threshold are set as central nodes, an initial label is assigned to the central node, and the initial label is propagated to other nodes according to the dependency relationship and the order of node importance. The traceability nodes with the same label are regarded as a traceability class;

[0008] Obtaining access records of each of the traceability classes and calculating access correlations between the traceability classes;

[0009] Predicting the access probability of the traceability class at the target time online according to the access correlation, and pre-fetching the traceability class with an access probability greater than an access threshold as hot data to the solid-state drive;

[0010] The traceability classes that have not been accessed within the preset time interval are compressed and stored in a mechanical hard disk as cold data.

[0011] Optionally, traversing the traceability nodes in the traceability graph, calculating the sum of the initial importance of the traceability node and the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes as the node importance of the traceability node, and sorting the traceability nodes according to the node importance includes:

[0012] Traversing the traceability nodes in the traceability graph, and calculating the sum of the initial importance of the traceability node and the importance of each of its child nodes according to the dependency relationship between the traceability nodes and the importance calculation formula, as the node importance of the traceability node. The node importance calculation process is implemented recursively.

[0013] Calculate the importance of all traceability nodes in the traceability graph, and sort the traceability nodes in descending order of the importance of the nodes;

[0014] The calculation formula of the importance W is:

[0015]

[0016] Among them, W0 represents the initial importance of node p, which defaults to 1; M p Represents the set of child nodes of node p.

[0017] Optionally, the step of setting at least two nodes whose node importance is greater than a threshold as central nodes, assigning an initial label to the central node, and then propagating the initial label to other nodes according to the order of the dependency relationship and node importance, and treating the traceability nodes with the same label as a traceability class includes:

[0018] The node importance is greater than the threshold T W At least two nodes are set as central nodes, and each central node is assigned an initial label, where the initial label is the pnode number of the node;

[0019] Traverse all traceability nodes according to the order of node importance and the dependency relationship. If the current traceability node has a label, assign the label of the current traceability node to the traceability nodes directly connected to it and without a label. If the current node has no label, skip it. Stop traversal until the total number of traceability nodes without a label is less than a preset value.

[0020] The traceability nodes with the same label are defined as the same traceability class and stored in the same folder.

[0021] Optionally, obtaining access records of each traceability class and calculating access correlations between the traceability classes includes:

[0022] The access correlation between the traceability classes is calculated based on the access records of each traceability class. The formula for the access correlation is:

[0023]

[0024] Among them, f(p i ) represents the traceability class p i The number of occurrences, f(ab) represents the number of consecutive occurrences of the traceability class a and the traceability class b, and α and β represent normalization constants.

[0025] Optionally, the online prediction of the access probability of the traceability class according to the access correlation, and pre-fetching the traceability class with an access probability greater than an access threshold as hot data to the solid-state drive, includes:

[0026] Online prediction of the access probability of the traceability class being associated with the access at the target time based on the access correlation, wherein the access probability is positively correlated with the access correlation;

[0027] The traceability class with an access probability greater than an access threshold is regarded as the hot data; and the hot data is pre-fetched to a solid state drive.

[0028] Optionally, compressing and storing the traceability classes that have not been accessed within a preset time interval as cold data on a mechanical hard disk includes:

[0029] Count the number of visits to the traceability category within a preset time interval;

[0030] If the number of visits to the traceability class is less than the visit threshold T F , the corresponding traceability class is determined as a traceability class with a low access probability, and the traceability class with a low access probability is compressed and stored in a mechanical hard disk as the cold data.

[0031] Optionally, the compressed storage method adopts a Bzip compression method.

[0032] Optionally, the traceability data includes 9 data sets from PASS and SPADE.

[0033] In a second aspect, the present invention further provides a traceability data management system based on clustering and online prediction, comprising:

[0034] A clustering module is used to traverse the traceability nodes in the traceability graph, calculate the initial importance of the traceability node and the sum of the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes, and use it as the node importance of the traceability node; set at least two nodes whose node importance is greater than the threshold as the central node, assign an initial label to the central node, and then propagate the initial label to other nodes according to the dependency relationship, and regard the traceability nodes with the same label as a traceability class;

[0035] A prediction module is used to obtain the access correlation between the traceability classes and to predict online the access probability of the traceability class at the target moment;

[0036] The data management module is used to divide the traceability class into hot data and cold data according to the prediction results, adjust the storage layout of the traceability class, pre-fetch the hot data to the solid-state disk, and compress and store the cold data on the mechanical hard disk.

[0037] In general, the above technical solutions conceived by the present invention have the following technical features and beneficial effects compared with the prior art:

[0038] (1) The traceability node importance calculation method provided by the present invention can accurately determine the central node and other nodes in the traceability graph, providing a basis for establishing the initial labels of subsequent clustering.

[0039] (2) The traceability label propagation algorithm provided by the present invention can store closely connected and highly correlated traceability nodes in the same database as much as possible, reducing the query time of traceability data;

[0040] (3) The traceability access correlation calculation method provided by the present invention can sense the access requirements of traceability data online, adjust the storage layout of traceability data in real time, and further speed up the access speed of traceability data;

[0041] (4) The present invention compresses the traceability data that is not frequently accessed, greatly reducing the storage overhead of the traceability data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a process for tracing data management based on clustering and online prediction provided in the first embodiment of the present invention;

[0043] Figure 2A schematic diagram of a local traceability graph provided in Example 1 of the present invention;

[0044] Figure 3 Schematic diagram of the execution process of the traceability label propagation algorithm provided in Example 1 of the present invention;

[0045] Figure 4 A schematic structural diagram of a traceability data management system based on clustering and online prediction provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0047] The contents involved in the above embodiment are described below in conjunction with a preferred embodiment.

[0048] like Figure 1 As shown, a traceability data management method based on clustering and online prediction includes:

[0049] S1. Traverse the traceability nodes in the traceability graph, calculate the initial importance of the traceability node and the sum of the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes, and use this as the node importance of the traceability node. Sort the traceability nodes by node importance.

[0050] S2. Set at least two nodes whose node importance is greater than the threshold as central nodes, assign an initial label to the central node, and then propagate the initial label to other nodes according to the order of dependency and node importance. Treat the traceability nodes with the same label as a traceability class.

[0051] S3. Obtain access records for each traceability class and calculate access correlation between traceability classes.

[0052] S4. Predict the access probability of the traceability class at the target time online based on the access correlation, and pre-fetch the traceability class with an access probability greater than the access threshold as hot data to the solid-state drive.

[0053] S5. The traceability classes that have not been accessed within the preset time interval are compressed and stored in a mechanical hard disk as cold data.

[0054] During the traceability clustering process, the traceability data includes 9 data sets from PASS and SPADE. The traceability nodes in the traceability graph are traversed, and the dependency relationship between the traceability nodes is determined based on the local characteristics of the traceability graph. The importance of each traceability node is calculated separately. The initial importance of the traceability node and the importance of each child node of the traceability node are summed as the node importance of the traceability node. The traceability nodes are sorted from small to large according to the node importance. The central node with an importance higher than the threshold is given an initial label. The traceability label propagation algorithm is used to propagate the central node label to other nodes according to the order of dependency and node importance. The traceability data with the same label are classified into the same traceability class. Based on the historical access records of the traceability class, the access correlation between the traceability classes is calculated, and the traceability classes that may be accessed in the future are predicted. The storage layout of the traceability data to be accessed is adjusted to speed up the access speed of the traceability data. The traceability classes that have not been accessed for more than a preset time are compressed and stored to reduce the storage space required for the traceability data.

[0055] Optionally, S1 includes:

[0056] S11. Traverse the traceability nodes in the traceability graph, and calculate the initial importance of the traceability node and the sum of the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes and the importance calculation formula, as the node importance of the traceability node. The node importance calculation process is implemented recursively.

[0057] S12. Calculate the importance of all traceability nodes in the traceability graph, and sort the traceability nodes in descending order of importance. The calculation formula for importance W is:

[0058]

[0059] Among them, W0 represents the initial importance of node p, which defaults to 1; M p Represents the set of child nodes of node p.

[0060] like Figure 2 As shown, let P represent a traceability node in the traceability graph ( G(P) is the set of all nodes), L P is the label of the traceability node P, which is used to indicate the traceability class to which the traceability node P belongs. The importance W of all nodes is calculated using the node importance calculation formula.

[0061] Optionally, S2 includes:

[0062] S21. Set threshold T W , the node importance is greater than the threshold T WAt least two nodes are set as central nodes, and an initial label is assigned to each central node, which is the pnode number of the node.

[0063] S22. Traverse all traceability nodes according to the order of node importance and dependency. If the current traceability node has a label, assign the label of the current traceability node to the traceability node that is directly connected to it and has no label. If the current node has no label, skip it. Stop traversal until the total number of traceability nodes that have not obtained a label is less than the preset value.

[0064] S23: define the traceability nodes with the same label as the same traceability class and store them in the same folder.

[0065] like Figure 3 As shown, the traceability nodes are arranged in descending order according to their importance W; the threshold T is set W , the importance W of each traceability node is compared with the threshold T W Compare and compare the importance W in the traceability graph to be greater than the threshold T W The node is given an initial label, which is its pnode number. For example, if the node importance of multiple traceability nodes is greater than the threshold T W , then the corresponding nodes are assigned initial labels, the initial label of each traceability node. All traceability nodes are traversed in order of importance and dependency. When traversing each traceability node, if the current node does not have a label, it is skipped; if the current traceability node has a label, the traceability node that is directly connected to the current traceability node and does not have a label is directly assigned the label of the current traceability node. The cycle continues until the number of traceability nodes without labels is less than the preset value.

[0066] Specific combination Figure 3 As shown in the diagram of the traceability label propagation algorithm, it is assumed that the importance of the nodes in the local traceability graph is as follows from high to low: node 1, node 2, node 3, node 4, node 5. Since node 1 has the highest importance and is greater than the threshold T W , get the initial label; when traversing node 1, look for nodes that have a relationship with node 1 and no label. Since node 2 and node 5 have an associated relationship with node 1, they get the label during the first propagation process; when traversing node 2, node 2 has no other connected nodes; traverse to nodes 3 and 4 according to the node importance. At this time, nodes 3 and 4 have no labels and are skipped; when traversing node 5, look for nodes that have a relationship with node 5 and no label. Node 3 gets the label during the second propagation process; determine whether the number of traceability nodes that have not obtained a label is less than the preset value. If the condition for stopping traversal is not met, traverse node 3 again to find nodes that have a relationship with node 3 and no label. Node 4 gets the label during the third propagation process.

[0067] After traversing all traceability nodes, traceability nodes with the same label are defined as the same traceability class and stored in the same folder, greatly reducing the storage space required for traceability data.

[0068] Optionally, S3 includes:

[0069] The access correlation between traceability classes is calculated based on the access records of each traceability class. The formula for access correlation is:

[0070]

[0071] Among them, f(p i ) represents the traceability class p i The number of occurrences, f(ab) represents the number of consecutive occurrences of traceability class a and traceability class b, and α and β represent normalization constants.

[0072] Get the historical access records of the traceability class from time t1 to time t2, and get the traceability class set S1={C i}, get the traceability class set S2 with the top 20% of the number of visits = {C j}, for the traceability class C that appears in S2 j , calculate C j with C i The access correlation R(C j →C i ).

[0073] Optionally, S4 includes:

[0074] S41 , predicting online the access probability of the traceability class being associated with the access at the target time according to the access correlation, where the access probability is positively correlated with the access correlation.

[0075] S42: The traceability class with an access probability greater than an access threshold is regarded as hot data; and the hot data is pre-fetched to the solid state drive.

[0076] For example, the access correlation degree of the currently accessed traceability class a is greater than the threshold T R The traceability class set R(a)={a1,a2,…a n}, sort the access associations between the traceability classes in the set R(a) and the traceability class a, and pre-fetch the traceability class ranked first into the solid-state drive.

[0077] Optionally, the S5 includes:

[0078] S51. Count the number of visits to the traceability category within a preset time interval.

[0079] S52: If the number of accesses to the traceability class is less than the access threshold T F, the corresponding traceability class is determined as a traceability class with low access probability, and the traceability class with low access probability is compressed and stored in a mechanical hard disk as cold data.

[0080] Specifically, the Bzip compression method is used to compress and store the traceability classes that have not been accessed for a long time to save storage space.

[0081] The technical solution of the present invention traverses the traceability nodes in the traceability graph, clusters the traceability nodes based on the dependencies between them using a traceability label propagation algorithm, monitors the access records of traceability classes, calculates the node importance of each traceability node, accurately predicts data access trends, and adjusts the storage layout of traceability data based on the predicted access probability, speeding up traceability data queries. It also compresses and stores traceability classes with fewer accesses, reducing the required storage space. This solves the problem of wasting system resources and slowing down traceability data access speed caused by each application managing its own traceability data, achieving the beneficial effects of speeding up traceability data access, reducing traceability data query time, and lowering traceability data storage overhead.

[0082] Example 2

[0083] Figure 4 A schematic structural diagram of a traceability data management system based on clustering and online prediction provided in the second embodiment of the present invention.

[0084] like Figure 4 As shown, a traceability data management system based on clustering and online prediction includes:

[0085] A clustering module is used to traverse the traceability nodes in the traceability graph, calculate the initial importance of the traceability node and the sum of the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes, and use it as the node importance of the traceability node; set at least two nodes whose node importance is greater than the threshold as the central node, assign an initial label to the central node, and then propagate the initial label to other nodes according to the dependency relationship, and regard the traceability nodes with the same label as a traceability class;

[0086] A prediction module is used to obtain the access correlation between the traceability classes and to predict online the access probability of the traceability class at the target moment;

[0087] The data management module is used to divide the traceability class into hot data and cold data according to the prediction results, adjust the storage layout of the traceability class, pre-fetch the hot data to the solid-state disk, and compress and store the cold data on the mechanical hard disk.

[0088] A traceability data management system based on clustering and online prediction provided by an embodiment of the present invention can execute a traceability data management method based on clustering and online prediction provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0089] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A traceability data management method based on clustering and online prediction, characterized in that: include: Traverse the traceability nodes in the traceability graph, calculate the sum of the initial importance of the traceability node and the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes, and use this as the node importance of the traceability node. Then, sort the traceability nodes according to the node importance. At least two nodes whose node importance is greater than a threshold are set as central nodes, an initial label is assigned to the central node, and the initial label is propagated to other nodes according to the dependency relationship and the order of node importance. The traceability nodes with the same label are regarded as a traceability class; The access correlation between the traceability classes is calculated based on the access records of each traceability class. The formula for the access correlation is: ; in, Represents the traceability class The number of occurrences, represents the number of consecutive occurrences of the traceability class a and the traceability class b, and α and β represent normalization constants; Predicting the access probability of the traceability class at the target time online according to the access correlation, and pre-fetching the traceability class with an access probability greater than an access threshold as hot data to the solid-state drive; The traceability classes that have not been accessed within the preset time interval are compressed and stored in a mechanical hard disk as cold data.

2. The method according to claim 1, wherein The traversing of the traceability nodes in the traceability graph, calculating the sum of the initial importance of the traceability node and the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes as the node importance of the traceability node, and sorting the traceability nodes according to the node importance, including: Traverse the traceability nodes in the traceability graph, and calculate the sum of the initial importance of the traceability node and the importance of each of its child nodes based on the dependency relationship between the traceability nodes and the importance calculation formula. This is used as the node importance of the traceability node. The node importance calculation process is implemented recursively. Calculate the importance of all traceability nodes in the traceability graph, and sort the traceability nodes in descending order of the importance of the nodes; The calculation formula of the importance W is: ； in, Representation node The initial importance of is 1 by default; Representation node The collection of child nodes.

3. The method according to claim 1, wherein The method sets at least two nodes whose node importance is greater than a threshold as central nodes, assigns an initial label to the central node, propagates the initial label to other nodes according to the dependency relationship and the order of node importance, and regards the traceability nodes with the same label as a traceability class, including: The node importance is greater than the threshold T W At least two nodes are set as central nodes, and each central node is assigned an initial label, where the initial label is the pnode number of the node; Traverse all traceability nodes according to the order of node importance and the dependency relationship. If the current traceability node has a label, assign the label of the current traceability node to the traceability nodes directly connected to it and without a label. If the current node has no label, skip it. Stop traversal until the total number of traceability nodes without a label is less than a preset value. The traceability nodes with the same label are defined as the same traceability class and stored in the same folder.

4. The method according to claim 1, wherein The online prediction of the access probability of the traceability class according to the access correlation, and pre-fetching the traceability class with an access probability greater than an access threshold as hot data to the solid-state hard disk, includes: Online prediction of the access probability of the traceability class being associated with the access at the target time based on the access correlation, wherein the access probability is positively correlated with the access correlation; The traceability class with an access probability greater than an access threshold is regarded as the hot data; and the hot data is pre-fetched to a solid state drive.

5. The method according to claim 1, wherein The method of compressing and storing the traceability classes that have not been accessed within a preset time interval as cold data on a mechanical hard disk includes: Count the number of visits to the traceability category within a preset time interval; If the number of visits to the traceability class is less than the visit threshold T F , the corresponding traceability class is determined as a traceability class with a low access probability, and the traceability class with a low access probability is compressed and stored in a mechanical hard disk as the cold data.

6. The method according to claim 5, wherein The compression storage method adopts the Bzip compression method.

7. The method according to claim 1, wherein The traceability data include 9 data sets from PASS and SPADE.

8. A traceability data management system based on clustering and online prediction, characterized in that: include: A clustering module is used to traverse the traceability nodes in the traceability graph and calculate the sum of the initial importance of the traceability node and the importance of each child node of the traceability node according to the dependency relationship between the traceability nodes as the node importance of the traceability node; At least two nodes whose node importance is greater than a threshold are set as central nodes, an initial label is assigned to the central node, and the initial label is propagated to other nodes according to the dependency relationship, and the traceability nodes with the same label are regarded as a traceability class; The prediction module is used to calculate the access correlation between the traceability classes based on the access records of each traceability class. The formula for the access correlation is: ； in, Indicates the number of times the traceability class appears. represents the number of consecutive occurrences of the traceability class a and the traceability class b, α and β represent normalization constants, and the probability of the traceability class being accessed at the target time is predicted online; The data management module is used to divide the traceability class into hot data and cold data according to the prediction results, adjust the storage layout of the traceability class, pre-fetch the hot data to the solid-state disk, and compress and store the cold data on the mechanical hard disk.

Images