A big data difference detection method based on summary tree

Abstract

The invention relates to a big data difference detection method based on a summary tree, and in particular to the field of big data analysis. A big data set is divided according to a time series and a geographical location, a hash function is applied to each big data block to generate a unique hash value, which is used as a unique identifier of the big data block and represented as a big data block summary, a summary tree is constructed from bottom to top using a balanced binary tree algorithm using the generated big data block summary as a node, the nodes of the summary tree are compared layer by layer in a depth-first manner, the summary tree structure is judged and the summary information of two summary tree nodes is compared, the difference reasons between the summary trees of two big data sets are analyzed, specific difference data is fed back, and a visual interface is created to display the location and data content of the difference point, a Z-score algorithm is used to realize the conversion of the storage data set into a standard normal distribution with a zero mean and a unit standard deviation, and a three-dimensional difference data storage table is created in a selected database management system to realize data storage management.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis, and more specifically, to a big data difference detection method based on a summary tree. Background Art

[0002] Summary tree is a data structure used for summarization and difference detection of large data sets. In the era of big data, the amount of data is growing explosively. Enterprises and individuals need to process large amounts of data to extract valuable information and insights. Data difference detection has become an important task.

[0003] Traditional difference detection methods usually use row-by-row comparison and hash comparison. When the data scale is large, the algorithm complexity is high, resulting in slow detection speed, difficulty in accurately locating the differences between data sets, and failure to provide detailed difference information, making it difficult for users to understand the specific differences between data sets. The lack of intuitive visual display reduces user experience and operational efficiency, and is relatively limited for complex tree-structured data. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a big data difference detection method based on summary tree to solve the problems raised in the above background technology.

[0005] The present invention solves the above technical problems with the following technical solution: A big data difference detection method based on summary tree comprises the following steps:

[0006] S101. Divide the large data set according to time series and geographical location, apply a hash function to each large data block to generate a unique hash value, which is used as a unique identifier of the large data block and represented as a large data block summary, and use the generated large data block summary as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm;

[0007] S102. Obtain two summary trees to be compared and compare the nodes of the summary trees layer by layer using a depth-first approach, determine the summary tree structure and compare the summary information of the two summary tree nodes, and recursively compare until the complete summary tree is traversed;

[0008] S103. Output the difference nodes, analyze the differences between the summary trees of the two large data sets, feedback the specific difference data, and create a visual interface to display the location and data content of the difference points;

[0009] S104. Arrange the large data blocks, node summary information and difference reasons into a storage data set, use the Z-score algorithm to convert the storage data set into a standard normal distribution with zero mean and unit standard deviation, and create a three-dimensional difference data storage table in the selected database management system to implement data storage management;

[0010] In a preferred embodiment, in S101, the large data set is divided according to time series and geographic location, and a large data block structure is designed, wherein each large data block contains metadata information, including time series range, geographic location range, data block size and data record information, an index is constructed based on the time series and geographic location, and creation, update and deletion operations of large data blocks are added, a hash function library is imported, and a hash function is applied to each large data block to generate a unique hash value, which is used as a unique identifier of the large data block and represented as a large data block summary, and the generated large data block summary is used as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm, and the specific steps are: inserting the node to the bottom position of the balanced binary tree, starting from the initial node, and updating the height of the ancestor node upward along the branch path, and checking the balance factor of each ancestor node, that is, the height difference between its left subtree and right subtree, and the specific formula of the balance factor is:

[0011] BalanceFactor=height(leftSubtree)-height(rightSubtree)

[0012] Among them, BalanceFactor represents the balance factor, height(leftSubtree) represents the height of the left subtree, and height(rightSubtree) represents the height of the right subtree. When the balance factor of the node exceeds 1, it means that the left subtree is too deep. When the balance factor of the node is less than -1, it means that the right subtree is too deep. The corresponding rotation operation is performed to rebalance the connection relationship between the nodes. The division, generation, and construction steps are repeated until the large data set is processed.

[0013] In a preferred embodiment, in S102, two summary trees to be compared are obtained and the nodes of the summary trees are compared layer by layer using a depth-first method. The specific steps of the depth-first method are: taking the root node of the summary tree as the starting node and marking it as visited, and checking the adjacent nodes of the current node, including child nodes and connected nodes, selecting one of the unvisited neighbor nodes as the next visited node, recursively calling the selected neighbor node and using the neighbor node as the new current node, and when the current node has no unvisited neighbor node, backtracking to the previous level node, repeating the checking, selecting, recursing and backtracking Steps, until all nodes are traversed, it is determined that the summary tree is a single-branch tree, and the summary information of the root nodes of the two summary trees is compared. When the summary information of the root nodes is different, there is a difference between the two data sets, and the comparison ends. When the summary information of the root nodes is the same, the summary information of the child nodes is compared again until the summary information of the child nodes is different. In this case, there is a difference between the two data sets, and the comparison ends. It is determined that the summary tree is a multi-branch tree, and each child node is recursively compared. For each child node, the summary information is compared layer by layer repeatedly. When the summary information of a certain layer is different, it is recorded according to the difference position and content, and the comparison is performed recursively until the complete summary tree is traversed.

[0014] In a preferred embodiment, in S103, the difference nodes existing in the summary trees of the two large data sets are determined, and the difference positions and contents of the records are connected, wherein the position information includes the index and path of the node, wherein the content includes the node size and the information stored in the node, the difference nodes are output, and the difference reasons of the summary trees of the two large data sets are analyzed. When the summary information of the summary tree nodes of the first large data set is more than that of the summary tree nodes of the second large data set, it is judged that the difference reason is an insertion or update operation. When the summary information of the summary tree nodes of the first large data set is less than that of the summary tree nodes of the second large data set, it is judged that the difference reason is a deletion operation, the difference reason is exported and the specific difference data is fed back, a visualization interface is created, and the summary tree structure of the large data set is displayed using a tree diagram, wherein blue is used to mark new nodes, red is used to mark deleted nodes, and yellow is used to mark modified nodes. Transparent overlap is used to compare the summary tree structures of the two large data sets, different parts are marked with circles, a relationship diagram between the summary tree nodes is drawn and the reasons and impact ranges of the differences are marked, and a user-defined node display and interactive operation are implemented using a Plotly interactive visualization tool.

[0015] In a preferred embodiment, in S104, the large data blocks, node summary information and difference reasons are sorted into storage data sets, and the storage data sets with naming rules and unit definition differences are standardized. The standardization scales the storage data sets in proportion to eliminate the dimension and unit differences between the storage data sets, and uses the Z-score algorithm to convert the storage data sets into a standard normal distribution with zero mean and unit standard deviation. The specific formula is:

[0016]

[0017] Where Z represents the standardized storage data set, x represents the storage data set, μ represents the mean of the storage data set, and σ represents the standard deviation of the storage data set. The storage data set is selectively stored in the cloud and the local database. The cloud storage has redundancy and backup mechanisms. The local database allows full control of the storage data set, including storage structure, backup strategy and data migration. The database table structure is designed, in which each table corresponds to a summary tree structure. One table is used to store large data blocks, and the other table stores node summary information and difference reasons. A three-dimensional difference data storage table is created in the selected database management system to realize data storage management.

[0018] The beneficial effects of the present invention are as follows: by dividing a large data set according to time series and geographical location, the required data blocks can be quickly located, the query scope can be reduced and the query efficiency can be improved; a unique summary generated by a hash function is used as an identifier to avoid repeated storage of the same metadata information; by adding creation, update and deletion operations of a large data set, the integrity and consistency of the large data set can be maintained; by using a balanced binary tree algorithm to construct a summary tree, summary information can be quickly accessed and retrieved, and the data search efficiency can be improved; by monitoring the balance factor of the node and performing a rotation operation to maintain the balance of the balanced binary tree, the time complexity of traversing the entire summary tree can be avoided by comparing nodes layer by layer in a depth-first manner, and the comparison efficiency can be improved, and the comprehensiveness and accuracy of the comparison can be ensured; by connecting and recording the location information and content of the difference nodes, the difference points between the summary trees of two large data sets can be accurately identified; for different summary information and difference reasons, the user can be helped to understand the reasons for the changes between the two data sets and quickly understand the differences; by creating a visual interface, the user can intuitively understand the differences between the data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention; DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0021] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0022] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0023] This embodiment provides Figure 1 A big data difference detection method based on summary tree is shown, which specifically includes the following steps:

[0024] S101. Divide the large data set according to time series and geographical location, apply a hash function to each large data block to generate a unique hash value, which is used as a unique identifier of the large data block and represented as a large data block summary, and use the generated large data block summary as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm;

[0025] Furthermore, the large data set is divided according to time series and geographic location, and a large data block structure is designed, wherein each large data block contains metadata information, including time series range, geographic location range, data block size and data record information. An index is constructed based on the time series and geographic location, and creation, update and deletion operations of large data blocks are added. A hash function library is imported, and a hash function is applied to each large data block to generate a unique hash value, which is used as a unique identifier of the large data block and represented as a large data block summary. The generated large data block summary is used as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm. The specific steps are: inserting the node to the bottom position of the balanced binary tree, starting from the initial node, and updating the height of the ancestor node upward along the branch path. For each ancestor node, its balance factor is checked, that is, the height difference between its left subtree and its right subtree. The specific formula of the balance factor is:

[0026] BalanceFactor=height(leftSubtree)-height(rightSubtree)

[0027] Among them, BalanceFactor represents the balance factor, height(leftSubtree) represents the height of the left subtree, and height(rightSubtree) represents the height of the right subtree. When the balance factor of the node exceeds 1, it means that the left subtree is too deep. When the balance factor of the node is less than -1, it means that the right subtree is too deep. The corresponding rotation operation is performed to rebalance the connection relationship between the nodes. The division, generation, and construction steps are repeated until the large data set is processed.

[0028] S102. Obtain two summary trees to be compared and compare the nodes of the summary trees layer by layer using a depth-first approach, determine the summary tree structure and compare the summary information of the two summary tree nodes, and recursively compare until the complete summary tree is traversed;

[0029] Furthermore, two summary trees to be compared are obtained and the nodes of the summary trees are compared layer by layer using a depth-first method. The specific steps of the depth-first method are: taking the root node of the summary tree as the starting node and marking it as visited, and checking the adjacent nodes of the current node, including child nodes and connected nodes, selecting one of the unvisited neighbor nodes as the next visited node, recursively calling the selected neighbor node and using the neighbor node as the new current node, and when the current node has no unvisited neighbor nodes, backtracking to the previous level node, repeating the steps of checking, selecting, recursing and backtracking until the traversal is complete. After all nodes are searched, the summary tree is judged to be a single-branch tree, and the summary information of the root nodes of the two summary trees is compared. When the summary information of the root nodes is different, there are differences between the two data sets and the comparison ends. When the summary information of the root nodes is the same, the summary information of the child nodes is compared until the summary information of the child nodes is different. In this case, there are differences between the two data sets and the comparison ends. The summary tree is judged to be a multi-branch tree, and each child node is recursively compared. For each child node, the summary information is compared layer by layer. When the summary information of a certain layer is different, it is recorded according to the difference position and content, and recursively compared until the complete summary tree is traversed.

[0030] S103. Output the difference nodes, analyze the differences between the summary trees of the two large data sets, feedback the specific difference data, and create a visual interface to display the location and data content of the difference points;

[0031] Furthermore, the difference nodes in the summary trees of the two large data sets are determined, and the difference positions and contents of the records are connected, wherein the position information includes the index and path of the node, wherein the content includes the node size and the information stored in the node, the difference nodes are output, and the reasons for the differences in the summary trees of the two large data sets are analyzed. When the summary information of the summary tree nodes of the first large data set is more than that of the summary tree nodes of the second large data set, it is judged that the difference reason is an insertion or update operation, and when the summary information of the summary tree nodes of the first large data set is less than that of the summary tree nodes of the second large data set, it is judged that the difference reason is a deletion operation, the difference reason is exported and the specific difference data is fed back, a visualization interface is created, and a tree diagram is used to display the summary tree structure of the large data set, wherein blue is used to mark new nodes, red is used to mark deleted nodes, and yellow is used to mark modified nodes, transparent overlay is used to compare the summary tree structures of the two large data sets, different parts are marked with circles, a relationship diagram between the summary tree nodes is drawn and the reasons and impact range of the differences are marked, and the P l otly interactive visualization tool is used to implement user-defined node display and interactive operations.

[0032] S104. Arrange the large data blocks, node summary information and difference reasons into a storage data set, use the Z-score algorithm to convert the storage data set into a standard normal distribution with zero mean and unit standard deviation, and create a three-dimensional difference data storage table in the selected database management system to implement data storage management;

[0033] Furthermore, the large data blocks, node summary information and difference reasons are sorted into storage data sets, and the storage data sets with naming rules and unit definition differences are standardized. The standardization scales the storage data sets in proportion to eliminate the dimension and unit differences between the storage data sets, and uses the Z-score algorithm to convert the storage data sets into a standard normal distribution with zero mean and unit standard deviation. The specific formula is:

[0034]

[0035] Where Z represents the standardized storage data set, x represents the storage data set, μ represents the mean of the storage data set, and σ represents the standard deviation of the storage data set. The storage data set is selectively stored in the cloud and the local database. The cloud storage has redundancy and backup mechanisms. The local database allows full control of the storage data set, including storage structure, backup strategy and data migration. The database table structure is designed, in which each table corresponds to a summary tree structure. One table is used to store large data blocks, and the other table stores node summary information and difference reasons. A three-dimensional difference data storage table is created in the selected database management system to realize data storage management.

[0036] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0037] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0038] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0039] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0040] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0041] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0042] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A big data difference detection method based on summary tree, characterized in that: The specific steps include: S101. Divide the large data set according to time series and geographical location, apply a hash function to each large data block to generate a unique hash value, which is used as a unique identifier of the large data block and represented as a large data block summary, and use the generated large data block summary as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm; S102. Obtain two summary trees to be compared and compare the nodes of the summary trees layer by layer using a depth-first approach, determine the summary tree structure and compare the summary information of the two summary tree nodes, and recursively compare until the complete summary tree is traversed; S103. Output the difference nodes, analyze the differences between the summary trees of the two large data sets, feedback the specific difference data, and create a visual interface to display the location and data content of the difference points; S104. Arrange the large data blocks, node summary information and difference reasons into a storage data set, use the Z-score algorithm to convert the storage data set into a standard normal distribution with zero mean and unit standard deviation, and create a three-dimensional difference data storage table in the selected database management system to implement data storage management.

2. The big data difference detection method based on summary tree according to claim 1 is characterized in that: In S101, the large data set is divided according to time series and geographic location, and a large data block structure is designed. An index is built based on the time series and geographic location, and creation, update, and deletion operations of the large data block are added. A hash function library is imported, and a hash function is applied to each large data block to generate a unique hash value, which serves as a unique identifier of the large data block and is represented as a large data block summary. The generated large data block summary is used as a node to construct a summary tree from bottom to top using a balanced binary tree algorithm.

3. The big data difference detection method based on summary tree according to claim 2 is characterized in that: The specific steps of the balanced binary tree algorithm are: inserting a node to the bottom of the balanced binary tree, starting from the initial node, updating the height of the ancestor node upward along the branch path, and checking the balance factor of each ancestor node, that is, the height difference between its left subtree and right subtree.

4. The big data difference detection method based on summary tree according to claim 3 is characterized in that: The specific formula of the balance factor is: BalanceFactor=height(leftSubtree)-height(rightSubtree) Among them, BalanceFactor represents the balance factor, height(leftSubtree) represents the height of the left subtree, and height(rightSubtree) represents the height of the right subtree.

5. The big data difference detection method based on summary tree according to claim 1 is characterized in that: In S102, two summary trees to be compared are obtained and the nodes of the summary trees are compared layer by layer in a depth-first manner, the summary tree structure is determined and the summary information of the two summary tree nodes is compared, and the comparison is performed recursively until the complete summary tree is traversed.

6. The big data difference detection method based on summary tree according to claim 5 is characterized in that: The specific steps of the depth-first method are: taking the root node of the summary tree as the starting node and marking it as visited, and checking the adjacent nodes of the current node, including child nodes and connected nodes, selecting one of the unvisited neighbor nodes as the next node to be visited, recursively calling the selected neighbor node and using the neighbor node as the new current node, and when the current node has no unvisited neighbor nodes, backtracking to the previous level node, repeating the checking, selecting, recursive and backtracking steps until all nodes are traversed.

7. The big data difference detection method based on summary tree according to claim 5 is characterized in that: The specific steps of determining the summary tree structure and comparing the summary information of two summary tree nodes are as follows: determining that the summary tree is a single-branch tree, comparing the summary information of the root nodes of the two summary trees, when the summary information of the root nodes is different, there is a difference between the two data sets and the comparison ends, when the summary information of the root nodes is the same, continuing to compare the summary information of the child nodes until the summary information of the child nodes is different, there is a difference between the two data sets and the comparison ends, determining that the summary tree is a multi-branch tree, recursively comparing each child node, repeatedly comparing the summary information of each child node layer by layer, when different summary information is found in a certain layer, recording the difference according to the position and content, and recursively comparing until the complete summary tree is traversed.

8. The big data difference detection method based on summary tree according to claim 1 is characterized by: In the S103, the difference nodes existing in the summary trees of the two large data sets are determined, and the difference positions and contents of the records are connected, wherein the position information includes the index and path of the node, wherein the content includes the node size and the information stored in the node, the difference nodes are output, and the reasons for the differences in the summary trees of the two large data sets are analyzed. When the summary information of the summary tree nodes of the first large data set is more than that of the summary tree nodes of the second large data set, it is determined that the difference reason is an insert or update operation; when the summary information of the summary tree nodes of the first large data set is less than that of the summary tree nodes of the second large data set, it is determined that the difference reason is a delete operation, the difference reason is exported, and the specific difference data is fed back, and a visualization interface is created.

9. The big data difference detection method based on summary tree according to claim 8 is characterized in that: In the visualization interface, a tree diagram is used to display the summary tree structure of a large data set. Blue is used to mark new nodes, red is used to mark deleted nodes, and yellow is used to mark modified nodes. Transparent overlays are used to compare the summary tree structures of two large data sets, and circles are used to mark different parts. A relationship diagram between summary tree nodes is drawn and the causes and impact range of the differences are marked. The Plotly interactive visualization tool is used to implement user-defined node display and interactive operations.

10. The big data difference detection method based on summary tree according to claim 1, characterized in that: Arrange the big data blocks, node summary information and the reasons for the differences into storage data sets, scale the storage data sets by ratio, eliminate the dimension and unit differences between the storage data sets, and use the Z-score algorithm to convert the storage data sets into a standard normal distribution with zero mean and unit standard deviation. The specific formula is: Where Z represents the standardized storage data set, x represents the storage data set, μ represents the mean of the storage data set, and σ represents the standard deviation of the storage data set. The storage data set is selectively stored in the cloud and local database, and the database table structure is designed, in which each table corresponds to a summary tree structure. One table is used to store large data blocks, and the other table stores node summary information and difference reasons. A three-dimensional difference data storage table is created in the selected database management system to realize data storage management.

Images