Processing method, system and equipment for migrating tree structure data and medium

Abstract

The invention discloses a processing method, system and equipment for migrating tree structure data and a medium, and the method specifically comprises the following steps: constructing a father-child node relation tree of a test environment according to a father-child node relation of the tree structure data in the test environment; after the tree structure data is imported into the target environment, a sub-node mapping table is formed by establishing a mapping relation for each old sub-node in the test environment and each new sub-node in the target environment; constructing a target environment father-child node relation tree by determining a new father node corresponding to each new child node in the target environment; and according to the test environment father-child node relation tree, the child node mapping table and the target environment father-child node relation tree, determining an old father node in the test environment corresponding to each new father node in the target environment, and updating data of each new father node. According to the method, the tree structure data in the test environment can be quickly imported into the target environment, and the risk of data inconsistency does not need to be worried about.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for processing migrated tree structure data. Background Art

[0002] In the backend business system, tree-structured data is a common form of data organization, which is widely used in various functional modules, such as menu modules, role modules, product category information, and file system directory structures. These tree-structured data usually contain hierarchical nodes, each of which may have its child nodes, thus forming a complete tree structure. For example, in the menu module, the directory is the parent of the menu, and the menu is the parent of the permission, forming a clear and hierarchical structure. Similarly, in the role module that adopts the RBAC (role-based access control) mode, there is often a tree relationship between roles. Each role has a corresponding parent role, and the parent role can inherit the corresponding permissions of the child role, thereby simplifying permission management.

[0003] However, in actual function development, the processing of tree-structured data faces a difficult problem. During the development process, it is usually necessary to create the corresponding data in the test environment and conduct sufficient testing and verification to ensure the accuracy of the data and the integrity of the function. However, after the test environment is verified, how to quickly and accurately import these tree-structured data into the target environment has become a thorny problem.

[0004] For flat data (i.e., data that is not in a tree structure), you can directly import it from the test environment to the target environment by copying it. However, for tree structured data, since each node contains an ID pointing to its parent node (parent ID), and the corresponding parent ID values ​​differ between different environments, directly copying tree structured data will lead to the risk of data inconsistency. If you re-enter this data in the target environment, it will not only be time-consuming and labor-intensive, but also prone to errors, greatly reducing development efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a processing method, system, device and medium for migrating tree-structured data, which can quickly import tree-structured data in a test environment into a target environment without worrying about the risk of data inconsistency, thereby greatly improving development efficiency and system stability, so as to solve at least one of the above-mentioned prior art problems.

[0006] In a first aspect, the present invention provides a method for processing tree structure data migration, the method specifically comprising:

[0007] According to the parent-child node relationship of the tree structure data in the test environment, a parent-child node relationship tree of the test environment is constructed;

[0008] After the tree structure data is imported into the target environment, a sub-node mapping table is formed by establishing a mapping relationship between each old sub-node in the test environment and each new sub-node in the target environment;

[0009] By determining a new parent node corresponding to each new child node in the target environment, a parent-child node relationship tree of the target environment is constructed;

[0010] According to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, the old parent node in the test environment corresponding to each new parent node in the target environment is determined, and the data of each new parent node is updated.

[0011] In a second aspect, the present invention provides a processing system for migrating tree structure data, the system specifically comprising:

[0012] The first processing module is used to construct a test environment parent-child node relationship tree according to the parent-child node relationship of the tree structure data in the test environment;

[0013] A second processing module is used for, after importing the tree structure data into the target environment, establishing a mapping relationship between each old child node in the test environment and each new child node in the target environment to form a child node mapping table;

[0014] A third processing module, configured to construct a parent-child node relationship tree of the target environment by determining a new parent node corresponding to each new child node in the target environment;

[0015] The fourth processing module is used to determine the old parent node in the test environment corresponding to each new parent node in the target environment according to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, and update the data of each new parent node.

[0016] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor and a computer program stored in the memory, and when the computer program is executed on the processor, it implements a method for processing migration tree structure data as described in any one of the above methods.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for processing migration tree structure data as described in any one of the above methods is implemented.

[0018] Compared with the prior art, the present invention has at least one of the following technical effects:

[0019] 1. The present invention can quickly import tree structure data in the test environment into the target environment without worrying about the risk of data inconsistency, thereby greatly improving development efficiency and system stability.

[0020] 2. The present invention realizes the accurate migration of tree structure data between different environments by constructing the parent-child node relationship tree of the test environment and the target environment, and establishing a child node mapping table, thereby ensuring the integrity of the data and the accuracy of the hierarchical structure, and avoiding the problem of inconsistent parent IDs that may be caused by direct copying.

[0021] 3. The present invention can automatically establish a mapping relationship between old child nodes and new child nodes by constructing attribute feature vectors and calculating similarities, which greatly improves the automation of migration, reduces manual intervention, and reduces the error rate.

[0022] 4. The MerkleTree algorithm of the present invention can efficiently compare the differences between two tree structure data and generate change records. Combined with the breadth-first search algorithm, it can quickly and synchronously update the tree structure data of the target environment to ensure data consistency.

[0023] 5. The present invention can automatically identify data differences and abnormal nodes by training the verification model, thereby improving the accuracy and reliability of migration.

[0024] 6. The present invention constructs an AVL tree and converts it back into tree structure data. This method can optimize the data structure and improve system performance.

[0025] 7. The present invention trains a migration failure detection model, and the method can timely discover migration failure nodes and their causes, thereby improving the monitoring and troubleshooting capabilities of migration.

[0026] 8. The present invention adopts technical means such as partition algorithm and Map task to optimize the migration of large-scale tree-structured data, thereby improving the migration efficiency and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flowchart of a method for processing tree structure data migration provided by the first embodiment of the present invention;

[0029] Figure 2It is a flowchart of a method for processing tree structure data migration provided by the second embodiment of the present invention;

[0030] Figure 3 is a flowchart of a method for processing tree structure data migration provided by the third embodiment of the present invention;

[0031] Figure 4 is a flowchart of a method for processing tree structure data migration provided by a fourth embodiment of the present invention;

[0032] Figure 5 is a flowchart of a method for processing tree structure data migration provided by a fifth embodiment of the present invention;

[0033] Figure 6 is a flowchart of a method for processing tree structure data migration provided by a sixth embodiment of the present invention;

[0034] Figure 7 is a flowchart of a method for processing tree structure data migration provided by the seventh embodiment of the present invention;

[0035] Figure 8 It is a structural schematic diagram of a processing system for migrating tree structure data provided by an embodiment of the present invention;

[0036] Fig. 9 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0039] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0041] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0042] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0043] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A flowchart of a method for processing tree structure data migration disclosed in the first embodiment of the present invention is shown, and the details are as follows:

[0044] S101, constructing a test environment parent-child node relationship tree according to the parent-child node relationship of the tree structure data in the test environment;

[0045] S102, after importing the tree structure data into the target environment, a mapping relationship is established between each old child node in the test environment and each new child node in the target environment to form a child node mapping table;

[0046] S103, constructing a parent-child node relationship tree of the target environment by determining a new parent node corresponding to each new child node in the target environment;

[0047] S104, determining the old parent node in the test environment corresponding to each new parent node in the target environment according to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, and updating the data of each new parent node.

[0048] In this embodiment, firstly, the corresponding tree structure data oldTreeDatas is obtained from the test environment; then, oldTreeDatas is parsed to obtain a map information, where the key is the id of the data in the test environment (called oldId), and the value is the corresponding parent id in the test environment (called oldPid), that is, map[oldId]oldPid; when the id and pid in oldTreeDatas are ignored, the other information of oldTreeDatas is imported into the target environment. For each piece of data in the test environment, a new id (newId) can definitely be obtained, and at the same time, it is known that Knowing oldId, we can establish a corresponding relationship between oldId and newId, that is, we can obtain map[oldId]newId and map[newId]oldId; for each piece of data in the target environment, we know its newId, and we can obtain its oldId in the test environment through map[newId]oldId, and then obtain its parent menu id (oldPid) in the test environment according to menu[oldId]oldPid, and then obtain the id of the parent id in the target environment according to map[oldId]newId, and then update the corresponding parent id data in the target environment.

[0049] For example, in a certain system, 10 new module functions are developed, each module has 5 menu / interface data, the test environment has manually added the menu / interface data, after the function is verified, it is now necessary to migrate these menu / interface information to the target environment. According to the traditional existing technology, 50 pieces of data need to be manually re-entered in the target environment, but in this embodiment, it needs to be achieved through the following steps:

[0050] 1. First, obtain the menu / interface data of the test environment. Assume that the data of id->pid is obtained (other basic information data, such as menu name, menu creator, etc. are ignored): 1->0, 10->1, 11->1, that is, the corresponding ids in the test environment are 1, 10, 11, and their corresponding parent ids are: 0, 1, 1;

[0051] 2. Ignore id and pid, and write the basic data of the menu / interface of the test environment to the target environment. Each old data can correspond to a new data, and the id of the new data is inconsistent with the id on the test environment, for example:

[0052] The ids in the test environment are 1, 10, and 11. After inserting data in the target environment, the corresponding ids in the target environment are 2, 20, and 21. At this time, the parent id in the target environment has not been updated. After completing this step, you can get:

[0053] newId->oldId:2->1,20->10,21->11, then oldId->newId:1->2,10->20,11->21

[0054] 3. Get the id of each new data added in the target environment and update its corresponding pid. For example, when newId is 2, its corresponding oldId is 1. In the test environment, the pid of oldId 1 is 0, and the corresponding pid in the target environment is also set to 0. When newId is 20, its corresponding oldId is 10. In the test environment, the pid of oldId 10 is 1. When the corresponding pid is 1, the id of the data in the target environment is 2. Then update the pid of the data with id 20 in the target environment to 2. The node ids of the test environment and the target environment are shown in Table 1.

[0055] Table 1

[0056] oldId oldPid newId newPid 1 0 2 0 10 1 20 2 11 1 21 2

[0057] In some embodiments, Figure 2 As shown, in the above step S102, the sub-node mapping table is formed by establishing a mapping relationship between each old sub-node in the test environment and each new sub-node in the target environment, which specifically includes:

[0058] S1021, constructing an attribute feature vector for each old child node in the test environment and each new child node in the target environment respectively;

[0059] S1022, obtaining a node similarity matrix by comparing and calculating the similarity between the attribute feature vectors of each old child node in the test environment and each new child node in the target environment;

[0060] S1023, establishing a mapping relationship between old child nodes and new child nodes by comparing the similarity between each old child node and each new child node in the node similarity matrix with a preset similarity threshold, and forming a child node mapping table.

[0061] In this embodiment, an old sub-node set in a test environment and a new sub-node set in a target environment are obtained; for each old sub-node and new sub-node, attribute features of the node are extracted to construct an attribute feature vector; a cosine similarity algorithm is used to calculate the similarity between the attribute feature vector of each old sub-node and the attribute feature vector of each new sub-node to obtain a node similarity matrix; according to a preset similarity threshold, it is determined whether the similarity between each old sub-node and each new sub-node in the node similarity matrix exceeds the threshold; if the similarity between the old sub-node and the new sub-node exceeds the threshold, a mapping relationship between the old sub-node and the new sub-node is established in a sub-node mapping table; the above determination steps are repeated until all old sub-nodes and new sub-nodes in the node similarity matrix are traversed; and the constructed sub-node mapping table is output to guide the migration process from the test environment to the target environment.

[0062] Among them, the attribute feature vector is a set of numerical values ​​used to describe the characteristics of the child node. These numerical values ​​can be various attributes of the child node. For example, if the child node represents a user, the attributes can include the user's age, gender, city, registration time, number of purchases, etc. Which attributes to choose depends on the specific business scenario and needs. For example, if you want to migrate user data of an e-commerce platform, the user's purchase history and preferences are more important than height and weight. For example, in the scenario of game server migration, you can build a feature vector based on the player's level, equipment, game time and other attributes, and then migrate the player data of the test server to the official server. In this way, the player's data can be guaranteed to remain consistent after migration to avoid player losses.

[0063] In some embodiments, Figure 3 As shown, the method also includes:

[0064] S301, using the MerkleTree algorithm to compare the tree structure data of the test environment and the target environment, determine the change status of each node, and generate a change record;

[0065] S302: According to the change record, a breadth-first search algorithm is used to synchronously update the tree structure data of the target environment.

[0066] In this embodiment, the MerkleTree algorithm, also called the Merkle tree, is a hash tree structure that can be used to efficiently verify the integrity of data. The core idea is to recursively perform hash operations on data blocks and eventually generate a root hash value. Any change to a data block will affect the hash value of its parent node and ultimately the root hash value. Therefore, by comparing the root hash values ​​of two MerkleTrees, it is possible to quickly determine whether their data is consistent. In the scenario of tree structure data comparison, a MerkleTree can be constructed for the test environment and the target environment respectively.

[0067] Specifically, the tree structure data of the test environment and the target environment are obtained and converted into a MerkleTree tree structure. Each node of the MerkleTree is traversed and compared to determine whether the hash values ​​of the nodes are consistent. If they are inconsistent, it means that the node has changed. The path and change type of the changed node are recorded to generate a change record. According to the change record, the MerkleTree of the target environment is traversed using the breadth-first search algorithm to locate each changed node. For newly added nodes, new nodes are inserted at the corresponding positions of the target environment; for deleted nodes, the corresponding nodes are removed in the target environment; for modified nodes, the data of the nodes in the target environment are updated. During the synchronization update process, if the target environment is missing a parent node, the missing parent node is created according to the tree structure of the test environment. Continue to traverse the next changed node and repeat the above steps until all changed nodes have been processed. Compare the updated target environment MerkleTree with the MerkleTree of the test environment to determine whether they are completely consistent. If they are consistent, the synchronization update is successful, otherwise, an exception is handled.

[0068] For nodes newly added in the test environment, insert new nodes at the corresponding positions in the target environment. For nodes deleted in the test environment, if the node still has child nodes in the target environment, update the parent node ID of these child nodes to the parent node ID of the deleted node to maintain the integrity of the tree structure; if the node has no child nodes in the target environment, delete the node directly. For nodes modified in the test environment, update the attribute information of the corresponding nodes in the target environment. When traversing the tree structure in depth-first search, start from the root node, first traverse all child nodes of the current node, and then traverse all child nodes of each child node in turn until the leaf node. This ensures that when building the tree structure object in memory, the parent node must be accessed and created before the child node. When updating the target environment tree structure in breadth-first search, start from the root node, first update all nodes of the current level, and then update all nodes of the next level in turn until the leaf node. This ensures that when updating a node, the parent node must be updated before the child node. After data synchronization is completed, the target environment tree structure object in memory is persistently stored in the MongoDB database in JSON format for subsequent quick access and operation.

[0069] For example, in the version update of the game server, the game data can be organized into a tree structure, and then the MerkleTree algorithm is used to compare the differences between the new and old versions, and the breadth-first search algorithm is used to synchronize the differences to the player's client, thereby achieving fast updates.

[0070] In some embodiments, Figure 4 As shown, the method also includes:

[0071] S401, based on the tree structure data of the test environment, using a depth-first search algorithm to traverse the reference relationship, data integrity and data legitimacy between each node of the tree structure data of the target environment to obtain a verification result;

[0072] S402, determining data differences between the test environment and the target environment and abnormal nodes in the target environment according to the verification result, to form a first training data set;

[0073] S403, taking the first training data set as input, adopting a decision tree algorithm to perform modeling training to form a verification model, wherein the verification model is used to verify the content of data differences existing in the process of migrating the tree structure data of the test environment to the target environment and whether there are abnormal nodes in the target environment.

[0074] In this embodiment, the tree structure data of the test environment is obtained as the input data of the algorithm. The depth-first search algorithm is used to traverse the tree structure data of the target environment to obtain the reference relationship of each node, the integrity and legality information of the data. By comparing the node reference relationship, data integrity and legality of the test environment and the target environment, the data difference between the two environments is judged, and the abnormal nodes in the target environment are determined. According to the judgment result, a first training data set containing data differences and abnormal nodes is generated. The first training data set is input into the decision tree algorithm, and a verification model is formed through modeling training. The verification model is used to verify the process of migrating the tree structure data of the test environment to the target environment. The verification result is obtained to determine whether there are data differences and abnormal nodes in the migration process. If so, the data is corrected until the verification passes.

[0075] Exemplarily, in the scenario of comparing tree structure data, a depth-first search algorithm may be used to traverse the tree structure data of the target environment and compare it with the tree structure data of the test environment to find differences and abnormal nodes.

[0076] Assume that the tree structure data of the test environment is as follows: Node A (data: 1) - Node B (data: 2) - Node D (data: 4) | - Node C (data: 3) - Node E (data: 5).

[0077] The tree structure data of the target environment is as follows: Node A (data: 1) - Node B (data: 4) - Node D (data: 4) | - Node C (data: 3) - Node F (data: 6).

[0078] First, perform a depth-first search traversal on the tree structure data of the target environment, and record the reference relationship, data integrity, and data legitimacy of each node. Starting from the root node A, first visit node B, then visit node D. After visiting node D, return to node B, then visit node C, and finally visit node F. During the traversal process, record the reference relationship between nodes, such as A->B, B->D, A->C, C->F. At the same time, verify the data integrity of each node, such as checking whether the data of node B is the expected value. And check the data legitimacy, such as checking whether the node data conforms to the predetermined format or range. Assuming that the data of node B is 4 in the target environment and is inconsistent with 2 in the test environment, record the difference. At the same time, if it is found that there is node F in the target environment, but there is no corresponding node in the test environment, node F is marked as an abnormal node. The results of the depth-first search traversal, that is, the data differences between the test environment and the target environment and the abnormal nodes in the target environment, form the first training data set. Take the first training data set as input, and use the decision tree algorithm to learn how to judge whether there are data differences or abnormal nodes in the tree structure data according to the first training data set. The trained verification model can be used to verify what data differences exist when the tree-structured data in the test environment is migrated to the target environment, and whether there are abnormal nodes. For example, when new tree-structured data is migrated from the test environment to the target environment, the verification model can be used to verify the data in the target environment to quickly identify data differences and abnormal nodes, thereby improving the efficiency and accuracy of data migration.

[0079] In some embodiments, Figure 5 As shown, the method also includes:

[0080] S501, using an AVL tree algorithm to construct an AVL tree according to the tree structure data of the test environment or the target environment;

[0081] S502, during the construction of the AVL tree, by calculating the balance factor of each node in the tree structure data, it is determined whether a rotation operation is required, and if the absolute value of the balance factor of the current node is greater than 1, the rotation operation is performed;

[0082] S503, converting the AVL tree into corresponding tree structure data to obtain optimized tree structure data, and replacing the original tree structure data in the test environment or the target environment with the optimized tree structure data.

[0083] In this embodiment, tree structure data in a test environment or a target environment is obtained as input data for constructing an AVL tree. The AVL tree algorithm is used to insert nodes one by one according to the input tree structure data to construct an AVL tree. In the process of constructing the AVL tree, the balance factor of each newly inserted node is calculated. It is determined whether the absolute value of the balance factor of the node is greater than 1. If so, the imbalance type of the node is determined. According to the imbalance type, the corresponding rotation operation is used to locally adjust the AVL tree to rebalance it. Repeat the above steps until the AVL tree is constructed to obtain a balanced AVL tree. The constructed AVL tree is converted into optimized tree structure data to replace the original tree structure data.

[0084] For example, the AVL tree algorithm is a self-balancing binary search tree that can maintain the balance of the tree through rotation operations during the process of inserting and deleting nodes, thereby preventing the tree from degenerating into a linked list and improving search efficiency. The balance factor refers to the value of the height of the left subtree minus the height of the right subtree.

[0085] Assume that an AVL tree needs to be constructed according to the tree structure data of the test environment mentioned above. The tree structure data of the test environment is as follows: node A (data: 1) - node B (data: 2) - node D (data: 4) | - node C (data: 3) - node E (data: 5). First, insert node A as the root node into the AVL tree. At this time, both the left subtree and the right subtree of A are empty, and the balance factor is 0. Next, insert node B as the left child of A. At this time, the height of the left subtree of A is 1, the height of the right subtree is 0, and the balance factor is 1. Then, insert node D as the right child of B. At this time, the height of the left subtree of B is 0, the height of the right subtree is 1, and the balance factor is -1. The height of the left subtree of A is 2, the height of the right subtree is 0, and the balance factor is 2, which exceeds the range allowed by the AVL tree (-1, 0, 1). Therefore, a right rotation operation is required. The specific process of the right rotation operation is: use B as the new root node, A as the right child of B, and D as the left child of B. After the rotation, the balance factor of B is 0, and the balance factor of A is 0. Next, insert node C as the left child of A. At this time, the height of A's left subtree is 1, the height of the right subtree is 0, and the balance factor is 1. The height of B's ​​left subtree is 1, the height of the right subtree is 2, and the balance factor is -1. Finally, insert node E as the right child of C. At this time, the height of C's left subtree is 0, the height of the right subtree is 1, and the balance factor is -1. The height of A's left subtree is 2, the height of the right subtree is 0, and the balance factor is 2. The height of B's ​​left subtree is 1, the height of the right subtree is 3, and the balance factor is -2, which exceeds the range allowed by the AVL tree and needs to be rotated left. Take A as the new root node, C as the left child of A, B as the right child of A, D as the left child of B, and E as the right child of C. After the rotation, the balance factor of A is 0, the balance factor of B is 0, and the balance factor of C is 0. After constructing the AVL tree, a balanced tree structure is obtained. Convert the AVL tree to the corresponding tree structure data to obtain the optimized tree structure data: Node A (data: 1) - Node C (data: 3) - Node E (data: 5) | - Node B (data: 2) - Node D (data: 4). Replacing the original tree structure data in the test environment with the optimized tree structure data can improve the efficiency of subsequent data comparison. When performing a depth-first search, a balanced tree structure can avoid a long search path, thereby reducing the search time.

[0086] In some embodiments, Figure 6 As shown, the method also includes:

[0087] S601, using the Prometheus monitoring system to collect in real time various indicator data in the process of migrating the tree structure data of the test environment to the target environment, and forming a monitoring report;

[0088] S602, determining migration failure nodes and corresponding migration failure reasons according to the monitoring report to form a second training data set;

[0089] S603, taking the second training data set as input, adopting the decision tree algorithm to perform modeling training, and forming a migration failure detection model, wherein the migration failure detection model is used to determine whether there is a migration failure node and the migration failure reason corresponding to the migration failure node during the migration of the tree structure data of the test environment to the target environment.

[0090] In this embodiment, the Prometheus monitoring system is used to collect various indicator data during the tree structure data migration process from the test environment to the target environment, including migration progress, migration speed, resource occupancy, etc., to form a real-time monitoring data stream. The monitoring data stream is analyzed and visualized in real time to generate an intuitive monitoring report, which contains a key indicator trend chart and abnormal situation warnings of the migration process. According to the abnormal situation warning in the monitoring report, the positioning algorithm of the migration failure node is automatically triggered. The algorithm determines whether the node has failed to migrate by analyzing the node's status information and log data, and determines the corresponding failure cause, such as network interruption, data corruption, etc. The located migration failure node and failure cause are used as sample data and merged with the historical migration failure case data to form a second training data set. The data set is preprocessed to extract key features. The decision tree algorithm is used to model the second training data set, and the optimal partition attribute is selected by information gain or Gini index, and a decision tree is recursively constructed until the conditions of the leaf node are met. At the leaf nodes of the decision tree, corresponding migration failure detection rules are generated according to the data distribution of each node to form a complete migration failure detection model. The migration failure detection model is deployed to the target environment to predict and judge the migration indicator data of the tree structure data collected in real time. If the characteristics of a node are detected to meet the rules of the failure model, an alert is issued that the node may fail to migrate, and the potential cause of failure is given, triggering the corresponding exception handling process. At the same time, the data of the node is fed back to the training data set to achieve continuous optimization and updating of the model.

[0091] For example, assume that the tree structure data of the test environment consists of nodes A, B, C, D, and E, and there is a parent-child relationship between them, for example, A is the parent node of B and C, B is the parent node of D, and C is the parent node of E. The target environment is a new database system. The migration process is to copy these node data and their relationships from the test environment to the target environment.

[0092] The metrics data that Prometheus can monitor include: the migration status of each node data (e.g., success, failure, in progress), migration time, data transmission rate, network connection status, database load of the target environment, disk space usage, etc. These metrics data can be queried and aggregated in real time using the query language PromQL provided by Prometheus. For example, you can query the migration status of a node, or count the average migration time of all nodes.

[0093] Prometheus stores the collected indicator data in its own time series database and provides a visual interface, Grafana, to display monitoring reports. The monitoring report can intuitively show the changing trends of various indicator data during the migration process, helping to quickly discover abnormal situations during the migration process. For example, if the migration of a node takes too long, or the database load of the target environment is too high, these can be clearly shown in the monitoring report.

[0094] By analyzing the monitoring report, you can determine the nodes that failed to migrate and the corresponding reasons for the migration failure. For example, the monitoring report shows that the migration status of node D is "failed" and the migration failure reason is "target environment database connection timeout". This information can be recorded to form a second training data set.

[0095] Assume that the following migration failure records are collected: Node D, database connection timeout; Node E, insufficient disk space; Node F, data verification failure. Using the second training data set, the decision tree algorithm can be used for modeling training to form a migration failure detection model. In this example, the decision tree algorithm can be used to learn the rules of migration failure. For example, if the target environment database connection times out, the node migration is likely to fail; if the target environment disk space is insufficient, the node migration is likely to fail. The trained migration failure detection model can be used to determine whether there is a migration failure node and the migration failure reason corresponding to the migration failure node during the migration of the tree structure data of the test environment to the target environment. For example, when migrating node G, the model can predict whether node G will fail to migrate and the possible reasons for migration failure based on the current monitoring indicator data (for example, database connection status, disk space usage, etc.). This can provide early warning of potential migration failure risks and guide appropriate measures to avoid migration failure. For example, if the model predicts that node G may fail to migrate due to insufficient disk space, the disk space of the target environment can be cleaned up in advance.

[0096] The training process of the decision tree model can be simply understood as: building a tree based on the existing data (the second training data set), each node of the tree represents a judgment condition (for example: whether the database connection status is normal), each branch represents a judgment result (for example: normal, abnormal), and each leaf node represents a final classification result (for example: migration success, migration failure). When there is new data to be predicted, it starts from the root node of the tree, selects branches according to the judgment conditions of each node, and finally reaches a leaf node to obtain the prediction result.

[0097] In some embodiments, Figure 7 As shown, the method also includes:

[0098] S701, dividing the tree structure data into a plurality of partitions using a partitioning algorithm according to the hierarchical relationship of the tree structure data;

[0099] S702, when migrating the tree structure data of the test environment to the target environment, for each partition, a Map task is started to traverse the node data in the partition to form local tree structure data;

[0100] S703, constructing complete tree structure data by merging the local parent-child node relationship trees output by multiple Map tasks.

[0101] In this embodiment, according to the hierarchical relationship of the tree structure data, a partitioning algorithm is used to divide the tree structure data into multiple partitions to obtain the partitioned tree structure data. For each partition, a Map task is started to traverse the node data in the partition, and local tree structure data is constructed by analyzing the parent-child relationship between the nodes. The local parent-child node relationship trees output by the Map task are merged, and the complete tree structure data is constructed by judging the hierarchical relationship between the nodes. The tree structure data in the test environment is obtained, and it is divided into multiple partitions according to the partitioning algorithm to obtain the partitioned tree structure data of the test environment. For each partition of the test environment, a Map task is started to traverse the node data in the partition, and the local tree structure data of the test environment is constructed by analyzing the parent-child relationship between the nodes. The local parent-child node relationship trees output by multiple Map tasks of the test environment are merged, and the complete tree structure data of the test environment is constructed by judging the hierarchical relationship between the nodes. The complete tree structure data of the test environment is migrated to the target environment, and the node data that needs to be updated or added is determined by comparing the differences between the tree structure data of the target environment and the test environment, and the migration of the tree structure data is completed.

[0102] Exemplarily, the partitioning algorithm can divide the data into multiple logical partitions according to the hierarchical relationship of the tree-structured data for parallel processing. Each partition contains a portion of nodes and their child nodes, so that a large tree-structured data can be decomposed into multiple small blocks for easy distributed processing.

[0103] For example, suppose there is the following tree structure data: A is the root node, B and C are the child nodes of A, D and E are the child nodes of B, and F and G are the child nodes of C. This tree can be divided into two partitions: Partition 1 contains nodes B, D, and E; Partition 2 contains nodes C, F, and G. The information of the root node A can be shared in all partitions. The Map task can traverse the node data in each partition to form local tree structure data. For the above example, start two Map tasks: Map1 processes the data of partition 1, and Map2 processes the data of partition 2. Map1 reads the data of partition 1, recognizes that B is the parent node of D and E, and outputs the local parent-child node relationship tree: B->{D, E}. Map2 reads the data of partition 2, recognizes that C is the parent node of F and G, and outputs the local parent-child node relationship tree: C->{F, G}. The local parent-child node relationship trees output by multiple Map tasks are merged to construct a complete tree structure data. In the above example, the output results of Map1 and Map2 are merged. Since A is known to be the parent node of B and C, combined with the output of Map1 B->{D, E} and the output of Map2 C->{F, G}, the complete tree structure data is finally constructed: A->{B->{D, E}, C->{F, G}}. The partitioning algorithm and MapReduce parallel processing can significantly improve the efficiency of tree structure data migration. Large tree structure data usually contains a large number of nodes. If a single-threaded serial processing method is used, the migration speed will be very slow. The partitioning algorithm can decompose the data into multiple small blocks, each of which can be processed in parallel by a Map task, thereby speeding up the migration speed. At the same time, parallel processing can also make full use of the computing resources of multi-core CPUs to further improve the migration efficiency. This method also improves the fault tolerance of the migration process. If an error occurs during the migration of a partition, you only need to re-migrate the partition instead of re-migrating the entire tree structure data.

[0104] Reference Figure 8 An embodiment of the present invention provides a processing system 8 for migrating tree structure data, and the system 8 specifically includes:

[0105] The first processing module 801 is used to construct a test environment parent-child node relationship tree according to the parent-child node relationship of the tree structure data in the test environment;

[0106] The second processing module 802 is used to form a sub-node mapping table by establishing a mapping relationship between each old sub-node in the test environment and each new sub-node in the target environment after the tree structure data is imported into the target environment;

[0107] The third processing module 803 is used to construct a parent-child node relationship tree of the target environment by determining a new parent node corresponding to each new child node in the target environment;

[0108] The fourth processing module 804 is used to determine the old parent node in the test environment corresponding to each new parent node in the target environment according to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, and update the data of each new parent node.

[0109] It is understandable that if Figure 1 The contents of the processing method for migrating tree structure data in the embodiment shown in the figure are all applicable to the processing system for migrating tree structure data in the embodiment. The functions specifically implemented by the processing system for migrating tree structure data in the embodiment are similar to those in the embodiment shown in the figure. Figure 1 The processing method for migrating tree structure data shown in the embodiment is the same as that in the embodiment shown in the embodiment, and the beneficial effects achieved are the same as those in the embodiment shown in the embodiment. Figure 1 The beneficial effects achieved by the embodiment of the method for processing the migration tree structure data shown are also the same.

[0110] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0111] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0112] Reference Fig. 9The embodiment of the present invention further provides a computer device 9, comprising: a memory 902 and a processor 901 and a computer program 903 stored in the memory 902. When the computer program 903 is executed on the processor 901, a method for processing migration tree structure data as described in any one of the above methods is implemented.

[0113] The computer device 9 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 9 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will appreciate that Fig. 9 It is only an example of the computer device 9 and does not constitute a limitation on the computer device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0114] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0115] In some embodiments, the memory 902 may be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 902 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 9. Further, the memory 902 may also include both an internal storage unit of the computer device 9 and an external storage device. The memory 902 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 902 may also be used to temporarily store data that has been output or is to be output.

[0116] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for processing the migration tree structure data as described in any one of the above methods is implemented.

[0117] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for processing tree structure data migration, characterized in that: The method specifically comprises: According to the parent-child node relationship of the tree structure data in the test environment, a parent-child node relationship tree of the test environment is constructed; After the tree structure data is imported into the target environment, a sub-node mapping table is formed by establishing a mapping relationship between each old sub-node in the test environment and each new sub-node in the target environment; By determining a new parent node corresponding to each new child node in the target environment, a parent-child node relationship tree of the target environment is constructed; According to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, the old parent node in the test environment corresponding to each new parent node in the target environment is determined, and the data of each new parent node is updated.

2. The method according to claim 1, characterized in that The step of establishing a mapping relationship between each old sub-node in the test environment and each new sub-node in the target environment to form a sub-node mapping table specifically includes: Constructing an attribute feature vector for each old child node in the test environment and each new child node in the target environment respectively; Obtaining a node similarity matrix by comparing and calculating the similarity between the attribute feature vectors of each old child node in the test environment and each new child node in the target environment; By comparing the similarity between each old child node and each new child node in the node similarity matrix with a preset similarity threshold, a mapping relationship between the old child nodes and the new child nodes is established to form a child node mapping table.

3. The method according to claim 2, characterized in that The method further comprises: Using the MerkleTree algorithm to compare the tree structure data of the test environment and the target environment, determine the change status of each node, and generate a change record; According to the change record, the tree structure data of the target environment is synchronously updated using a breadth-first search algorithm.

4. The method according to claim 3, characterized in that The method further comprises: Based on the tree structure data of the test environment, a depth-first search algorithm is used to traverse the reference relationship, data integrity and data legality between each node of the tree structure data of the target environment to obtain a verification result; According to the verification result, determining the data difference between the test environment and the target environment and the abnormal nodes in the target environment to form a first training data set; The first training data set is used as input, and a decision tree algorithm is used for modeling training to form a verification model. The verification model is used to verify the data differences existing in the process of migrating the tree structure data of the test environment to the target environment and whether there are abnormal nodes in the target environment.

5. The method according to claim 4, characterized in that The method further comprises: Using an AVL tree algorithm, constructing an AVL tree according to the tree structure data of the test environment or the target environment; In the process of constructing the AVL tree, the balance factor of each node in the tree structure data is calculated to determine whether a rotation operation is required. If the absolute value of the balance factor of the current node is greater than 1, the rotation operation is performed; The AVL tree is converted into corresponding tree structure data to obtain optimized tree structure data, and the optimized tree structure data is used to replace the original tree structure data in the test environment or the target environment.

6. The method according to claim 5, characterized in that The method further comprises: Use the Prometheus monitoring system to collect various indicator data in the process of migrating the tree structure data of the test environment to the target environment in real time to form a monitoring report; Determine migration failure nodes and corresponding migration failure reasons according to the monitoring report to form a second training data set; The second training data set is used as input, and a decision tree algorithm is used for modeling training to form a migration failure detection model. The migration failure detection model is used to determine whether there is a migration failure node and the migration failure reason corresponding to the migration failure node during the migration of the tree structure data of the test environment to the target environment.

7. The method according to claim 6, characterized in that The method further comprises: According to the hierarchical relationship of the tree structure data, a partitioning algorithm is used to divide the tree structure data into a plurality of partitions; When migrating the tree structure data of the test environment to the target environment, for each partition, a Map task is started to traverse the node data in the partition to form local tree structure data; By merging the local parent-child node relationship trees output by multiple Map tasks, a complete tree structure data is constructed.

8. A processing system for migrating tree structure data, characterized in that: The system specifically comprises: The first processing module is used to construct a test environment parent-child node relationship tree according to the parent-child node relationship of the tree structure data in the test environment; A second processing module is used for, after importing the tree structure data into the target environment, establishing a mapping relationship between each old child node in the test environment and each new child node in the target environment to form a child node mapping table; A third processing module, configured to construct a parent-child node relationship tree of the target environment by determining a new parent node corresponding to each new child node in the target environment; The fourth processing module is used to determine the old parent node in the test environment corresponding to each new parent node in the target environment according to the test environment parent-child node relationship tree, the child node mapping table and the target environment parent-child node relationship tree, and update the data of each new parent node.

9. A computer device, characterized in that: include: A memory and a processor and a computer program stored in the memory, when the computer program is executed on the processor, implements the processing method for migrating tree structure data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for processing migration tree structure data as claimed in any one of claims 1 to 7 is implemented.

Images