A data synchronization method, terminal device and storage medium based on tree structure

Through the data synchronization method based on tree structure, using text similarity matching and preprocessing technology, the problems of low data synchronization efficiency and poor accuracy in the existing technology are solved, and efficient and accurate data synchronization is achieved, which is suitable for large-scale Unit data adjustment.

CN114443673BActive Publication Date: 2024-11-19XIAMEN MEIYA PICO INFORMATION CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111533478.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-15
Publication Date
2024-11-19
Estimated Expiration
2041-12-15

AI Technical Summary

Technical Problem

The prior art is inefficient and has poor accuracy when processing unit data adjustments of business systems. Especially when a large amount of data is synchronized, the efficiency and accuracy of manual operation are significantly reduced, making misjudgments and errors prone to large-scale data. synchronous.

Method used

The data synchronization method based on the tree structure is adopted, and the node abbreviation is extracted by receiving and preprocessing data, and matching based on text similarity is performed to reduce manual operations and improve efficiency and accuracy.

Benefits of technology

On the basis of reducing manual operations, the efficiency and accuracy of data synchronization are significantly improved, and the adjustment of large units of data can be processed in a short time, which improves work efficiency and ensures the accuracy of modification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114443673B_ABST
    Figure CN114443673B_ABST
Patent Text Reader

Abstract

The present invention relates to a data synchronization method, terminal device and storage medium based on a tree structure, the method comprising: S1: receiving a data group before synchronization and a data group to be synchronized, and extracting the abbreviation of the content corresponding to each node; S2: synchronizing data level by level starting from the top node, when synchronizing to the i-th level, searching for nodes with different abbreviations in the two data groups, and matching them in pairs according to the principle of maximum text similarity, and sending the matching results for manual confirmation; if the manual confirmation is consistent, the content of the node in the data group before synchronization is updated to the content of the node in the data group to be synchronized; if the manual confirmation is inconsistent, the node in the data group before synchronization is deleted, or the node in the data group to be synchronized is added to the data group before synchronization. The present invention improves efficiency and ensures accuracy on the basis of minimizing manual operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data synchronization method, terminal equipment and storage medium based on a tree structure. Background Art

[0002] As society develops and changes, some organizations need to adjust their internal structures to adapt to development. Big ones include administrative division changes, and small ones include company department adjustments. As a business system, there is generally a corresponding set of unit data, and corresponding adjustments must be made to adapt to business changes. At this time, if the unit data in the business system is associated with other data based on a custom primary key such as ID, it cannot be simply deleted and rebuilt. At this time, it is necessary to compare the new and old data to add, delete, modify, and other operations. The current practice is generally to compare the names to determine whether the new and old units are consistent, because the name is a simple and direct key field. The common practice is: operate the data in the business system, check one by one whether the new unit exists in the system, and add new units if there are no new units; if the information has changed, modify it; the old unit that does not exist in the new unit needs to be deleted.

[0003] The above manual operation method is relatively slow and cannot be applied to large amounts of data synchronization. If you use tools (such as SQL table association or Excel functions, etc.) to make a simple comparison based on the unit name, misjudgment may occur. For example: Tong'an County, Fujian Province and Tong'an District, Xiamen City, Fujian Province, Wenquan Town, Kai County, Chongqing City and Wenquan Town, Kaizhou District, Chongqing City. In these cases, only individual words are different. If you make a simple comparison, you cannot determine whether they are the same. Manual intervention is required to search for similar names. The accuracy of manual retrieval is low. If you want to improve the accuracy, you need to search several times, but the efficiency is not high. In addition, the more manual operations you do, the greater the probability of errors, which affects the accuracy.

[0004] In addition, in actual applications, when dealing with units above 10,000, manual operations will see an exponential decrease in efficiency and accuracy. For example, a certain platform product has been deployed in more than 300 sets across the country, and the unit list of each platform may not be the same as that of its superior system. Now that relevant departments have issued standards, it is necessary to unify all unit data, and after aggregating data at all levels, unified statistical assessment is required. The data of these old systems cannot be completely deleted and re-entered. In this case, each deployed product has to be modified to the published standards. For example, there are 340,000 units nationwide, generally about 10,000 at the provincial level, and about 1,000 at the municipal level. So how to quickly and uniformly modify such a large amount of unit data becomes a problem that needs to be solved. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a data synchronization method, terminal equipment and storage medium based on a tree structure.

[0006] The specific plan is as follows:

[0007] A data synchronization method based on a tree structure comprises the following steps:

[0008] S1: receiving a pre-synchronization data group and a to-be-synchronized data group, and extracting the abbreviation of the corresponding content of each node in the pre-synchronization data group and the to-be-synchronized data group;

[0009] S2: Synchronize data level by level starting from the top-level node. When synchronizing to the i-th level, find the nodes whose abbreviations are different between the i-th level of the data group before synchronization and the i-th level of the data group to be synchronized, and based on the nodes with different abbreviations, match the abbreviations in the data group before synchronization with the abbreviations in the data group to be synchronized in pairs according to the principle of maximum text similarity, and send the matching results for manual confirmation; if the manual confirmation matching result is that the abbreviations are consistent, update the content of the node in the data group before synchronization corresponding to the matching result to the content of the node in the data group to be synchronized corresponding to the matching result; if the manual confirmation matching result is that the abbreviations are inconsistent, delete the node in the data group before synchronization corresponding to the matching result, or add the node in the data group to be synchronized corresponding to the matching result to the data group before synchronization.

[0010] Furthermore, before extracting the abbreviation in step S1, each received set of data is preprocessed, and the preprocessing includes: removing node contents without a parent in each set of data to ensure the integrity of the tree structure; and removing duplicate contents in each set of data.

[0011] Furthermore, the method for extracting the abbreviation of the content corresponding to each node in each set of data is: for each node, the result of subtracting the content corresponding to the parent node of the node from the content corresponding to the node is used as the abbreviation of the content corresponding to the node.

[0012] Furthermore, after extracting the abbreviations of the contents corresponding to each node in each set of data in step S1, it is also necessary to perform a duplication check on all the abbreviations corresponding to each set of data, and if there are duplicate abbreviations, they are modified.

[0013] Furthermore, when the tree-structured data is administrative division data, the text similarity calculation also includes: further extracting the abbreviation, that is, deleting the words representing the administrative division unit in the abbreviation.

[0014] A tree-structured data synchronization terminal device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0015] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0016] The present invention adopts the above technical solution, which improves efficiency and ensures accuracy while minimizing manual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of Embodiment 1 of the present invention.

[0018] Figure 2 Shown is a schematic diagram of a tree structure data set in this embodiment. DETAILED DESCRIPTION

[0019] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0020] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0021] Embodiment 1:

[0022] A set of unit data generally has a hierarchical structure with a clear superior-subordinate relationship. A unit has only one superior and may have multiple subordinates, which is a typical tree structure. The embodiment of the present invention provides a data synchronization method based on a tree structure, which is used to synchronize the data in the data group before synchronization with the data in the data group to be synchronized, such as Figure 1 As shown, the method comprises the following steps:

[0023] S1: receiving a pre-synchronization data group and a to-be-synchronized data group, and extracting the abbreviation of the content corresponding to each node in the pre-synchronization data group and the to-be-synchronized data group.

[0024] In this embodiment, before extracting the abbreviation, each group of received data is also preprocessed. The preprocessing includes two parts. One part is that nodes without superiors are invalid and should be removed. Therefore, the node content without superiors in each group of data is removed in the preprocessing to ensure the integrity of the tree structure. In this embodiment, the superior of the first level is set to the 0th level, indicating that the first level is the top level, as shown in Table 1. The other part is to remove the duplicate content in each group of data to avoid the corresponding subsequent matching.

[0025] Table 1

[0026]

[0027]

[0028] The method for extracting the abbreviation set in this embodiment is: for each node, the result of subtracting the corresponding content of the node from the corresponding content of the parent node is used as the abbreviation of the corresponding content of the node. For example, the content corresponding to the parent node in "Tong'an County, Fujian Province" is "Fujian Province", and the abbreviation after the subtraction of the two is "Tong'an County", and the content corresponding to the parent node in "Tong'an District, Xiamen City, Fujian Province" is "Xiamen City, Fujian Province", and the abbreviation after the subtraction of the two is "Tong'an District".

[0029] In addition, in order to ensure that the abbreviations corresponding to different subordinate nodes of the same superior are not repeated, after extracting the abbreviations in step S1, the embodiment also needs to perform a repeatability check on all the abbreviations corresponding to each group of data, and if there are repeated abbreviations, they are modified. The modification method can be manual adjustment (i.e., manual modification of the abbreviation).

[0030] S2: Synchronize data level by level starting from the top-level node. When synchronizing to the i-th level, find the nodes with different abbreviations in the i-th level of the data group before synchronization and the i-th level of the data group to be synchronized, and match the abbreviations in the data group before synchronization and the abbreviations in the data group to be synchronized according to the principle of maximum similarity based on the nodes with different abbreviations, and send the matching results for manual confirmation; if the manual confirmation matching result is that the abbreviations are consistent, update the content of the node in the data group before synchronization corresponding to the matching result to the content of the node in the data group to be synchronized corresponding to the matching result; if the manual confirmation matching result is that the abbreviations are inconsistent, delete the node in the data group before synchronization corresponding to the matching result, or add the node in the data group to be synchronized corresponding to the matching result to the data group before synchronization. i is a positive integer.

[0031] like Figure 2 As shown, the left figure is the tree structure corresponding to the data group before synchronization, and the right figure is the tree structure corresponding to the data group to be synchronized. Start synchronization from the top-level (level 1) node. If the abbreviations of the top-level nodes corresponding to the two data groups are the same, do not modify them, and enter the second-level data synchronization. The node with the same abbreviation in the second level is Yuzhou District, and the nodes with different abbreviations are "Kai County" and "Liangping County", and the data groups to be synchronized are "Kaizhou District" and "Liangping District". The abbreviations in the data group before synchronization and the abbreviations in the data group to be synchronized are matched in pairs, that is, an iterative method is used to calculate the text similarity between each abbreviation in the data group before synchronization and each abbreviation in the data group to be synchronized, and the abbreviation with the highest text similarity calculated is used as the matching abbreviation of the abbreviation in the data group before synchronization, and the two are matched as a group.

[0032] The text similarity comparison is used to determine the similarity between two abbreviations. The more similar they are, the more likely they are the same unit. The similarity between the abbreviations in this embodiment is calculated using the edit distance algorithm, that is, the minimum number of edit operations required to convert one string into another string between two strings. Edit operations include: inserting, replacing, and deleting a character. The fewer the number of operations, the smaller the edit distance, and the greater the similarity between the two strings.

[0033] Assume that the first string S1 has length i and the second string S2 has length j, and edit[i][j] represents the edit distance between S1 and S2. The subscript of the string starts at 1, and subscript 0 means that a null value is reserved in front of the string. It is possible to add an empty string in front of S1 and S2 to get the edit distance.

[0034]

[0035] Among them, S1[i]=S2[j], flag=0; S1[i]≠S2[j], flag=1.

[0036] 1. edit[i-1][j]+1 means adding one character to S1.

[0037] 2. edit[i][j-1]+1 means adding one character to S2.

[0038] 3. edit[i-1][j-1]+flag means that S1 and S2 replace at most one character.

[0039] Find the minimum value of the above three cases, that is, the minimum number of editing operations, which is the edit distance.

[0040] According to the formula, the edit distance between all S1 and S2 substrings can be recursively calculated. As shown in Table 2, the edit distance is calculated using fusion and fashion as examples:

[0041] Table 2

[0042]

[0043] It can be seen that the specific change process is: fusion, fasion, fashion, and the value of the last unit cell in the lower right corner is the edit distance between the two 2. Then, text similarity = 1-edit distance / Max(i,j), and the calculated similarity between fusion and fashion is 71.4%.

[0044] Since the similarity between Liangping County and Liangping District and Liangping District and Liangping District and Liangping District may be 66.7% in the calculation process of the abbreviation text similarity, this phenomenon does not meet the requirements. Therefore, in order to solve this problem, in this embodiment, it is set that before the text similarity calculation, the abbreviation is further extracted, that is, the words representing the administrative division units (i.e., "province", "city", "district", etc.) in the abbreviation are deleted. The matching results between the abbreviations in the second level after further extraction are shown in Table 3.

[0045] Table 3

[0046] Data group before synchronization Data group to be synchronized Similarity Manual confirmation of matching results Kai County Kaizhou District 50% Consistency Liangping County Liangping District 100% Consistency Abolition of County 0% Inconsistency New Area 0% Inconsistency

[0047] When manually confirming the matching results, you can check the administrative division change records. After checking, it is found that Kai County was abolished and a district was established, and it was changed to Kaizhou District, so it is determined that the abbreviation is consistent. For matching results with low similarity (such as 0%), it means that the node in the data group to be synchronized should not be the synchronization node of the node in the data group before synchronization. If the node in the data group before synchronization cannot find a node in the data group to be synchronized with the manually confirmed matching result as consistent with the abbreviation, then the node should be deleted; if the node in the data group to be synchronized cannot find a node in the data group before synchronization with the manually confirmed matching result as consistent with the abbreviation, then the node should be added to the data group before synchronization. It should be noted that when deleting a node, all child nodes corresponding to the node should be recursively deleted.

[0048] When the second level synchronization is completed, the above steps are repeated to synchronize the nodes of the subsequent levels in sequence, which will not be described in detail here.

[0049] The embodiment of the present invention does not require excessive manual intervention. Once the data is sorted, the method of this embodiment determines the consistent units, and recommends matching results with high similarity for manual judgment of whether the inconsistent units are consistent. Compared with the previous manual search and comparison of different units, the adjustment of units at the thousand level took about a week, but now it can be completed in half a day, which improves the work efficiency by 10 times and ensures the accuracy of the modification.

[0050] Since tree-structured data such as units and addresses are widely present in information systems of all walks of life, the method based on hierarchical recursion and text similarity comparison can efficiently and accurately handle large-scale data changes. It can be independently developed into a business-independent project or product and applied to various fields.

[0051] Embodiment 2:

[0052] The present invention also provides a tree-structured data synchronization terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.

[0053] Further, as an executable solution, the tree-structured data synchronization terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The tree-structured data synchronization terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art may understand that the composition structure of the tree-structured data synchronization terminal device is merely an example of a tree-structured data synchronization terminal device, and does not constitute a limitation on the tree-structured data synchronization terminal device. It may include more or less components than the above, or a combination of certain components, or different components. For example, the tree-structured data synchronization terminal device may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0054] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the data synchronization terminal device based on the tree structure, and uses various interfaces and lines to connect the various parts of the entire data synchronization terminal device based on the tree structure.

[0055] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the data synchronization terminal device based on the tree structure by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and an application required for at least one function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0056] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0057] If the module / unit integrated in the tree-structured data synchronization terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0058] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A data synchronization method based on a tree structure, for synchronizing data in a data group before synchronization into data in a data group to be synchronized, characterized in that: The following steps are involved: S1: receiving a pre-synchronization data group and a to-be-synchronized data group, and extracting the abbreviation of the content corresponding to each node in the pre-synchronization data group and the to-be-synchronized data group; the abbreviation of the content corresponding to each node in each data group is extracted by: for each node, the result of subtracting the content corresponding to the node from the content corresponding to the parent node of the node is used as the abbreviation of the content corresponding to the node; S2: Synchronize data level by level starting from the top-level node. When synchronizing to the i-th level, find the nodes whose abbreviations are different between the i-th level of the data group before synchronization and the i-th level of the data group to be synchronized. Based on the nodes with different abbreviations, match the abbreviations in the data group before synchronization with the abbreviations in the data group to be synchronized in pairs according to the principle of maximum text similarity, and send the matching results for manual confirmation; If the matching result is manually confirmed to be consistent with the abbreviation, the content of the node in the pre-synchronization data group corresponding to the matching result with the consistent abbreviation is updated to the content of the node in the to-be-synchronized data group corresponding to the matching result with the consistent abbreviation; If the matching result is manually confirmed to be inconsistent in abbreviation, the node in the pre-synchronization data group corresponding to the matching result with inconsistent abbreviation is deleted, and all child nodes corresponding to the node are recursively deleted, or the node in the to-be-synchronized data group corresponding to the matching result with inconsistent abbreviation is added to the pre-synchronization data group.

2. The tree-structure-based data synchronization method according to claim 1, characterized in that: Step S1 also includes preprocessing each received set of data before extracting the abbreviation, and the preprocessing includes: removing node content without a parent in each set of data to ensure the integrity of the tree structure; and removing duplicate content in each set of data.

3. The tree-structure-based data synchronization method according to claim 1, characterized in that: After extracting the abbreviations of the contents corresponding to each node in each set of data in step S1, it is also necessary to perform a duplication check on all the abbreviations corresponding to each set of data. If there are duplicate abbreviations, they are modified.

4. The tree-structure-based data synchronization method according to claim 1, characterized in that: When the tree-structured data is administrative division data, the text similarity calculation also includes: further extracting the abbreviation, that is, deleting the words representing the administrative division unit in the abbreviation.

5. A data synchronization terminal device based on a tree structure, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.