A multi-source data management method and system applied to grassroots platforms

Abstract

The present invention relates to the technical field of multi-source data management, and in particular to a multi-source data management method and system applied to a grassroots platform. The method comprises the following steps: S1: obtaining multi-source data of a grassroots platform and preprocessing; dividing the multi-source data into form data 1, form data 2, form data 3, ..., form data N according to the order of timestamps; obtaining the number of data items in the form data, and dividing the data items in the form data into data item 1, data item 2, data item 3, ..., data item M according to the order of data item distribution. The present invention preprocesses data by efficient sorting and classification, constructs mapping relationships using hash tables and dynamic programming, and improves comparison efficiency; initially manages and marks problem data items, and secondarily manages and calculates error probabilities to provide a scientific basis; and tertiary management ensures data consistency by balancing the management tree, thereby improving the overall performance and reliability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source data management, and in particular to a multi-source data management method and system applied to a grassroots platform. Background Art

[0002] The advantage of multi-source data management methods is that they can achieve data consistency and high quality by integrating, cleaning, converting and fusing data from different sources, thereby supporting more accurate analysis and decision-making. Multi-source data management methods for grassroots platforms further improve data processing capabilities on this basis, and can effectively integrate and process data from different sources, enhance data consistency and availability, and thus better support the operation and decision-making of grassroots platforms. However, existing multi-source data management methods for grassroots platforms have shortcomings in real-time processing of large amounts of dynamic data, and it is difficult to efficiently cope with the real-time processing requirements of dynamic data streams. In addition, these methods still face challenges in data fusion and consistency assurance, which affects data quality and decision-making accuracy. Specifically, existing methods also have disadvantages in data comparison, making it difficult to achieve real-time and accurate comparative analysis, especially in processing dynamic data streams and ensuring data consistency. Summary of the invention

[0003] In order to overcome the shortcoming of low accuracy of multi-source data in data comparison, the present invention provides a multi-source data management method and system applied to a grassroots platform.

[0004] The technical implementation scheme of the present invention is: a multi-source data management method applied to a grassroots platform, comprising the following steps:

[0005] S1: Obtain multi-source data from the grassroots platform and perform preprocessing; divide the multi-source data into form data 1, form data 2, form data 3, ..., form data N according to the order of timestamps; obtain the number of data items in the form data, and divide the data items in the form data into data item 1, data item 2, data item 3, ..., data item M according to the order of data item distribution;

[0006] S2: constructing a mapping relationship for form data comparison based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item M;

[0007] S3: Based on the mapping relationship, perform preliminary management on multi-source data; based on the preliminary management results, perform secondary management on multi-source data; based on the secondary management results, perform tertiary management on multi-source data;

[0008] The preliminary management is to record and mark the consistent and inconsistent data items in the form data, and especially mark the inconsistent items;

[0009] The secondary management is to calculate the error probability of inconsistent data items, evaluate the overall error rate of each data item and form data, generate a detailed error analysis report, and guide subsequent data correction work;

[0010] The three managements are based on the error analysis report, by constructing and adjusting the balance management tree, re-comparing and confirming the inconsistent data items, and updating their status records according to the comparison results.

[0011] Preferably, the method of obtaining multi-source data of the grassroots platform and preprocessing it; dividing the multi-source data into form data 1, form data 2, form data 3, ..., form data N in order of timestamps; obtaining the number of data items in the form data, and dividing the data items in the form data into data item 1, data item 2, data item 3, ..., data item M in order of distribution of the data items, includes: using merge sort in order of timestamps and in the order of time when the multi-source data is collected, dividing the multi-source data into form data 1, form data 2, form data 3, ..., form data N, N being the last form data collected in the multi-source data; using bucket sort in the order of distribution of the data items and in the order of spatial distribution of the data items in the form data, dividing all data items in the form data into data item 1, data item 2, data item 3, ..., data item M, M being the last data item in the form data.

[0012] Preferably, the constructing of a mapping relationship for form data comparison based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item M comprises: using a hash table to assign a unique serial number to the N form data in the order of timestamps; using dynamic programming to assign M data items to the N form data for comparison according to the serial number of the form data; sequentially forming a sequence to be compared based on the number of assigned data items; and comparing the N form data based on the constructed mapping relationship.

[0013] Preferably, the comparing of N form data based on the constructed mapping relationship includes:

[0014] First, for form data 1, only data item 1 is extracted; data item 1 in form data 1 is compared one-to-one with data item 1 in form data 2, and the comparison result is recorded;

[0015] Secondly, for form data 2, extract data item 1 and data item 2; after obtaining the comparison result of the previous step, compare data item 1 in form data 2 with data item 1 in form data 3; then, compare data item 2 in form data 2 with data item 2 in form data 3; repeat this process until the last form data N is processed. For form data N, extract all data items 1 to data item N; if form data N+1 exists, continue to compare the data items of form data N with the corresponding data items of form data N+1; if form data N+1 does not exist, directly enter the reverse verification stage; reversely verify the data items that have not been compared in all form data.

[0016] Preferably, the reverse verification of the uncompared data items in all form data includes: starting from form data N, arranging in reverse order as form data N, form data N-1, ..., form data 1; and comparing the reversed form data in turn according to the same rules as above.

[0017] Preferably, the multi-source data is initially managed based on the mapping relationship, including: recording consistent and inconsistent data items during each comparison process; using a binary search tree to mark the data items that have completed the comparison and the data items to be compared; and specially marking the data items with inconsistent comparison results.

[0018] Preferably, the secondary management of multi-source data based on the preliminary management result includes:

[0019] When evaluating the error probability of inconsistent data items, the initial error probability is first calculated using the Bayesian estimation formula, and the first error rate is calculated based on the number of compared and remaining uncompared data items; then, the second error rate is calculated based on the number of related data items that have been compared in all previous form data and will be compared in the future; finally, the two error rates are combined and the final calculation is performed using the data item error probability formula and the form data overall error probability formula.

[0020] Preferably, the final calculation using the data item error probability formula and the form data overall error probability formula includes: the data item error probability formula is, in, For the The error probability of a data item, for The number of times the data item has been compared, The total number of form data involved in the comparison. for The number of times a data item is marked as an error, For the The number of data items remaining uncompared, is the weight factor, is the adjustment factor;

[0021] The overall error probability formula for form data is: in, is the overall error probability of the form data, is the total number of data items in the form data, is the weight vector for each data item, is the comprehensive weight.

[0022] Preferably, based on the secondary management result, the multi-source data is managed three times, including: sorting all erroneous form data from small to large according to the error probability, obtaining the error probability of data items in N form data, and sorting them from large to small; using the AVL tree to construct an initial balanced management tree for the N form data, and balancing the error probability of each data item by adjusting the left and right subtrees; combining adjacent form data, and re-constructing the balanced management tree according to the comprehensive error probability, and finally forming an overall form data management balanced tree; replacing data items in different form data to further balance the left and right subtrees; checking whether the left and right subtrees of each node are balanced, using a greedy algorithm to select a reference standard for unbalanced nodes, re-comparing relevant data items, confirming their correctness and updating the error probability.

[0023] A multi-source data management system applied to a grassroots platform, further comprising:

[0024] The data acquisition and preprocessing module is responsible for acquiring multi-source data from the grassroots platform and preprocessing it; sorting the data by timestamp, dividing the data into form data 1 to N, and dividing the data items according to the spatial distribution order;

[0025] The mapping relationship building module builds a mapping relationship for form data comparison based on timestamps and data item distribution order; assigns unique serial numbers to N form data, gradually increases the number of extracted data items, and forms a sequence that needs to be compared;

[0026] The data comparison and preliminary management module compares the form data according to the mapping relationship, records the consistent and inconsistent data items, marks the completed and pending data items, and performs reverse verification on the uncompared data items;

[0027] The error probability calculation module uses a formula to calculate the error probability of inconsistent data items during the comparison process, evaluates the overall error probability of each data item and form data, and provides a basis for subsequent management and correction;

[0028] The balance management tree construction module builds the form data balance management tree based on the error probability of the data items;

[0029] The verification and correction module uses the balance management tree to check whether the nodes are balanced. For unbalanced nodes, it re-compares the relevant data items to confirm their actual correctness and updates the error probability.

[0030] Beneficial effects: The present invention uses efficient merge sort and bucket sort algorithms to pre-process and classify multi-source data according to timestamps and spatial distribution order, ensuring the orderliness and integrity of the data and laying a solid foundation for subsequent management. Then, in the mapping relationship construction stage, hash tables and dynamic programming techniques are used to not only achieve unique identification and orderly comparison of form data, but also optimize the data item extraction sequence, improve comparison efficiency, and enhance the flexibility and adaptability of the system. On this basis, the preliminary management stage records and marks consistent and inconsistent data items, especially marks problem data items, effectively identifies and marks data that require subsequent processing, and provides a clear direction for data correction. In the secondary management, Bayesian estimation and weighted average method are used to calculate the error probability, and the overall error rate of each data item and form data is quantitatively evaluated, providing a scientific basis for data correction and enhancing the accuracy and reliability of decision-making. Ultimately, the three-step management ensures the optimal consistency of each data item by constructing and adjusting the balanced management tree, and forms an effective feedback mechanism. By re-comparing unbalanced nodes, confirming the actual correctness of data items and updating error probabilities, it ensures the data consistency and accuracy of the entire system and significantly improves the overall performance and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the multi-source data management method applied to the grassroots platform of the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of the multi-source data management system applied to the grassroots platform of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1: A multi-source data management method applied to a grassroots platform, such as Figure 1 As shown, the following steps are included:

[0035] S1: Obtain multi-source data from the grassroots platform and perform preprocessing; divide the multi-source data into form data 1, form data 2, form data 3, ..., form data N according to the order of timestamps; obtain the number of data items in the form data, and divide the data items in the form data into data item 1, data item 2, data item 3, ..., data item M according to the order of data item distribution;

[0036] S2: constructing a mapping relationship for form data comparison based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item M;

[0037] S3: Based on the mapping relationship, perform preliminary management on multi-source data; based on the preliminary management results, perform secondary management on multi-source data; based on the secondary management results, perform tertiary management on multi-source data;

[0038] The preliminary management is to record and mark the consistent and inconsistent data items in the form data, and especially mark the inconsistent items;

[0039] The secondary management is to calculate the error probability of inconsistent data items, evaluate the overall error rate of each data item and form data, generate a detailed error analysis report, and guide subsequent data correction work;

[0040] The three managements are based on the error analysis report, by constructing and adjusting the balance management tree, re-comparing and confirming the inconsistent data items, and updating their status records according to the comparison results.

[0041] Acquire multi-source data from a grassroots platform and pre-process it; divide the multi-source data into form data 1, form data 2, form data 3, ..., form data N in order of timestamps; obtain the number of data items in the form data, and divide the data items in the form data into data item 1, data item 2, data item 3, ..., data item M in order of data item distribution, including: using merge sort in order of timestamps and in the order of time when the multi-source data is collected, divide the multi-source data into form data 1, form data 2, form data 3, ..., form data N, where N is the last form data collected in the multi-source data; using bucket sort in the order of data item distribution and in the order of spatial distribution of the data items in the form data, divide all data items in the form data into data item 1, data item 2, data item 3, ..., data item M, where M is the last data item in the form data.

[0042] To further illustrate, assume that there is a grassroots platform that receives multi-source data forms of residents from different departments (such as community service centers, civil affairs offices, and enterprises and institutions) every day. Each form data contains ID number, name, gender, and age. The acquisition and preprocessing steps are as follows: 1. Acquire and preprocess form data: 9:00 a.m.: Community Service Center. 2:00 p.m.: Civil Affairs Office. 7:00 p.m.: Enterprises and institutions. 2. Use merge sort to sort the form data according to the collection time to ensure that they are arranged in chronological order: Form data 1: Community Service Center records at 9:00 a.m. Form data 2: Civil Affairs Office records at 2:00 p.m. Form data 3: Enterprises and institutions records at 7:00 p.m. 3. Each form data contains four data items (ID number, name, gender, age). Use bucket sort to classify by their position in the form data to ensure that each data item is accurately extracted and managed. Merge sort ensures that the data is arranged in the order of collection, and bucket sort ensures that the data items are accurately classified.

[0043] Based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item N, a mapping relationship is constructed for form data comparison, including: using a hash table to assign a unique serial number to the N form data in the order of timestamps; using dynamic programming to assign M data items to the N form data for comparison according to the serial number of the form data; based on the number of assigned data items, sequentially forming a sequence to be compared; and comparing the N form data based on the constructed mapping relationship.

[0044] Further explanation is that, also taking the above-mentioned grassroots platform as an example, a mapping relationship is constructed to use a hash table to assign a unique serial number to each form data to ensure fast matching: Form data 1 (community service center) is assigned serial number 1, Form data 2 (civil affairs office) is assigned serial number 2, and Form data 3 (enterprises and institutions) is assigned serial number 3. Dynamic programming is used to gradually increase the number of extracted data items to form a contrast sequence: Form data 1 only extracts the ID number; Form data 2 extracts the ID number and name; Form data 3 extracts all data items (ID number, name, gender, age).

[0045] Further explanation is that the use of dynamic programming to allocate M data items for comparison to N form data according to the sequence number of the form data includes: by constructing a two-dimensional dynamic programming table dp[i][j], where i represents the i-th form data currently being processed, and j represents the number of data items that have been considered, the table is filled according to the best allocation method of the first j data items in the first i forms. During initialization, the initial conditions are set assuming that there are no data items (j=0) or the first form is processed (i=1). Then, the state transition equation is used to recursively calculate whether the data items between each form match, and if they do not match, the corresponding costs are recorded and the minimum cost path is selected. Finally, the optimal comparison sequence is obtained by backtracking from dp[N][M] to guide which data items between which form data should be compared first, thereby achieving efficient and accurate data comparison. In the dynamic programming process, dp[N][M] represents the best state or minimum cumulative error probability achieved when the N-th form data is processed and the first M data items have been considered. This value is obtained by step-by-step construction and optimization, ensuring that each step is based on the best previous decision.

[0046] Based on the constructed mapping relationship, N form data are compared, including:

[0047] First, for form data 1, only data item 1 is extracted; data item 1 in form data 1 is compared one-to-one with data item 1 in form data 2, and the comparison result is recorded;

[0048] Secondly, for form data 2, extract data item 1 and data item 2; after obtaining the comparison result of the previous step, compare data item 1 in form data 2 with data item 1 in form data 3; then, compare data item 2 in form data 2 with data item 2 in form data 3; repeat this process until the last form data N is processed. For form data N, extract all data items 1 to data item M; if form data N+1 exists, continue to compare the data items of form data N with the corresponding data items of form data N+1; if form data N+1 does not exist, directly enter the reverse verification stage; reversely verify the data items that have not been compared in all form data.

[0049] Further explanation is that, also taking the above-mentioned grassroots platform as an example, the comparison sequence is formed in sequence to form the sequence to be compared: Based on the number of allocated data items, the sequence to be compared is formed in sequence: Step 1: Compare the ID card number in form data 1 with the ID card number in form data 2. Step 2: Compare the ID card number and name in form data 2 with the corresponding data items in form data 3. Continue this process until all data items are compared.

[0050] Reverse verification is performed on all data items in the form data that have not been compared, including: starting from form data N, arranging in reverse order to form data N, form data N-1, ..., form data 1; and comparing the reversed form data in turn according to the same rules as above.

[0051] Further explanation is that, also taking the above-mentioned grassroots platform as an example, after completing the forward form data comparison, the uncompared data items need to be reversely verified. The specific steps are as follows: 1. Arrange the form data in reverse order according to the collection time: form data 3 (enterprise and institution records at 7:00 pm), form data 2 (civil affairs office records at 2:00 pm), form data 1 (community service center records at 9:00 am). 2. Use a hash table to assign a unique serial number to the reversed form data to ensure fast matching: form data 3 is assigned serial number 1, form data 2 is assigned serial number 2, and form data 1 is assigned serial number 3. 3. According to the reverse serial number, the number of extracted data items is gradually increased to form a comparison sequence: Form data 3 (serial number 1): only the ID card number is extracted. Form data 2 (serial number 2): extract the ID card number and name. Form data 1 (serial number 3): extract all data items (ID card number, name, gender, age). 4. Construct the sequence to be compared in sequence: Step 1: Compare the ID card number in form data 3 with form data 2. Step 2: Compare the ID card number and name in form data 2 with form data 1. Continue this process to ensure that all data items are fully compared. To further ensure that all data items in the form data are compared, in the reverse verification process, the data items in the N form data are first sorted in reverse order, and then these reversed data items are compared in turn according to the same rules. If there are still uncompared data items, they are compared separately with the corresponding data items that have been compared.

[0052] Based on the mapping relationship, the multi-source data is preliminarily managed, including: recording consistent and inconsistent data items during each comparison process; using a binary search tree to mark the data items that have been compared and the data items to be compared; and specially marking the data items with inconsistent comparison results.

[0053] A further explanation is that, based on the mapping relationship constructed previously, the data is initially managed to ensure that consistent and inconsistent data items are recorded during each comparison process, and a binary search tree is used to mark the completed and pending comparison data items. The specific steps are as follows: 1. Create a binary search tree, where nodes represent data items, and each node contains the following fields: Name: the identification name of the data item; Comparison status: whether the comparison has been completed (yes / no); Comparison result: comparison result (consistent / inconsistent). 2. The form data is sorted by timestamp, and the extracted data items are gradually added for comparison. 3. Update the binary search tree, mark the comparison status of each data item; especially mark inconsistent items. 4. Arrange the form data in reverse order, repeat the comparison process and update the status. 5. Confirm the status of each data item to ensure that all data items have been correctly marked and compared.

[0054] Based on the preliminary management results, secondary management of multi-source data is carried out, including:

[0055] When evaluating the error probability of inconsistent data items, the initial error probability is first calculated using the Bayesian estimation formula, and the first error rate is calculated based on the number of compared and remaining uncompared data items; then, the second error rate is calculated based on the number of related data items that have been compared in all previous form data and will be compared in the future; finally, the two error rates are combined and the final calculation is performed using the data item error probability formula and the form data overall error probability formula.

[0056] Further explanation is that the Bayesian estimation formula is used to calculate the initial error probability of inconsistent data items, taking into account prior knowledge (such as historical error rates) and currently observed data (such as inconsistencies found during the comparison process), providing a preliminary assessment as the basis for subsequent calculations. The first error rate: based on the number of compared and uncompared data items in the current form data, reflects the internal consistency of the form data and provides supplementary information for Bayesian estimation. The second error rate: based on the number of compared data items in all previous form data, provides a historical perspective to help evaluate the performance of the data item in the system. The data item error probability formula combines the Bayesian estimation results and the first and second error rates, comprehensively considers historical and potential errors, and flexibly adjusts the weight factors to calculate the specific error probability of each data item. The form data overall error probability formula summarizes the error probabilities of all data items, and obtains the overall error probability of the entire form data by weighted summation of the error probabilities of each data item, providing a comprehensive quality assessment.

[0057] Further explanation is that the initial error probability is first calculated using the Bayesian estimation formula, including:

[0058] Define events and prior probabilities: Let represents the event that M data items are marked as inconsistent; let It means that there are indeed wrong assumptions about M data items; is the prior probability that there are errors in the M data items;

[0059] Calculate the likelihood: represents the probability of being marked as inconsistent given that M data items have errors; It represents the probability that a given M data items are still incorrectly marked as inconsistent when there are no errors;

[0060] Calculate the posterior probability: Use Bayes' theorem to calculate the posterior probability , that is, the probability that an error actually exists given that M data items are marked as inconsistent: ,in, is the total probability that M data items are marked as inconsistent, calculated using the total probability formula: ;

[0061] Initialization and update: For every M newly discovered inconsistent data items, calculate the initial error probability according to the above formula .

[0062] The final calculation is performed using the data item error probability formula and the form data overall error probability formula, including: The data item error probability formula is, in, For the The error probability of a data item, for The number of times the data item has been compared, The total number of form data involved in the comparison. for The number of times a data item is marked as an error, For the The number of data items remaining uncompared, is the weight factor, is the adjustment factor;

[0063] The overall error probability formula for form data is: in, is the overall error probability of the form data, is the total number of data items in the form data, is the weight vector for each data item, is the comprehensive weight.

[0064] Further explanation is, Reflecting that in all comparisons, The probability that a data item is evaluated as being wrong. Refers to the specific number of times that the data item was found to be inconsistent or erroneous during all completed comparisons. Indicates how many times the data item participates in the comparison process and is used to calculate the average error rate. It represents the number of all form data involved in the comparison in the system and is the basis for evaluating the remaining uncompared data items. Indicates how many times the data item needs to be compared to estimate the future error rate. Used to adjust the importance of the second part of the formula to reflect the emphasis on potential errors in the uncompared data items. Used to fine-tune the results of the formula so that the final calculated error probability is more in line with the actual situation, for example, according to the specific needs of the system or the results of historical data analysis. Refers to the combined error probability of all data items in the entire form data set, providing an overall evaluation of the quality of the form data. Indicates the number of all independent data items contained in a form data. Each data item is given a different weight, which is determined based on the importance and impact of the data item, thereby ensuring that important data items have a greater influence in the overall evaluation. It is a global weight factor used to further adjust the error probability assessment results of the entire form data. It is set based on the strategy or policy at the system level. The relationship and role of parameters, the design of the data item error probability formula is to use the existing comparison results (i.e. ) and future miscalculations (i.e. ), giving an accurate data item error probability assessment. The overall error probability formula for form data aggregates the error probabilities of all data items and weights them together through their respective weight vectors and comprehensive weights to produce a comprehensive and quantitative form data quality indicator. This two-layer structure not only takes into account the error conditions of individual data items, but also evaluates the quality of form data as a whole, providing a solid foundation for subsequent data management and correction.

[0065] Based on the secondary management results, the multi-source data is managed three times, including: sorting all erroneous form data from small to large according to the error probability, obtaining the error probability of data items in N form data, and sorting them from large to small; using the AVL tree to build an initial balanced management tree for the N form data, and balancing the error probability of each data item by adjusting the left and right subtrees; combining adjacent form data, and re-building the balanced management tree according to the comprehensive error probability, and finally forming an overall form data management balanced tree; replacing data items in different form data to further balance the left and right subtrees; checking whether the left and right subtrees of each node are balanced, using a greedy algorithm to select a reference standard for unbalanced nodes, re-comparing relevant data items, confirming their correctness and updating the error probability.

[0066] Further explanation is that, also taking the grassroots platform as an example, the form data is sorted according to the error probability and a balanced management tree is constructed to ensure data consistency and accuracy. The specific steps are as follows: 1. Sort form data: Form data 3 (enterprise and institution records): The "gender" data item is inconsistent, and the error probability is 1. Form data 2 (Civil Affairs Office records): No obvious inconsistency, low error probability. Form data 1 (Community Service Center records): No obvious inconsistency, low error probability. Sort by error probability from small to large: Form data 1, Form data 2, Form data 3. 2. Get and sort the error probability of data items: Form data 3: ID number (0), name (0), gender (1), age (0). Sort by error probability from large to small: Gender (1), ID number (0), name (0), age (0). 3. Construct the initial AVL tree: Construct an initial balanced management tree for each form data according to the error probability of each data item, paying special attention to data items with high error probability (such as "gender"). 4. Adjust subtrees: Adjust the unbalanced left and right subtrees through rotation operations to ensure that the tree structure remains balanced. 5. Combine and rebuild: Combine adjacent form data, reconstruct the balanced management tree based on the comprehensive error probability, and update the AVL tree. 6. Summarize the overall tree: Summarize the balanced management trees constructed by all form data into an overall form data management balanced tree to ensure system consistency. 7. Check and adjust: Use a greedy algorithm to ensure that the height difference between the left and right subtrees of each node does not exceed 1, and re-compare and update the error probability for unbalanced nodes.

[0067] A further explanation is that the use of a greedy algorithm to select a reference standard for unbalanced nodes includes: for each unbalanced node, first consider the error probability of all data items in the left and right subtrees, and select data items with low error probability and close logical association with other data items as reference standards. Then, based on the selected reference standard, compare and correct inconsistent data items one by one, and only select the current optimal data item for correction each time, in the hope of rebalancing the node and reducing the overall error probability with the least number of operations. After completing a correction, check again whether the node and subtree are still unbalanced, and repeat the above steps until all nodes are balanced again. This process not only ensures the high balance of the AVL tree, but also significantly improves the consistency and reliability of the data through iterative optimization.

[0068] Embodiment 2: Based on embodiment 1, a multi-source data management system applied to a grassroots platform includes:

[0069] The data acquisition and preprocessing module is responsible for acquiring multi-source data from the grassroots platform and preprocessing it; sorting the data by timestamp, dividing the data into form data 1 to N, and dividing the data items according to the spatial distribution order;

[0070] The mapping relationship building module builds a mapping relationship for form data comparison based on timestamps and data item distribution order; assigns unique serial numbers to N form data, gradually increases the number of extracted data items, and forms a sequence that needs to be compared;

[0071] The data comparison and preliminary management module compares the form data according to the mapping relationship, records the consistent and inconsistent data items, marks the completed and pending data items, and performs reverse verification on the uncompared data items;

[0072] The error probability calculation module uses a formula to calculate the error probability of inconsistent data items during the comparison process, evaluates the overall error probability of each data item and form data, and provides a basis for subsequent management and correction;

[0073] The balance management tree construction module builds the form data balance management tree based on the error probability of the data items;

[0074] The verification and correction module uses the balance management tree to check whether the nodes are balanced. For unbalanced nodes, it re-compares the relevant data items to confirm their actual correctness and updates the error probability.

[0075] The above is a detailed introduction to the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-source data management method applied to a grassroots platform, characterized in that: The following steps are involved: S1: Obtain multi-source data from the grassroots platform and perform preprocessing; divide the multi-source data into form data 1, form data 2, form data 3, ..., form data N according to the order of timestamps; obtain the number of data items in the form data, and divide the data items in the form data into data item 1, data item 2, data item 3, ..., data item M according to the order of data item distribution; S2: Based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item M, a mapping relationship is constructed for form data comparison; based on the constructed mapping relationship, the N form data are compared; S3: Based on the mapping relationship, perform preliminary management of multi-source data; based on the preliminary management results, perform secondary management of multi-source data; Based on the secondary management results, perform tertiary management on multi-source data; The preliminary management is to record and mark the consistent and inconsistent data items in the form data, and especially mark the inconsistent items; The secondary management is to calculate the error probability of inconsistent data items, evaluate the overall error rate of each data item and form data, generate a detailed error analysis report, and guide subsequent data correction work; The three managements are based on the error analysis report, by constructing and adjusting the balance management tree, re-comparing and confirming the inconsistent data items, and updating their status records according to the comparison results; Based on the secondary management result, the multi-source data is managed three times, including: sorting the overall error probability of all form data from small to large, obtaining the error probability of data items in N form data, and sorting them from large to small; Use AVL tree to build an initial balanced management tree for N form data, and balance the error probability of each data item by adjusting the left and right subtrees; combine adjacent form data, and build a balanced management tree again according to the comprehensive error probability, and finally form an overall form data management balanced tree; replace data items in different form data to further balance the left and right subtrees; check whether the left and right subtrees of each node are balanced, use a greedy algorithm to select a reference standard for unbalanced nodes, re-compare relevant data items, confirm their correctness and update the error probability.

2. The multi-source data management method applied to a grassroots platform according to claim 1, characterized in that: The method comprises: obtaining multi-source data of a grassroots platform and preprocessing the data; dividing the multi-source data into form data 1, form data 2, form data 3, ..., form data N according to the order of timestamps; obtaining the number of data items in the form data, and dividing the data items in the form data into data item 1, data item 2, data item 3, ..., data item M according to the order of distribution of the data items, including: using merge sort according to the order of timestamps and the order of time in which the multi-source data is collected, dividing the multi-source data into form data 1, form data 2, form data 3, ..., form data N, where N is the last form data collected in the multi-source data; using bucket sort according to the order of spatial distribution of the data items in the form data, dividing all data items in the form data into data item 1, data item 2, data item 3, ..., data item M, where M is the last data item in the form data.

3. The multi-source data management method applied to a grassroots platform according to claim 1, characterized in that: The method of constructing a mapping relationship for form data comparison based on form data 1, form data 2, form data 3, ..., form data N and data item 1, data item 2, data item 3, ..., data item M includes: using a hash table to assign a unique serial number to the N form data in the order of timestamps; using dynamic programming to assign M data items to the N form data for comparison according to the serial number of the form data; and sequentially forming a sequence to be compared based on the number of assigned data items.

4. The multi-source data management method applied to a grassroots platform according to claim 1, characterized in that: The N form data are compared based on the constructed mapping relationship, including: First, for form data 1, only data item 1 is extracted; data item 1 in form data 1 is compared one-to-one with data item 1 in form data 2, and the comparison result is recorded; Secondly, for form data 2, extract data item 1 and data item 2; after obtaining the comparison result of the previous step, compare data item 1 in form data 2 with data item 1 in form data 3; then, compare data item 2 in form data 2 with data item 2 in form data 3; repeat this process until the last form data N is processed. For form data N, extract all data items 1 to data item N; if form data N+1 exists, continue to compare the data items of form data N with the corresponding data items of form data N+1; if form data N+1 does not exist, directly enter the reverse verification stage; reversely verify the data items that have not been compared in all form data.

5. The multi-source data management method applied to a grassroots platform according to claim 4 is characterized in that: The reverse verification of the uncompared data items in all form data includes: starting from form data N, arranging in reverse order as form data N, form data N-1, ..., form data 1; and comparing the reversed form data in turn according to the same rules as above.

6. The multi-source data management method applied to a grassroots platform according to claim 1, characterized in that: The multi-source data is initially managed based on the mapping relationship, including: recording consistent and inconsistent data items during each comparison process; using a binary search tree to mark the data items that have been compared and the data items to be compared; and specially marking the data items with inconsistent comparison results.

7. The multi-source data management method applied to a grassroots platform according to claim 1, characterized in that: Based on the preliminary management results, the multi-source data is secondary managed, including: When evaluating the error probability of inconsistent data items, the initial error probability is first calculated using the Bayesian estimation formula, and the first error rate is calculated based on the number of data items that have been compared and the number of data items that remain to be compared. Then, the second error rate is calculated based on the number of related data items that have been compared in all previous form data and will be compared in the future. Finally, the two error rates are combined and the final calculation is performed using the data item error probability formula and the form data overall error probability formula. The formula for the data item error probability is: Among them, P err,i is the error probability of the ith data item, C i is the number of times the i data item has been compared, T is the total number of form data involved in the comparison, and E err,i is the number of times the i data item is marked as an error, R i =TC i is the number of remaining uncompared times of the i-th data item, W is the weight factor, and A is the adjustment factor; The overall error probability formula for form data is: Among them, P err,form is the overall error probability of the form data, M is the total number of data items in the form data, and W i is the weight vector for each data item, W total is the comprehensive weight.

8. A multi-source data management system applied to a grassroots platform, used to implement a multi-source data management method applied to a grassroots platform as described in any one of claims 1 to 7, characterized in that: Included are: The data acquisition and preprocessing module is responsible for acquiring multi-source data from the grassroots platform and preprocessing it; sorting the data by timestamp, dividing the data into form data 1 to N, and dividing the data items according to the spatial distribution order; The mapping relationship building module builds a mapping relationship for form data comparison based on timestamps and data item distribution order; assigns unique serial numbers to N form data, gradually increases the number of extracted data items, and forms a sequence that needs to be compared; The data comparison and preliminary management module compares the form data according to the mapping relationship, records the consistent and inconsistent data items, marks the completed and pending data items, and performs reverse verification on the uncompared data items; The error probability calculation module uses the data item error probability formula to calculate the error probability of inconsistent data items during the comparison process, evaluates the overall error probability of each data item and form data, and provides a basis for subsequent management and correction; The balance management tree construction module builds the form data balance management tree based on the error probability of the data items; The verification and correction module uses the balance management tree to check whether the nodes are balanced. For unbalanced nodes, it re-compares the relevant data items to confirm their actual correctness and updates the error probability.

Images