A method for batch capturing target data during mass data synchronization

By performing attribute identification and feature evaluation of data sources, dynamically adjusting the crawling strategies and tasks, identifying and resolving conflict points, solving the problems of data inconsistency and high resource consumption in the existing technology, and achieving efficient and accurate data synchronization of massive data Crawl.

CN118673034BActive Publication Date: 2024-11-29SICHUAN CAIZI SOFTWARE INFORMATION NETWORK CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411162493.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-29
Estimated Expiration
2044-08-23

Smart Images

  • Figure CN118673034B_ABST
    Figure CN118673034B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data capture technology, specifically a method for batch capturing target data during the synchronization of massive data, comprising the following steps: identifying the attributes of a data source, analyzing the data update frequency, data volume, and data type heterogeneity, classifying the data source, and calculating a feature vector to generate a data source characteristic evaluation result. In the present invention, by identifying the key attributes of the data source and analyzing the impact of the attributes on data synchronization, conflicts in data updates are more accurately predicted and processed. During data synchronization and batch capture, task scheduling and capture strategies are dynamically adjusted to ensure data accuracy and timely updates, effectively reduce resource consumption, and increase the speed of data processing. Potential conflicts are identified by simulating the data synchronization process, and risks are effectively avoided. This is of significant significance for maintaining data consistency and stability, and further ensures high efficiency and quality of data capture.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data capture, and in particular to a method for batch capturing target data during mass data synchronization. Background Art

[0002] The field of data scraping technology focuses on methods and tools for automatically retrieving information from various data sources, which can be web pages, databases, APIs, or data-rich platforms. Data scraping technology involves automated scripts or software that can quickly and accurately extract and store large amounts of data. Key technologies include web crawlers, content parsers, data formatting, and error handling mechanisms. Data scraping plays a vital role in business intelligence, market analysis, the construction of machine learning datasets, and automated monitoring. With the advancement of technology, data scraping tools are also evolving to cope with the challenges of diversification of data sources and anti-scraping technologies.

[0003] Among them, the method of batch crawling target data during the massive data synchronization process involves realizing efficient batch data extraction in a specific data synchronization environment. The main purpose is to improve the efficiency of data crawling in a multi-data source environment and ensure the accuracy and timely update of data. Application scenarios include real-time update of data warehouses, multi-source data integration, and improving the speed of data support for enterprise decision-making. By optimizing the crawling strategy and synchronization mechanism, it is possible to reduce the consumption of system resources and increase the speed of data processing while ensuring data quality.

[0004] Existing data crawling technologies rely on fixed data crawling strategies and synchronization mechanisms, which limit their adaptability and efficiency in processing multiple data sources. Especially in environments with large data volumes and frequent updates, fixed strategies lead to data inconsistency and redundant crawling tasks, increasing the burden and reducing operational efficiency. For example, in a system without dynamic adjustment of crawling tasks, large-scale data updates will trigger unnecessary crawling operations, consuming a large amount of network and storage resources, and the lack of error handling mechanisms will lead to error accumulation, affecting the accuracy and reliability of data. Existing technologies are not sufficient to provide optimal performance and resource management in big data and rapidly changing data environments. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a method for batch capturing target data during the synchronization of massive data.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a method for batch capturing target data during mass data synchronization, comprising the following steps:

[0007] S1: Identify the attributes of data sources, analyze data update frequency, data volume and data type heterogeneity, classify data sources, calculate feature vectors, and generate data source characteristic evaluation results;

[0008] S2: Based on the data source characteristic evaluation result, analyze the batch data capture path in the massive data synchronization process, sort the data capture paths according to priority, and generate a data capture strategy;

[0009] S3: through the data capture strategy, set the operating parameters of the task scheduler, dynamically monitor the data capture task, dynamically adjust the capture task according to data changes, and generate a task scheduling record;

[0010] S4: using the task scheduling record, simulating the data update process in the massive data synchronization process, identifying the conflict points of batch capture of target data, avoiding potential risks, and building a data conflict model;

[0011] S5: According to the data conflict model, data at the conflict point is rolled back, a solution strategy is formulated for the conflict point, and the strategy is implemented to avoid data conflicts and generate a conflict resolution solution;

[0012] S6: Based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data receiver, and the data summary push result is obtained.

[0013] As a further solution of the present invention, the data source characteristic evaluation results include data update frequency analysis results, data volume statistics results and data type heterogeneity analysis results; the data crawling strategy includes data crawling path priority sorting, resource allocation optimization indicators and crawling task time plan; the task scheduling record includes task execution time log, task status update record and resource usage statistics; the data conflict model includes conflict point identification, potential risk analysis and prevention strategy plan; the conflict resolution solution includes data rollback, implementation of resolution strategy and evaluation of strategy execution effect; the data summary push result includes data consistency verification result, data format unification status and data push status.

[0014] As a further solution of the present invention, the steps of identifying the attributes of the data source, analyzing the data update frequency, data volume and data type heterogeneity, classifying the data source, and calculating the feature vector to generate the data source characteristic evaluation result are specifically as follows:

[0015] S101: Identify the attributes of the data source, measure the update frequency of the massive data during the massive data synchronization process, record the volume of the data source and analyze the type of the massive data, and generate data attribute records;

[0016] S102: Based on the data attribute records, the massive data is classified, the massive data is grouped according to update frequency, volume and type, a category is marked for each data source, the efficiency of massive data management is checked, and a data source classification result is generated;

[0017] S103: According to the data source classification result, characteristic vectors of differentiated massive data are calculated, central tendency and dispersion of each type of massive data are calculated, and data source characteristic evaluation results are obtained by optimizing the massive data synchronization process.

[0018] As a further solution of the present invention, based on the data source characteristic evaluation result, the batch data capture path in the massive data synchronization process is analyzed, the data capture paths are sorted according to priority, and the steps of generating the data capture strategy are specifically as follows:

[0019] S201: Based on the data source characteristic evaluation result, a decision tree algorithm is used to identify key attributes of the data source, evaluate the impact of the key attributes on the priority of batch crawling target data, and generate a key attribute list;

[0020] S202: Analyze the dependencies and connectivity between data sources using the key attribute list, calculate the optimal path in the process of mass data synchronization, avoid data transmission delay, and generate data capture path planning;

[0021] S203: According to the data capture path planning, the paths for batch capturing target data are sorted according to priority, the execution order and resource allocation of each path are set, the efficiency and response speed in the massive data synchronization process are verified, and the data capture strategy is obtained.

[0022] As a further solution of the present invention, the formula of the decision tree algorithm is as follows:

[0023]

[0024] in, is the Gini impurity, For Category The proportion of data items , and is the weight coefficient, For Category The frequency of n1 is the number of categories, For data items The distribution density of For Category The stability of the samples.

[0025] As a further solution of the present invention, the data capture strategy is used to set the operating parameters of the task scheduler, dynamically monitor the data capture task, and dynamically adjust the capture task according to data changes. The steps of generating task scheduling records are specifically as follows:

[0026] S301: Based on the data capture strategy, set the operating parameters of the task scheduler, analyze the task priority, resource allocation and time interval, check the response of the task scheduler to data capture and manage data capture requirements, and generate a scheduling parameter configuration record;

[0027] S302: using the scheduling parameter configuration record, dynamically monitoring the batch capture target data task in the massive data synchronization process, mining and processing anomalies and deviations in task execution by real-time tracking of task progress and resource usage status, and generating real-time monitoring records;

[0028] S303: According to the real-time monitoring record, dynamically adjust the execution strategy of the data capture task, including reordering of task priorities and reallocation of resources, match the real-time changes in massive data, and obtain task scheduling records.

[0029] As a further solution of the present invention, the task scheduling record is used to simulate the data update process in the massive data synchronization process, identify the conflict points of batch capture target data, avoid potential risks, and construct the data conflict model in the following steps:

[0030] S401: Based on the task scheduling record, simulate the data update in the massive data synchronization process, identify the dynamics of the data capture environment and potential synchronization problems by analyzing the data flow and task execution sequence, and generate data update simulation results;

[0031] S402: using the data update simulation results, adopting a conflict detection algorithm, identifying conflict points of batch-capturing target data during the massive data synchronization process, performing conflict point analysis on data versions and resource changes, and generating conflict point identification records;

[0032] S403: According to the conflict point identification records, conflict resolution strategies and preventive measures are designed, data rollback and priority adjustment are performed, conflicts in the massive data synchronization process are automatically processed and resolved, data consistency is checked, and a data conflict model is obtained.

[0033] As a further solution of the present invention, the formula of the conflict detection algorithm is as follows:

[0034]

[0035] in, For data elements and The degree of conflict between is the number of data updates, and Data elements and In the The state value at the time of the update, is the weight coefficient, and External factors that affect data updates and data access frequency.

[0036] As a further solution of the present invention, according to the data conflict model, data rollback of the conflict point is performed, a solution strategy is formulated for the conflict point, and the strategy is implemented to avoid data conflicts. The steps of generating a conflict solution are specifically as follows:

[0037] S501: Based on the data conflict model, data rollback is implemented, and the consistency and synchronization efficiency of data are verified by adjusting the affected massive data points to a stable state, avoiding data synchronization errors, and generating data rollback results;

[0038] S502: Utilizing the data rollback result, a resolution plan is formulated for the conflict types in the massive data synchronization process, and by implementing priority reordering and resource reconfiguration measures, the impact of conflicts in the future time period is avoided, and a conflict type analysis result is generated;

[0039] S503: According to the conflict type analysis result, a conflict resolution strategy is constructed and the execution effect is monitored. Through dynamic adjustment and effect verification of the conflict resolution strategy, massive data synchronization is performed to obtain a conflict resolution solution.

[0040] As a further solution of the present invention, based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data receiver, and the steps of obtaining the data summary push result are specifically as follows:

[0041] S601: Based on the conflict resolution method, the captured batch target data is integrated, and the synchronization and consistency of the information of the data sets are checked by merging the target data sets to generate a data integration result;

[0042] S602: Using the data integration result, perform verification of mass data synchronization, verify the consistency and integrity of the data, analyze the efficiency of batch capture of target data by comparing differentiated data source records, and generate data consistency verification records;

[0043] S603: Unify the format of the massive data according to the data consistency verification record and push it to the data recipient. Verify the recipient's use of the target data through format standardization and data transmission to obtain a data summary push result.

[0044] Compared with the prior art, the advantages and positive effects of the present invention are:

[0045] In the present invention, by identifying the key attributes of the data source and analyzing the impact of the attributes on data synchronization, it is allowed to more accurately predict and handle conflicts in data updates, and the data capture efficiency is significantly improved through attribute analysis and characteristic vector calculation. During data synchronization and batch capture, task scheduling and capture strategies are dynamically adjusted according to the evaluation results of data source characteristics to ensure data accuracy and timely update. Dynamic adjustment strategies effectively reduce resource consumption and increase the speed of data processing. Potential conflicts are identified by simulating the data synchronization process, and a data conflict model is constructed to effectively avoid risks, which is of great significance for maintaining data consistency and stability. The rapid resolution strategy for data conflicts further ensures the high efficiency and quality of data capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0047] Figure 2 It is a schematic diagram of the refinement of S1 of the present invention;

[0048] Figure 3 It is a schematic diagram of the refinement of S2 of the present invention;

[0049] Figure 4 It is a schematic diagram of the refinement of S3 of the present invention;

[0050] Figure 5 It is a schematic diagram of the refinement of S4 of the present invention;

[0051] Figure 6 It is a detailed schematic diagram of S5 of the present invention;

[0052] Figure 7 It is a detailed schematic diagram of S6 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0055] Embodiment 1

[0056] See also Figure 1 The present invention provides a technical solution, a method for batch capturing target data during mass data synchronization, comprising the following steps:

[0057] S1: Identify the attributes of massive data sources, analyze the data update frequency, data volume and data type heterogeneity during the massive data synchronization process, classify the data sources, calculate the feature vector for each category, and generate the data source characteristic evaluation results;

[0058] S2: Based on the data source characteristic evaluation results, analyze the batch data capture path in the massive data synchronization process, sort the batch capture target data paths according to priority, formulate resource allocation plans, and generate data capture strategies;

[0059] S3: Set the operating parameters of the task scheduler through the data capture strategy, start task scheduling, dynamically monitor the task of batch capture of target data, dynamically adjust the capture tasks according to data changes, and generate task scheduling records;

[0060] S4: Use task scheduling records to simulate the data update process during massive data synchronization, identify conflict points in batch capture of target data, and predict the path of target data capture to avoid potential risks and build a data conflict model;

[0061] S5: According to the data conflict model, roll back the data at the conflict point, merge the versions, re-order the priority of the target data crawling tasks, formulate a solution strategy for the conflict point, and implement the strategy to avoid data conflicts and generate a conflict resolution solution;

[0062] S6: Based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data receiver, and the data summary push result is obtained.

[0063] The data source characteristic assessment results include data update frequency analysis results, data volume statistics results and data type heterogeneity analysis results. The data crawling strategy includes data crawling path priority sorting, resource allocation optimization indicators and crawling task time plan. Task scheduling records include task execution time logs, task status update records and resource usage statistics. The data conflict model includes conflict point identification, potential risk analysis and prevention strategy plan. The conflict resolution method includes data rollback, implementation of resolution strategy and evaluation of strategy execution effect. The data summary push results include data consistency verification results, data format unification status and data push status.

[0064] See also Figure 2 , identify the attributes of the data source, analyze the data update frequency, data volume and data type heterogeneity, classify the data source, and calculate the feature vector. The specific steps to generate the data source characteristic evaluation results are as follows:

[0065] S101: Identify the attributes of the data source. During the synchronization of massive data, measure the update frequency of the massive data, record the volume of the data source and analyze the type of the massive data. The execution flow of generating data attribute records is as follows;

[0066] Sub-step S101 identifies the attributes of the data source, performs a preliminary scan of the massive data, obtains the basic information of each data source, including data size, data format and last modification time, analyzes the update frequency of each data block, and uses timestamp comparison to obtain the update cycle. Evaluates the volume of the data source, which is calculated by accumulating the size of the data block. Classifies the types of massive data, such as text, images or videos, and identifies them through file information and internal structure, records the volume of the data source and analyzes the type of massive data, generates data attribute records, and uses the formula:

[0067]

[0068] in, represents the average update frequency, It is n2 is the update interval of each data source, and n2 is the number of data sources.

[0069] S102: Based on the data attribute records, the massive data is classified, the massive data is grouped according to the update frequency, volume and type, a category is marked for each data source, the efficiency of the massive data management is checked, and the execution process of generating the data source classification result is as follows;

[0070] Sub-step S102 builds a data processing process based on data attribute records. The process includes data grouping and tagging, extracting update frequency, volume and type information from data attribute records, and using a grouping algorithm to classify massive data. For example, data is divided into three levels: high, medium and low according to update frequency, and divided into three categories: large, medium and small according to volume size, and further subdivided by type. A unique identifier is assigned to each data source for tracking and management. Through a series of performance tests, such as access speed and response time, the management efficiency of each type of data is evaluated, and the data source classification results are generated. The formula used is:

[0071]

[0072] Among them, G1 represents the grouping standard score, It is The weight coefficient of the class data, is the number of data in the dataset, is the number of data categories.

[0073] S103: According to the data source classification result, the characteristic vector of the differentiated massive data is calculated, the central tendency and dispersion of each type of massive data are calculated, and the execution process of obtaining the data source characteristic evaluation result by optimizing the massive data synchronization process is as follows;

[0074] In sub-step S103, based on the data source classification results, a feature vector generation algorithm is designed. The algorithm extracts key features by analyzing the central tendency and dispersion of each type of data, calculates the average update time, average size, and common data types of each category of data, and uses statistical values ​​to represent the central tendency. The standard deviation and coefficient of variation are calculated to describe the dispersion of the data. The feature vector includes statistical indicators to form a comprehensive data description. Through the feature vector, the data synchronization process is optimized, such as adjusting the synchronization frequency and batch size to improve efficiency, and the data source characteristic evaluation results are generated. The formula used is:

[0075]

[0076] in, represents the variance of the data, It is The value of the data point, is the mean value and m1 is the total number of data points.

[0077] See also Figure 3 ,Based on the data source characteristic evaluation results, the batch data capture paths in the massive data synchronization process are analyzed, and the data capture paths are sorted according to priority. The specific steps for generating data capture strategies are as follows:

[0078] S201: Based on the data source characteristic evaluation results, a decision tree algorithm is used to identify key attributes of the data source, and the impact of the key attributes on the priority of batch crawling target data is evaluated. The execution process of generating a key attribute list is as follows;

[0079] Sub-step S201 identifies key attributes in the data source based on the data source characteristic evaluation results, and uses a decision tree algorithm to determine the attributes that have the greatest impact on the priority of data crawling by analyzing the evaluation results such as the update frequency, size and type of the data source. The identification process involves building multiple decision tree models and scoring each attribute based on the ability to influence batch data crawling. The scoring of key attributes will be based on the contribution of the attribute to improving data crawling efficiency, including its role in reducing crawling time and improving data quality. After the scores of each attribute are aggregated, a list of key attributes is formed.

[0080] The formula for the decision tree algorithm is as follows:

[0081]

[0082] in, is the Gini impurity, For Category The proportion of data items , and is the weight coefficient, For Category The frequency of n1 is the number of categories, For data items The distribution density of For Category The stability of the samples.

[0083] The execution flow is as follows:

[0084] Calculate the proportion of data items of each category in each segmentation subset , introducing additional feature parameters for each category, category frequency , distribution density and sample stability , the parameters are obtained through the statistical information of the existing data set, and the parameters are compared with Combine, calculate weighted, weight coefficient , and The Gini impurity of each attribute is calculated through preprocessing and cross-validation of the data set. , select The smallest attribute is used as the node splitting attribute.

[0085] S202: Analyze the dependencies and connectivity between data sources using the key attribute list, calculate the optimal path in the process of mass data synchronization, avoid data transmission delay, and generate the execution process of data capture path planning as follows;

[0086] Sub-step S202 uses the key attribute list to establish a model of dependencies and connectivity between data sources. The model uses graph theory to treat each data source as a node in the graph and the dependencies between data sources as edges. A weight is assigned to each edge based on the weight of the key attribute, and the weight reflects the transmission priority and necessity in data synchronization. Using the model, the shortest path algorithm is applied to calculate the optimal transmission path from the source point to each data node, avoiding potential data transmission delays and generating data capture path planning. The formula used is:

[0087]

[0088] in, represents the total cost of the optimal data crawling path, Is a node To Node The connection cost, is the priority weight of the corresponding path, is the total number of nodes in the path.

[0089] S203: According to the data capture path planning, the paths for batch capturing target data are sorted according to priority, the execution order and resource allocation of each path are set, the efficiency and response speed in the massive data synchronization process are verified, and the execution process of the data capture strategy is obtained as follows;

[0090] Sub-step S203 performs batch data priority sorting and resource allocation based on data capture path planning. Based on the path planning results, each data path is prioritized according to the importance of key attributes and the urgency of data capture, and necessary network bandwidth and storage resources are allocated to each path to ensure that high-priority data paths can respond quickly. Implement simulation tests, evaluate the effects of various resource allocation strategies by simulating different data synchronization scenarios, and obtain data capture strategies using the formula:

[0091]

[0092] in, Indicates the overall response speed during synchronization. It is The resource allocation coefficient of the path, is the priority score of the path, is the total number of paths considered.

[0093] See also Figure 4, through the data capture strategy, set the operating parameters of the task scheduler, dynamically monitor the data capture task, and dynamically adjust the capture task according to data changes. The specific steps to generate task scheduling records are:

[0094] S301: Based on the data capture strategy, set the operating parameters of the task scheduler, analyze the task priority, resource allocation and time interval, check the response of the task scheduler to data capture and manage data capture requirements, and generate the execution process of the scheduling parameter configuration record as follows;

[0095] Sub-step S301 determines the core operating parameters of the task scheduler based on the data capture strategy, including task priority, resource allocation, and time interval for task execution. By analyzing the priority and resource requirements determined in the data capture strategy, the scheduler is configured to respond to the requirements in the best way. For example, high-priority tasks will be allocated more computing resources and shorter time intervals. Use simulation tests to evaluate the impact of parameter configuration on task execution efficiency, ensure that the scheduler can effectively manage resources in a multi-tasking environment, and generate scheduling parameter configuration records. The formula used is:

[0096]

[0097] in, Represents the overall scheduling efficiency of the task scheduler, It is The priority of each task, is the amount of resources allocated to the task, is the time interval between task executions, and n3 is the total number of tasks.

[0098] S302: Use the scheduling parameter configuration record to dynamically monitor the batch capture target data task in the massive data synchronization process, and mine and process the anomalies and deviations in task execution by real-time tracking of task progress and resource usage status. The execution process of generating real-time monitoring records is as follows;

[0099] Sub-step S302 uses scheduling parameter configuration records to set up dynamic monitoring equipment to track all tasks in the massive data synchronization process, and quickly identify any anomalies or deviations in execution by analyzing task progress and resource usage status in real time. This includes resource exhaustion, progress delays, and data quality issues. The problems found are analyzed immediately, and task parameters are adjusted or resources are reallocated according to the nature of the problem. Real-time monitoring records are generated using the formula:

[0100]

[0101] in, represents the total anomaly score, It is The deviation level of each task, is the sensitivity coefficient of the task, and m3 is the total number of monitored tasks.

[0102] S303: According to the real-time monitoring records, dynamically adjust the execution strategy of the data capture task, including re-ordering of task priorities and reallocation of resources, to match the real-time changes in massive data, and obtain the execution process of the task scheduling record as follows;

[0103] Substep S303 dynamically adjusts the execution strategy of the data crawling task based on the real-time monitoring records. This involves reordering the task priorities and reallocating resources based on the real-time data and task performance. Through the data analysis module, the performance and resource utilization efficiency of each task are evaluated to determine which tasks require more resources or priority. The adjusted task strategy will be implemented immediately to ensure that the data crawling process can adapt to the rapidly changing data environment and obtain the task scheduling record. The formula used is:

[0104]

[0105] in, represents the adjusted resource utilization, It is The adjusted priority of each task, is the new amount of resources assigned to the task, is the total number of tasks, is the total available resources.

[0106] See also Figure 5 ,Using task scheduling records, simulate the data update process in the process of massive data synchronization, identify the conflict points of batch capture target data, avoid potential risks, and the steps of building a data conflict model are as follows:

[0107] S401: Based on the task scheduling record, simulate the data update in the massive data synchronization process, identify the dynamics of the data capture environment and potential synchronization problems by analyzing the data flow and task execution sequence, and generate the execution process of the data update simulation result as follows;

[0108] Based on the task scheduling records, sub-step S401 sets up data synchronization simulation equipment to simulate the update process of massive data. It reproduces the data flow and task execution sequence through virtualization technology, which can reflect the dynamic changes of the data capture environment in real time and predict and identify data synchronization problems, such as data delay, packet loss or data inconsistency. The data collected during the simulation will be used to analyze the efficiency of data flow and the effectiveness of synchronization strategies. The analysis results will be recorded in detail and the data update simulation results will be generated using the formula:

[0109]

[0110] in, Represents data update efficiency, It is The amount of data updated by each data task, is the time required to complete the task, and n4 is the total number of tasks in the simulation.

[0111] S402: Using the data update simulation results, adopting the conflict detection algorithm, identifying the conflict points of batch grabbing target data during the massive data synchronization process, performing conflict point analysis of data versions and resource changes, and generating conflict point identification records. The execution process is as follows;

[0112] Substep S402 uses the data update simulation results to apply a conflict detection algorithm to identify data conflict points that occur during the synchronization process. The algorithm determines that data will cause synchronization conflicts by analyzing the differences in data versions and changes in resource allocation. For example, when two or more tasks attempt to access and modify the same data item at the same time, the operation will be marked as a potential conflict point. For each conflict point, the cause and impact will be analyzed and a conflict point identification record will be generated.

[0113] The formula for the conflict detection algorithm is as follows:

[0114]

[0115] in, For data elements and The degree of conflict between is the number of data updates, and Data elements and In the The state value at the time of the update, is the weight coefficient, and External factors that affect data updates and data access frequency.

[0116] The execution flow is as follows:

[0117] Collecting Data Elements and Updated status value and , considering the external influencing factors of the two data sources and , such as network delay or access frequency, introduce a weight coefficient , by adjusting The size of the original state value is used to balance the influence of external factors, calculate the vector length, and obtain the degree of conflict through calculation , weight coefficient The determination of can be done through optimization algorithms, such as genetic algorithms or gradient descent methods, and dynamically adjusted according to actual data to ensure the accuracy of the calculation results.

[0118] S403: According to the conflict point identification records, conflict resolution strategies and preventive measures are designed, data rollback and priority adjustment are performed, conflicts in the massive data synchronization process are automatically processed and resolved, and data consistency is checked to obtain the execution process of the data conflict model as follows;

[0119] Sub-step S403 identifies records based on conflict points and designs specific conflict resolution strategies and preventive measures, including rolling back conflicting data to restore to the last stable state and adjusting the priority of data tasks to prevent future conflicts. The automated processing flow will be applied to all identified conflict points, automatically resolving conflicts and ensuring data consistency through software tools. The effectiveness of the process is verified through subsequent data verification steps to ensure that all data can be synchronized and consistent, providing a comprehensive conflict handling and prevention framework and obtaining a data conflict model.

[0120] See also Figure 6 According to the data conflict model, data rollback is performed at the conflict point, a solution strategy is formulated for the conflict point, and the strategy is implemented to avoid data conflicts. The specific steps for generating conflict resolution solutions are as follows:

[0121] S501: Based on the data conflict model, data rollback is implemented to adjust the affected massive data points to a stable state, verify the consistency and synchronization efficiency of the data, avoid data synchronization errors, and generate the data rollback result. The execution process is as follows;

[0122] Based on the data conflict model, sub-step S501 performs a rollback operation on the data affected by the conflict to restore the data to the previous stable state. By identifying the data points where the data is inconsistent due to synchronization errors and rolling them back to a consistent state, data consistency and overall synchronization efficiency can be ensured. A series of synchronization tests will be performed to verify the consistency of the data and the effectiveness of the rollback operation. The test helps ensure that all data remains synchronized after the rollback process, without data loss or errors, and generates a data rollback result using the formula:

[0123]

[0124] in, represents the data synchronization efficiency ratio, is the number of data points in a stable state after the rollback, 1 is the total number of data points.

[0125] S502: Using the data rollback results, formulate a solution plan for the conflict types in the massive data synchronization process, and avoid the impact of conflicts in the future time period by implementing priority reordering and resource reconfiguration measures. The execution process of generating the conflict type analysis results is as follows;

[0126] Sub-step S502 uses the data rollback results to analyze the various conflict types that occur during the synchronization process, and formulates a specific solution plan accordingly. This includes a detailed classification of the types of conflicting data, such as timestamp conflicts, data overwrite conflicts, etc., and designing a special solution strategy for each type. For example, for timestamp conflicts, time synchronization calibration is adopted, and for data overwrite conflicts, priority needs to be redefined. Resources are reconfigured, such as increasing bandwidth or adjusting storage resources, to support the implementation of conflict resolution strategies. Through these measures, future conflicts can be effectively avoided, the reliability of data synchronization can be improved, and conflict type analysis results can be generated. The formula used is:

[0127]

[0128] in, represents the conflict avoidance efficiency, It is The expected number of unresolved conflicts of conflict type, 1 is the total number of data points and n5 is the total number of conflict types.

[0129] S503: Based on the conflict type analysis results, a conflict resolution strategy is constructed and the execution effect is monitored. Through dynamic adjustment and effect verification of the conflict resolution strategy, massive data synchronization is performed to obtain the execution process of the conflict resolution solution as follows;

[0130] Substep S503 builds and implements a set of conflict resolution strategies based on the conflict type analysis results, including dynamic adjustment of specific resolution measures for each conflict, and optimization based on the actual conflicts that occur during the synchronization process. For example, if the frequency of a certain conflict exceeds expectations, it is necessary to strengthen the corresponding resource allocation or optimize the processing algorithm. The execution effect of the conflict resolution strategy is tracked through real-time monitoring equipment to ensure that each measure can effectively resolve the target conflict. Through dynamic adjustment and effect verification, the efficiency and stability of the data synchronization process can be ensured, and a conflict resolution method can be obtained.

[0131] See also Figure 7 , based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data receiver, and the specific steps for obtaining the data summary push result are:

[0132] S601: Based on the conflict resolution method, the captured batch target data is integrated, and the synchronization and consistency of the information of the data sets are checked by merging the target data sets, and the execution process of generating the data integration result is as follows;

[0133] Sub-step S601 integrates the captured batch target data based on conflict resolution. The process involves merging data sets captured from different sources or at different time points to ensure data synchronization and consistency. By performing a series of consistency checks on the merged data set, such as comparing the timestamps of data items, content integrity, etc., it ensures that all data is logically consistent. It helps identify and resolve any inconsistencies that arise during the data merging process. After the integration is completed, the data integration result is generated using the formula:

[0134]

[0135] in, represents the synchronization consistency ratio of data integration, is the number of data sets that pass the consistency check, is the total number of datasets before merging.

[0136] S602: Use the data integration results to perform verification of mass data synchronization, verify the consistency and integrity of the data, analyze the efficiency of batch capture of target data by comparing differentiated data source records, and generate the execution process of data consistency verification records as follows;

[0137] Sub-step S602 uses the data integration results, including verifying the consistency and integrity of the data, and analyzing the efficiency of batch crawling of target data by comparing the records of differentiated data sources. For example, any unsynchronized or missing data can be detected by comparing the record versions of each data source with the version of the latest integrated data set. Through comparative analysis, any defects in the data synchronization process can be identified, and the data crawling and synchronization strategy can be optimized accordingly, and data consistency verification records can be generated. The formula used is:

[0138]

[0139] in, represents the data integrity verification ratio, It is The number of unsynchronized data sources. is the total number of datasets before merging, is the total number of data sources.

[0140] S603: According to the data consistency verification record, the format of the massive data is unified and pushed to the data recipient. Through format standardization and data transmission, the recipient's use of the target data is verified, and the execution process of obtaining the data summary push result is as follows;

[0141] Sub-step S603 unifies the format of massive data based on data consistency verification records and pushes it to the data recipient, which involves formatting all data into a unified standard, such as a unified time format, digital representation, etc., so that the data recipient can correctly parse and use the data. Through format standardization and effective data transmission strategies, ensure that the recipient can receive and use the pushed data without error. Implement the effect verification after data transmission, evaluate the practicality and accuracy of the data through the feedback of the recipient, and obtain the data summary push results. The formula used is:

[0142]

[0143] in, represents the data usage efficiency ratio, is the amount of data effectively utilized, is the total amount of data pushed.

[0144] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for batch capturing target data during mass data synchronization, characterized in that: The following steps are involved: Identify the attributes of data sources, analyze data update frequency, data volume, and data type heterogeneity, classify data sources, and calculate feature vectors to generate data source characteristic evaluation results; Based on the data source characteristic evaluation results, the batch data capture paths in the massive data synchronization process are analyzed, the data capture paths are sorted according to priority, and a data capture strategy is generated, which specifically includes: S201: Based on the data source characteristic evaluation result, a decision tree algorithm is used to identify key attributes of the data source, evaluate the impact of the key attributes on the priority of batch crawling target data, and generate a key attribute list; S202: Analyze the dependencies and connectivity between data sources using the key attribute list, calculate the optimal path in the process of mass data synchronization, avoid data transmission delay, and generate data capture path planning; S203: According to the data capture path planning, the paths for batch capturing target data are sorted according to priority, the execution order and resource allocation of each path are set, the efficiency and response speed in the process of mass data synchronization are verified, and the data capture strategy is obtained; Through the data capture strategy, the operating parameters of the task scheduler are set, the data capture tasks are dynamically monitored, and the capture tasks are dynamically adjusted according to data changes, and task scheduling records are generated, which specifically include: S301: Based on the data capture strategy, set the operating parameters of the task scheduler, analyze the task priority, resource allocation and time interval, check the response of the task scheduler to data capture and manage data capture requirements, and generate a scheduling parameter configuration record; S302: using the scheduling parameter configuration record, dynamically monitoring the batch capture target data task in the massive data synchronization process, mining and processing anomalies and deviations in task execution by real-time tracking of task progress and resource usage status, and generating real-time monitoring records; S303: According to the real-time monitoring record, dynamically adjust the execution strategy of the data capture task, including re-ordering of task priorities and reallocation of resources, match the real-time massive data changes, and obtain the task scheduling record; Using the task scheduling records, simulate the data update process in the massive data synchronization process, identify the conflict points of batch capture target data, avoid potential risks, and build a data conflict model, specifically including: S401: Based on the task scheduling record, simulate the data update in the massive data synchronization process, identify the dynamics of the data capture environment and potential synchronization problems by analyzing the data flow and task execution sequence, and generate data update simulation results; S402: using the data update simulation results, adopting a conflict detection algorithm, identifying conflict points of batch-capturing target data during the massive data synchronization process, performing conflict point analysis on data versions and resource changes, and generating conflict point identification records; S403: Design conflict resolution strategies and preventive measures based on the conflict point identification records, perform data rollback and priority adjustment, automatically process and resolve conflicts in the massive data synchronization process, check data consistency, and obtain a data conflict model; According to the data conflict model, data rollback of the conflict point is performed, a solution strategy is formulated for the conflict point, and the strategy is implemented to avoid data conflicts and generate a conflict solution, which specifically includes: S501: Based on the data conflict model, data rollback is implemented, and the consistency and synchronization efficiency of data are verified by adjusting the affected massive data points to a stable state, avoiding data synchronization errors, and generating data rollback results; S502: Utilizing the data rollback result, a resolution plan is formulated for the conflict types in the massive data synchronization process, and by implementing priority reordering and resource reconfiguration measures, the impact of conflicts in the future time period is avoided, and a conflict type analysis result is generated; S503: constructing a conflict resolution strategy and monitoring the execution effect according to the conflict type analysis result, and performing massive data synchronization to obtain a conflict resolution solution through dynamic adjustment and effect verification of the conflict resolution strategy; Based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data recipient, and the data summary push result is obtained.

2. The method for batch capturing target data in the process of mass data synchronization according to claim 1, characterized in that: The data source characteristic assessment results include data update frequency analysis results, data volume statistics results and data type heterogeneity analysis results; the data crawling strategy includes data crawling path priority sorting, resource allocation optimization indicators and crawling task time plan; the task scheduling record includes task execution time log, task status update record and resource usage statistics; the data conflict model includes conflict point identification, potential risk analysis and prevention strategy plan; the conflict resolution method includes data rollback, implementation of resolution strategy and evaluation of strategy execution effect; the data summary push result includes data consistency verification result, data format unification status and data push status.

3. The method for batch capturing target data during mass data synchronization according to claim 1, characterized in that: The steps to identify the attributes of data sources, analyze data update frequency, data volume and data type heterogeneity, classify data sources, and calculate feature vectors to generate data source characteristic evaluation results are as follows: Identify the attributes of the data source, measure the update frequency of the massive data during the synchronization of massive data, record the volume of the data source and analyze the type of massive data, and generate data attribute records; Based on the data attribute records, the massive data is classified, the massive data is grouped according to update frequency, volume and type, a category is marked for each data source, the efficiency of massive data management is checked, and a data source classification result is generated; According to the data source classification results, the characteristic vectors of differentiated massive data are calculated, the central tendency and dispersion of each type of massive data are calculated, and the data source characteristic evaluation results are obtained by optimizing the massive data synchronization process.

4. The method for batch capturing target data during mass data synchronization according to claim 3, characterized in that: The formula of the decision tree algorithm is as follows: in, is the Gini impurity, For Category The proportion of data items , and is the weight coefficient, For Category The frequency of n1 is the number of categories, For data items The distribution density of For Category The stability of the samples.

5. The method for batch capturing target data in the process of mass data synchronization according to claim 4, characterized in that: The formula of the conflict detection algorithm is as follows: in, For data elements and The degree of conflict between is the number of data updates, and Data elements and In the The state value at the time of the update, is the weight coefficient, and External factors that affect data updates and data access frequency.

6. The method for batch capturing target data in the process of mass data synchronization according to claim 1, characterized in that: Based on the conflict resolution method, the captured batch target data is integrated, the consistency of the data is verified through data synchronization, the format is unified and pushed to the data recipient, and the steps for obtaining the data summary push result are as follows: Based on the conflict resolution method, the captured batch target data is integrated, and by merging the target data sets, the synchronization and consistency of the information of the data sets are checked to generate a data integration result; Using the data integration results, perform verification of mass data synchronization, verify the consistency and integrity of the data, analyze the efficiency of batch capture of target data by comparing differentiated data source records, and generate data consistency verification records; According to the data consistency verification record, the format of the massive data is unified and pushed to the data recipient. Through format standardization and data transmission, the recipient's use of the target data is verified to obtain a data summary push result.

Citation Information

Patent Citations

  • Mobile data sync conflict preprocessing and resolution method based on data package

    CN104253858A

  • Method for extracting extended words based on Chinese educational videos

    CN108052630A

  • Methods and systems for detection in industrial internet of things data collection environment with large data sets

    CN110073301A

  • AGV anti-collision path planning method based on A3C

    CN116661456A

  • Cross-data-source real-time synchronization method based on timeline mapping

    CN118410106A