A method for automatically monitoring data collection progress

Abstract

The present invention discloses a method for automatically monitoring data collection progress, which relates to the field of data processing technology. The method comprises: before the start of a data collection task, establishing a unique identifier, timestamp and task priority of a data packet, creating a task progress list to display the collection progress in real time; evaluating the collection progress based on the timestamp, establishing independent data packet evaluation results and joint data packet evaluation results; locating progress optimization data packets using the independent data packet evaluation results and the joint data packet evaluation results, and performing collection optimization management of progress optimization data packets based on a time-space perception network and a load balancing network. The present invention solves the technical problem that traditional data collection methods in the prior art lack flexibility and load management capabilities in a dynamic network environment, which affects the efficiency and stability of data collection, and achieves the technical effect of improving the stability and efficiency of the data collection process through dynamic adjustment and optimization management of the data collection process.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for automatically monitoring data collection progress. Background Art

[0002] In the process of big data processing and transmission, data collection is a key step to ensure data integrity and accuracy. With the widespread application of the Internet of Things (IoT), edge computing, and cloud computing, data collection tasks have become increasingly complex, involving the coordination of multiple collection nodes and the real-time transmission and processing of a large number of data packets. However, traditional data collection methods often find it difficult to monitor the progress of data collection in real time when faced with a dynamically changing network environment, resulting in problems such as packet transmission delays, uneven bandwidth allocation, and even data loss, which greatly affects the efficiency of data collection tasks and the overall performance of the system. Summary of the invention

[0003] The present application provides a method for automatically monitoring data collection progress, which is used to solve the technical problem that traditional data collection methods in the prior art lack flexibility and load management capabilities in a dynamic network environment, affecting data collection efficiency and stability.

[0004] The present application provides a method for automatically monitoring data collection progress, the method comprising: before the start of a data collection task, establishing a unique identifier for a data packet, assigning a timestamp to the data packet, and creating a task priority for the data packet; creating a task progress list, the task progress list displays the real-time task progress of each data packet through a unique identifier, the task progress list responding to the real-time monitoring result of the task; performing real-time evaluation on the collection progress of each data packet in the task progress list based on the timestamp, and establishing an independent data packet evaluation result; performing joint application evaluation on the data packets according to the timestamp, and establishing a joint application evaluation result, and performing joint evaluation on the collection progress of the data packets in the task progress list using the joint application evaluation result, and establishing a joint data packet evaluation result; locating a progress-optimized data packet using the independent data packet evaluation result and the joint data packet evaluation result, and performing collection optimization management of the progress-optimized data packet based on a spatiotemporal perception network and a load balancing network.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0006] The present application provides a method for automatically monitoring data collection progress, which relates to the field of data processing technology. It creates a unique identifier, timestamp and task priority for a data packet before a data collection task, establishes a task progress list to monitor the progress in real time, performs independent and joint evaluations through timestamps, and locates data packets that need to be optimized based on the evaluation results. It uses time-space perception and load balancing networks to dynamically adjust collection strategies, optimize transmission paths and node loads, and solves the technical problems in the prior art that traditional data collection methods lack flexibility and load management capabilities in a dynamic network environment, which affects data collection efficiency and stability. It achieves the technical effect of improving the stability and efficiency of the data collection process through dynamic adjustment and optimization management of the data collection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A flowchart of a method for automatically monitoring data collection progress provided in an embodiment of the present application;

[0009] Figure 2 A flowchart of collection optimization management of progress-optimized data packets in a method for automatically monitoring data collection progress provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The present application provides a method for automatically monitoring data collection progress, which is used to solve the technical problem that traditional data collection methods in the prior art lack flexibility and load management capabilities in a dynamic network environment, affecting data collection efficiency and stability.

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0012] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0013] Embodiment 1, as Figure 1 As shown, the present application provides a method for automatically monitoring data collection progress, the method comprising:

[0014] P10: Before the data collection task starts, a unique identifier for the data packet is established, a timestamp is assigned to the data packet, and a task priority for the data packet is created.

[0015] It should be understood that before the data collection task begins, a unique identifier will be established for each data packet to be collected to ensure that each data packet can be uniquely identified during the entire collection, transmission and storage process to avoid data packet confusion and repeated processing. The unique identifier can be generated based on the content characteristics, time, sequence number, etc. of the data packet, and bound to the data packet, so that each data packet has independence and traceability.

[0016] Next, a timestamp is assigned to each data packet. The purpose of the timestamp is to record the exact time point when the data packet is generated or received, providing a basis for subsequent data packet sorting, delay analysis, and real-time monitoring. Timestamps are usually based on high-precision clock sources, such as GPS time, atomic clocks, or system time, to ensure the accuracy and consistency of time. Timestamps not only play a key role in the process of data packet processing, but also serve as a reference in task priority sorting and progress evaluation.

[0017] At the same time, a task priority is created for each data packet. Task priority is set based on the importance and urgency of the data packet, and is used to sort and schedule data packets during the collection process. Priority can be set based on multiple factors, such as the importance of the data packet (for example, important data is prioritized), the collection time limit (tasks that need to be completed within a specific time are prioritized), and the impact of the data packet on subsequent analysis (data that has a significant impact on the analysis results is prioritized). The priority setting enables the system to prioritize data packets that have a greater impact on the overall progress of the task or data quality during the collection task, thereby improving collection efficiency and data processing accuracy.

[0018] Through these steps, it can be ensured that each data packet is orderly and controllable during the collection process, and the real-time and priority processing capabilities of the data are guaranteed, thus laying the foundation for the smooth progress of the entire data collection task.

[0019] P20: Create a task progress list, which displays the real-time task progress of each data packet through a unique identifier, and the task progress list responds to the real-time monitoring result of the task.

[0020] Optionally, after the data collection task is started, a task progress list is created to track and display the collection progress of each data packet in real time. The task progress list is a dynamic data structure that helps managers and systems intuitively understand the progress of the task by displaying the real-time status of the data packet. Each data packet has a corresponding entry in the task progress list, which is associated with the unique identifier previously generated for each data packet, ensuring that each data packet in the list accurately corresponds to the actual data packet in the collection task.

[0021] In the task progress list, the status information of each data packet will be updated in real time, including key parameters such as the start time of data packet collection, completion status of collection, and data packet size. By regularly refreshing this status information, the current processing stage of each data packet can be accurately displayed, such as "collecting", "completed", "not started", etc., which facilitates comprehensive monitoring of task progress.

[0022] The dynamic update of the task progress list depends on the task real-time monitoring mechanism. The function of this mechanism is to continuously monitor the status changes of each data packet during the data collection process and feed back these changes to the task progress list. Specifically, the task real-time monitoring can obtain the collection status of each data packet in a timely manner by monitoring the network status, transmission rate, collection device status, etc. For example, when the transmission delay of a data packet exceeds the preset time range, the task monitoring mechanism will immediately capture this change and mark the progress status of the data packet as "delayed" in the task progress list. Through this real-time monitoring method, the system can promptly reflect abnormal situations in the collection process in the task progress list. As a key tool in the data collection process, the task progress list makes the management of the entire collection task more transparent and efficient, and also provides technical means and guarantees for dealing with uncertain factors in the collection process.

[0023] P30: Perform real-time evaluation of the collection progress of each data packet in the task progress list based on the timestamp, and establish independent data packet evaluation results.

[0024] Specifically, during the execution of the data collection task, the collection progress of each data packet in the task progress list needs to be evaluated in real time through the timestamp. The timestamp records the key time nodes of each data packet at different stages, including the start time, end time of the collection, and the transmission time of the data packet. Through the timestamp, the time consumption of each data packet at different stages can be accurately calculated, thus providing a detailed time basis for progress evaluation.

[0025] For example, during the evaluation process, the collection progress of each data packet can be analyzed in real time based on the timestamp. The collection progress refers to the time ratio from the start of the task to the completion of data collection of the data packet. This ratio can intuitively reflect the processing speed of the data packet. For example, if the expected collection time of a data packet is 10 seconds, and the actual collection time is 12 seconds, it can be determined that there is a delay in the progress of the data packet. By analyzing these time information, bottleneck data packets or data packets that need to be optimized in the collection process can be quickly identified.

[0026] The evaluation results are generated in the form of independent data packet evaluation results, and an evaluation report is given for each data packet. The independent data packet evaluation results may include the following information: whether the collection time is within the expected range, the transmission stability of the data packet, and whether there is any delay or packet loss. The evaluation results of each data packet can be further divided into "normal", "delayed", "failed" and other states to help the system and managers quickly identify possible problems in the collection process.

[0027] In order to ensure the accuracy of real-time evaluation, data analysis algorithms such as weighted time mean calculation and threshold judgment can be used. These algorithms can effectively analyze the collection of data packets, compare the actual collection time of data packets with the expected time, and obtain the evaluation results of each data packet. In this way, the system can continuously and carefully monitor the progress of each data packet during the data collection process.

[0028] The purpose of establishing independent data packet evaluation results is to provide reliable data support for subsequent data collection optimization. Through these evaluation results, the system can identify data packets that perform poorly during the collection process and provide a basis for subsequent progress adjustments, thereby improving the overall efficiency of the entire collection task. This process makes the data collection evaluation system more refined and scientific, ensuring the efficient and accurate execution of data collection tasks.

[0029] P40: Perform joint application evaluation of data packets according to timestamps, establish joint application evaluation results, use the joint application evaluation results to conduct joint evaluation of the collection progress of data packets in the task progress list, and establish joint data packet evaluation results.

[0030] It should be understood that in the process of executing the data collection task, in addition to evaluating a single data packet, it is also necessary to conduct a joint application evaluation of multiple data packets from an overall perspective to more comprehensively evaluate the overall progress of the task. The core of the joint application evaluation is to use the timestamp information of each data packet to analyze the collection progress and status of multiple data packets in the same time period to derive their collaborative performance and mutual influence in the task.

[0031] First, the individual data packets in the task progress list are integrated and analyzed according to the timestamp to generate the joint application evaluation results. The joint application evaluation results are a comprehensive evaluation that evaluates the overall collection efficiency and task completion of multiple data packets in the same time period by analyzing multiple parameters such as data packet collection time, delay, and transmission rate. For example, if some data packets are delayed while other data packets are transmitted normally in the same time period, this may indicate that the system load is unbalanced or that there are problems with certain network nodes. In this way, the joint application evaluation can identify systemic problems in the collection process, not just problems with individual data packets.

[0032] After obtaining the joint application evaluation results, the system will use these evaluation results to jointly evaluate the collection progress of the data packets in the task progress list. The purpose of the joint evaluation is to review the task progress as a whole to identify the synergy between data packets. Specifically, this process analyzes the progress relationship between each data packet and evaluates whether multiple data packets can complete the task together within the expected time. For example, when a group of data packets need to complete a collection task together within a specific time, if the progress of some of the data packets is delayed, the system can use the joint evaluation results to discover this problem and analyze the impact of the delay on other data packets and the progress of the entire task.

[0033] Through joint evaluation, a joint data packet evaluation result is generated. This result reflects the collaborative performance of multiple data packets in the acquisition task and is a comprehensive evaluation based on holistic analysis. The joint data packet evaluation result includes analysis results on transmission synchronization between data packets, delay propagation effects, and overall acquisition efficiency. Different from independent data packet evaluation, the joint evaluation focuses more on the relationship between data packets and their impact on the overall progress of the task, thereby providing a macro perspective for the system and helping to identify systemic problems and potential performance optimization points.

[0034] To achieve these evaluations, collaborative analysis algorithms and data clustering techniques can be used. Collaborative analysis algorithms can analyze the timestamps and transmission characteristics of multiple data packets to determine the dependencies and collaboration patterns between them. Data clustering techniques are used to identify groups of data packets that behave similarly in the same time period, thereby better conducting joint evaluations. These technical supports ensure the accuracy and comprehensiveness of joint evaluations.

[0035] Through the joint application evaluation of data packets and the joint evaluation of collection progress, the system can discover overall problems that are difficult to detect in the evaluation of individual data packets, and provide deep insights for the optimization of data collection tasks. This process enables the system to adjust and optimize from a global perspective when facing complex tasks, thereby improving the overall efficiency and quality of data collection tasks.

[0036] Further, the joint application evaluation of the data packets is performed according to the timestamps to establish the joint application evaluation results. Step P40 of the embodiment of the present application further includes:

[0037] P41: Perform basic correlation analysis on the data packets and establish basic correlation analysis results; P42: Use timestamps to perform time series correlation screening on the basic correlation analysis results and establish time series correlation screening results; P43: Use the time series correlation screening results as the evaluation as a whole to generate an overall criticality identifier; P44: Use the time series correlation screening results and the overall criticality identifier to establish a joint application evaluation result.

[0038] Preferably, a joint application evaluation of data packets is performed in depth, which not only relies on timestamp information but also incorporates complex data analysis and correlation techniques.

[0039] First, basic correlation analysis is performed on the data packets to identify the inherent correlation between the data packets. Basic correlation analysis finds out the data packets that are related to each other during the collection process by analyzing factors such as the content characteristics, collection source, and transmission path of the data packets. For example, if multiple data packets come from the same data source or are transmitted on the same network node, there may be a correlation between them. Through this analysis, a basic correlation analysis result can be formed, which provides the basis for correlation data for subsequent timing analysis.

[0040] Furthermore, after obtaining the basic correlation analysis results, the system will perform time-series correlation screening on these correlation data in combination with the timestamp information. Time-series correlation screening refers to screening and sorting the basic correlation data in the time dimension according to the timestamp of the data packet, so as to identify groups of data packets that have time-series correlation within a specific time period. For example, if some data packets experience delays or peak transmission at the same or similar time points, these data packets may have a time-series dependency. The result of time-series correlation screening is called the time-series correlation screening result, which reflects the collaborative state of the data packets in the time dimension.

[0041] Next, the timing correlation screening results are used as the basis for overall evaluation, the importance of the data packet group is evaluated, and an overall criticality identification is generated. The overall criticality identification refers to the evaluation identification of the importance and influence of the data packet group in the task execution process. This identification is usually based on a comprehensive evaluation of multiple factors, including the time sensitivity of the data packet group in the task, the degree of influence on other data packets, and its position in the transmission link. Through this identification, the system can quickly identify the critical data packet group that has the greatest impact on the overall acquisition progress, so that these data packets can be processed first in the subsequent optimization process.

[0042] Finally, the timing association screening results are combined with the overall criticality identification to generate the final joint application evaluation results. This result is a comprehensive evaluation of the data packets in terms of time and correlation, which not only takes into account the timing synergy between data packets, but also weighs the importance of each data packet to the overall task progress. In this process, a multi-dimensional data fusion algorithm can be used to fuse and calculate the timing information and criticality information to generate a more accurate evaluation result. The formation of the joint application evaluation results provides the system with a macro perspective to help identify the key links that need to be optimized in priority during task execution. This enables the system to allocate resources and adjust strategies more intelligently, thereby improving the overall efficiency and accuracy of data collection tasks.

[0043] Furthermore, step P40 of the embodiment of the present application also includes:

[0044] P45: Conduct data completion analysis on each evaluation entity within the joint application evaluation results, and establish a progress penalty coefficient based on the data completion ratio; P46: Conduct a scheduled packaging waiting evaluation on each evaluation entity within the joint application evaluation results, and establish a progress reward coefficient based on the scheduled packaging waiting evaluation results; P47: Use the progress penalty coefficient and progress reward coefficient to conduct a joint evaluation of the collection progress of the data packets in the task progress list, and establish a joint data packet evaluation result.

[0045] Specifically, the specific process of establishing the joint data package evaluation results through joint evaluation can be, first, to perform data completion analysis on each evaluation entity in the joint application evaluation results. Data completion analysis refers to calculating the degree of task completion of each data package group within a specific time period to obtain its actual completion degree. Data completion is usually expressed in the form of a completion ratio. For example, if a data package group only completes 80% of the collection tasks within the scheduled time, its data completion is 80%. According to these data completion ratios, a progress penalty coefficient is set for each evaluation entity. The progress penalty coefficient is used to indicate the degree of progress lag. Data package groups with lower completion will be assigned higher penalty coefficients to remind the system to invest more resources or optimize these tasks.

[0046] Next, a scheduled encapsulation waiting evaluation is performed on each evaluation entity within the joint application evaluation result, and a progress reward coefficient is established based on the scheduled encapsulation waiting evaluation result. The scheduled encapsulation waiting evaluation refers to the process of analyzing whether a data packet needs to wait for other data packets for joint processing or encapsulation after collection is completed. The purpose of this evaluation is to identify the dependencies between data packets and ensure that the data packets are encapsulated and processed at the most appropriate time. If a group of data packets remains efficient while waiting for other data packets and completes collection within the scheduled waiting time, a higher progress reward coefficient is set for the group. The progress reward coefficient is used to encourage groups of data packets that perform well during the encapsulation waiting period to balance the delays caused by waiting in progress management.

[0047] Furthermore, the progress penalty coefficient and progress reward coefficient are combined to jointly evaluate the data packet collection progress in the task progress list. Exemplarily, the collection progress of each data packet group is weighted, and the progress penalty coefficient is used to reduce the comprehensive score of the data packets that fail to complete the task on time, while the progress reward coefficient is used to improve the evaluation score of the data packets that complete the task on time or efficiently. In this way, the penalty and reward are balanced in the evaluation process, so as to more reasonably reflect the actual performance of each data packet group.

[0048] Finally, based on these comprehensive factors, the joint data packet evaluation results are generated. The joint data packet evaluation results not only consider the timing correlation between data packets, but also introduce detailed evaluation of data completion and packaging waiting status, so that the system can adjust the collection strategy more flexibly and intelligently during the data collection task management process. This process provides a deep optimization path for the progress management of data collection tasks, ensuring that the overall task is completed efficiently within the scheduled time, and that data quality and collection accuracy are guaranteed.

[0049] P50: Use the independent data packet evaluation results and the joint data packet evaluation results to locate the progress optimization data packets, and collect and optimize the progress optimization data packets based on the space-time perception network and the load balancing network.

[0050] Further, such as Figure 2 As shown, step P50 of the embodiment of the present application also includes:

[0051] P51: Activate the data backtracking layer of the space-time perception network, use the data backtracking layer to perform data backtracking of the collection nodes of the positioning progress optimization data packet, and create a mapping of the impact of the position change and time change of the collection nodes on the transmission effect based on the data backtracking results of the collection nodes; P52: Call the impact mapping with the space-time perception network, perform perception optimization of the collection position point and collection time point of the positioning progress optimization data packet, and establish the first optimization result; P53: Perform collection optimization management based on the first optimization result.

[0052] It should be understood that in order to further improve the efficiency and accuracy of data collection, the data packets that need to be optimized are located through independent data packet evaluation results and joint data packet evaluation results, and these data packets are collected and optimized and managed in combination with the space-time awareness network and the load balancing network.

[0053] First, activate the data backtracking layer of the spatiotemporal perception network. The function of the data backtracking layer is to backtrack and analyze the historical data of each node in the data collection process to understand the performance of the collection node under different time and location conditions. By analyzing these historical data, the specific impact of the collection node location and collection time on the data transmission effect (such as transmission rate, packet loss rate, etc.) can be identified, thereby generating an impact map. This impact map describes in detail how changes in node location and time affect data transmission efficiency and stability, providing a basis for subsequent collection optimization.

[0054] Then, after obtaining the impact map, the space-time perception network is used to call these mapping information to perform perception optimization on the collection location and time points of the positioned progress optimization data packets. Perception optimization refers to selecting the optimal collection location and time point to maximize the transmission effect and minimize the delay under the premise of considering space and time factors. For example, if the impact map shows that certain collection nodes have better transmission effects in specific time periods, the system will give priority to these time periods for collection tasks. Through this space-time perception optimization, the space-time point that is most suitable for data packet collection is found to generate the first optimization result.

[0055] After obtaining the first optimization results, the progress optimization data packets located are collected and managed based on these results. This management process includes adjusting the scheduling strategy of the collection tasks, optimizing the distribution of the collection nodes, and reallocating the network bandwidth to ensure that the data packets can be collected under the optimal time and space conditions. This process involves the coordinated work of the load balancing network to ensure that the transmission load between different collection nodes is reasonably distributed to avoid data transmission delays or interruptions caused by excessive load on individual nodes.

[0056] In the whole process, the combination of spatiotemporal perception network and load balancing network enables the system to flexibly respond to dynamically changing environmental conditions in the collection task. Through the historical analysis of the data backtracking layer and the forward-looking decision-making of perception optimization, it can not only optimize the collection efficiency of the current data packet, but also provide adjustment references for future tasks, ensure the efficiency and stability of the data collection process, and maximize the use of system resources, improving the overall quality of data collection tasks.

[0057] Furthermore, step P53 of the embodiment of the present application also includes:

[0058] P53-1: Activate the load balancing network, perform load analysis on each acquisition node under the current timing, and establish the load analysis results; P53-2: Establish the first load tilt constraint based on the load analysis results and the positioning progress optimization data packet; P53-3: Use the load balancing network to predict the load changes of each acquisition node, and establish the second load tilt constraint based on the load change prediction results and the positioning progress optimization data packet; P53-4: Establish the load balancing scheduling result based on the first load tilt constraint and the second load tilt constraint, and use the first optimization result and the load balancing scheduling result to perform acquisition optimization management.

[0059] Optionally, the specific process of performing collection optimization management based on the first optimization result may be, first, activating the load balancing network, and performing a detailed analysis of the load conditions of each collection node at the current time point. The content of the load analysis includes the data processing capacity of each collection node, the network bandwidth occupancy, and the load pressure of the current transmission task. Through this analysis process, the system can identify which nodes are in a high-load state and which nodes have more transmission margins, thereby generating load analysis results and providing basic data for subsequent load scheduling and optimization.

[0060] After obtaining the load analysis results, the first load tilt constraint is set in combination with the information of the positioning progress optimization data packet. The first load tilt constraint refers to adjusting the transmission path and task scheduling of the data packet according to the load of the current collection node to avoid overloaded nodes as much as possible. For example, if a node has a high load and affects the transmission speed of the data packet, the transmission path of the data packet is adjusted according to the constraint, and the node with a lower load is preferentially selected for collection and transmission, thereby reducing the transmission delay caused by uneven load.

[0061] Next, the load balancing network is used to predict the future load changes of each collection node. By analyzing the historical load data and the current load trend, the load changes of each collection node in the future period are predicted, thereby obtaining the load change prediction results. Based on these prediction results and the information of the positioning progress optimization data packet, the system establishes a second load tilt constraint. The second load tilt constraint is a mechanism for pre-regulating future loads, which aims to adjust the transmission path of the data packet according to the load change trend to prevent the problem of excessive load at a future time point. Through this pre-judgment mechanism, the system can better balance the load of each node and optimize the transmission path of the data packet.

[0062] After obtaining the first and second load tilt constraints, the load balancing scheduling result is generated by combining these constraints. The load balancing scheduling result is an optimized scheduling scheme generated by the system for each collection node and data packet, taking into account the current load and future load. This result ensures that the transmission task of the data packet is carried out without exceeding the load bearing capacity of the node, thereby avoiding node overload or resource waste. Then, the first optimization result (optimization for collection location points and time points) and the load balancing scheduling result are used to perform more accurate collection optimization management of the progress optimization data packet.

[0063] Through the above steps, not only can the data packets be reasonably scheduled according to the current load situation, but also future load changes can be predicted in advance, providing a longer-term optimization strategy for the transmission and collection of data packets. The setting of load tilt constraints and the load balancing scheduling combined with the space-time perception network enable the system to achieve more efficient resource allocation and task scheduling in a complex collection task environment, thereby improving the stability and transmission efficiency of the overall data collection. It can not only effectively reduce bottlenecks in the collection process, but also greatly improve the system's ability to respond to sudden load changes.

[0064] Furthermore, the embodiment of the present application further includes step P60, and step P60 further includes:

[0065] P61: Establish a warning threshold for the load; P62: Use the load analysis results to determine the trigger of the warning threshold; P63: If the determination result is a trigger result, generate a centralized scheduling instruction, and establish a sequential collection plan for the overall evaluation through the centralized scheduling instruction and the overall criticality identification; P64: Use the sequential collection plan to perform collection optimization management.

[0066] It should be understood that in order to further improve the system's ability to respond to load changes, the embodiment of the present application also achieves more efficient management of data collection tasks through a load warning mechanism and a centralized scheduling strategy.

[0067] Specifically, first set a load warning threshold for each collection node. The warning threshold means that when the load level of the collection node exceeds a certain set value, the system will trigger the warning mechanism to avoid data transmission delays or failures caused by node overload. The setting of the threshold can be optimized based on factors such as historical load data, node processing capabilities, and system transmission stability to ensure that when the load reaches a certain level, a warning signal can be issued in time.

[0068] When performing load analysis, the current load analysis results are compared with the pre-set warning threshold to determine whether the warning mechanism needs to be triggered. This process is called trigger judgment of the warning threshold. If the analysis results show that the load of a certain node has exceeded the set warning threshold, a trigger result is generated, indicating that the load of the node is too high and scheduling adjustments are required. The trigger judgment process ensures that the system can respond quickly when the load exceeds the safe range.

[0069] When the system determines that the load of a node exceeds the warning threshold, it automatically generates a centralized scheduling instruction. Centralized scheduling instructions are a control instruction for centralized management and optimized allocation of system resources. They aim to reduce the pressure on high-load nodes by rescheduling the transmission path and collection order of data packets. In order to ensure the rationality of the scheduling strategy, it is necessary to refer to the overall criticality identifier when generating centralized scheduling instructions. The overall criticality identifier reflects the importance of the data packet group in the task. Based on this identifier, the system can prioritize the scheduling of data packets that have a greater impact on the task to ensure the smooth progress of the overall task. Combined with this information, an overall sequential collection plan is established, that is, the collection order of data packets is rearranged to ensure that the key data packets of the task are collected and transmitted first under the premise that the node load is controlled.

[0070] After generating a sequential collection plan, the plan is used to optimize the collection management of data collection tasks to ensure that data packets are collected and transmitted in a reasonable order, avoiding transmission interruptions or delays caused by excessive node load. At the same time, the allocation of collection nodes and task scheduling are dynamically adjusted so that the data packet collection task can proceed smoothly under a relatively balanced system load. Through this process, the system can effectively alleviate the pressure of high-load nodes and ensure the overall efficiency and stability of data collection tasks.

[0071] Through the above steps, the system can not only predict load changes in advance and issue warnings in time, but also reasonably adjust data collection tasks under high load conditions through centralized scheduling instructions. The overall collection optimization management process ensures that the system can quickly adjust strategies when facing uncertain load changes to ensure the efficiency and reliability of data transmission.

[0072] Furthermore, the embodiment of the present application further includes step P70, and step P70 further includes:

[0073] P71: Establish a balance adjustment database for each data packet according to the task priority; P72: Calculate the comprehensive balance score of each data packet, and make a balance adjustment plan for delay and performance based on the comprehensive balance score and the balance adjustment database, and perform collection optimization compensation for each data packet based on the balance adjustment plan.

[0074] The comprehensive balance score of each data packet is calculated as follows:

[0075] ;

[0076] in, Represents the overall balance score, For the target delay, is the actual delay, For the performance of the current system, For target performance, is the delay weight coefficient, is the performance weight coefficient, and .

[0077] Specifically, the collection process is optimized by calculating the comprehensive balance score of each data packet, and combining the delay and performance balance adjustment strategy. First, a balance adjustment database is established for each data packet according to its task priority. The purpose of this database is to store data such as priority, delay information, performance indicators, etc. related to each data packet, providing a reference for subsequent balance adjustments. Task priority can help the system determine which data packets require higher transmission priority and which can be appropriately delayed when the network load is high. Through the balance adjustment database, the system can formulate personalized adjustment strategies for different data packets.

[0078] Next, the comprehensive balance score is calculated for each data packet as an indicator to measure the balance between latency and performance. The calculation formula for the comprehensive balance score is:

[0079] ;

[0080] in, Represents the overall balance score, is the target delay, i.e., the ideal packet transmission time, is the actual delay, which indicates the actual transmission time of the data packet in the current network environment. The performance of the current system, such as bandwidth utilization, processing speed, etc. is the target performance, i.e. the performance level of the system under ideal conditions, is the delay weight coefficient, is the performance weight coefficient, which is used to adjust the proportion of latency and performance in the overall balance score, and .

[0081] Through this formula, the system can comprehensively consider the transmission delay and system performance of each data packet and quantify the result into a comprehensive balance score. If the actual delay is low, or the system performance is better than expected, the comprehensive balance score of the data packet will be high, indicating that its transmission effect is good under the current conditions.

[0082] After obtaining the comprehensive balance score of each data packet, the system will generate a delay and performance balance adjustment plan for each data packet based on these scores and the balance adjustment database. This plan adjusts the data packet transmission strategy by weighing the relationship between the actual delay of the data packet and the system performance, such as appropriately adjusting the transmission priority or allocating more bandwidth to increase the transmission speed.

[0083] Finally, according to the balance adjustment plan, each data packet is collected and optimized for compensation. The collection optimization and compensation process includes bandwidth compensation, path optimization or rescheduling for data packets with poor transmission effects to ensure that they can better meet the target delay and performance requirements in subsequent transmissions. In this way, the collection and transmission strategy of data packets can be dynamically adjusted to achieve more intelligent and refined resource management, thereby improving the efficiency and stability of the entire data collection task.

[0084] In summary, the embodiments of the present application have at least the following technical effects:

[0085] This application creates a unique identifier, timestamp and task priority for the data packet before the data collection task, establishes a task progress list to display the collection progress in real time, and evaluates the progress through timestamps to generate independent and joint evaluation results. Based on these evaluation results, the data packets that need to be optimized are located, and the collection strategy is dynamically adjusted through the space-time perception network and load balancing network to optimize the transmission path and node load of the data packet.

[0086] The technical effect of improving the stability and efficiency of the data collection process by dynamically adjusting and optimizing the management of the data collection process has been achieved.

[0087] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0089] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A method for automatically monitoring data collection progress, characterized in that: The method comprises: Before the data collection task starts, a unique identifier for the data packet is established, a timestamp is assigned to the data packet, and a task priority for the data packet is created; Creating a task progress list, wherein the task progress list displays the real-time task progress of each data packet through a unique identifier, and the task progress list responds to the real-time monitoring result of the task; Based on the timestamp, the collection progress of each data packet in the task progress list is evaluated in real time, and the evaluation results of independent data packets are established; Conduct joint application evaluation of data packets according to timestamps, establish joint application evaluation results, conduct joint evaluation of collection progress of data packets in the task progress list using the joint application evaluation results, and establish joint data packet evaluation results; Use the independent data packet evaluation results and the joint data packet evaluation results to locate the progress optimization data packet, and perform collection and optimization management of the progress optimization data packet based on the time-space perception network and the load balancing network; The collection and optimization management of progress optimization data packets based on the time-space perception network and the load balancing network also includes: Activate the data backtracking layer of the space-time perception network, use the data backtracking layer to perform data backtracking of the collection nodes of the positioning progress optimization data packet, and create a mapping of the impact of the location change and time change of the collection nodes on the transmission effect based on the data backtracking results of the collection nodes; The influence mapping is called by the time-space perception network, and the collection location point and collection time point perception optimization of the positioning progress optimization data packet are performed to establish the first optimization result; Perform collection optimization management based on the first optimization result; The performing collection optimization management based on the first optimization result also includes: Activate the load balancing network, perform load analysis on each acquisition node under the current timing, and establish the load analysis results; Establishing a first load tilt constraint based on the load analysis result and the positioning progress optimization data packet; The load balancing network is used to predict the load changes of each collection node, and the second load tilt constraint is established based on the load change prediction result and the positioning progress optimization data packet; A load balancing scheduling result is established according to the first load tilt constraint and the second load tilt constraint, and collection optimization management is performed using the first optimization result and the load balancing scheduling result.

2. A method for automatically monitoring data collection progress as claimed in claim 1, characterized in that: The step of performing joint application evaluation of data packets according to the timestamp and establishing a joint application evaluation result further includes: Perform basic correlation analysis on the data packets and establish basic correlation analysis results; Use timestamps to perform time series correlation screening on basic correlation analysis results and establish time series correlation screening results; The time series correlation screening results are used as the evaluation whole to generate the overall criticality identification; The time series correlation screening results and overall criticality identification are used to establish joint application evaluation results.

3. A method for automatically monitoring data collection progress as claimed in claim 2, characterized in that: The step of using the joint application evaluation result to jointly evaluate the collection progress of the data packets in the task progress list and establishing the joint data packet evaluation result also includes: Conduct data completion analysis on each evaluation entity in the joint application evaluation results, and establish a progress penalty coefficient based on the data completion ratio; Each evaluation entity in the joint application evaluation result is scheduled to be packaged and wait for evaluation, and a progress reward coefficient is established based on the scheduled package waiting for evaluation result; The progress penalty coefficient and progress reward coefficient are used to jointly evaluate the collection progress of the data packets in the task progress list, and the joint data packet evaluation results are established.

4. A method for automatically monitoring data collection progress as claimed in claim 1, characterized in that: The method further comprises: Establish early warning thresholds for load; Use load analysis results to determine the triggering of warning thresholds; If the judgment result is a trigger result, a centralized scheduling instruction is generated, and a sequential collection plan for the overall evaluation is established through the centralized scheduling instruction and the overall criticality identification; Use sequential acquisition schemes to optimize acquisition management.

5. A method for automatically monitoring data collection progress as claimed in claim 1, characterized in that: The method further comprises: Establish a balance adjustment database for each data packet based on task priority; The comprehensive balance score of each data packet is calculated, and a balance adjustment scheme for delay and performance is performed based on the comprehensive balance score and a balance adjustment database, and collection optimization compensation for each data packet is performed based on the balance adjustment scheme.

6. A method for automatically monitoring data collection progress as claimed in claim 5, characterized in that: The comprehensive balance score of each data packet is calculated as follows: ; in, Represents the overall balance score, For the target delay, is the actual delay, For the performance of the current system, For target performance, is the delay weight coefficient, is the performance weight coefficient, and .

Images