A concentrator efficient data collection optimization method and system

Abstract

The present invention proposes a concentrator efficient data acquisition optimization method and system. Belonging to the field of electric power communication technology, the method includes: setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station; after the system is started, first determine whether there is an exclusive task in the task pool, if so, execute all exclusive tasks in order of priority; after the exclusive task is executed, obtain all parallel first reading tasks in the task pool, execute them in parallel through the task scheduling mode of the state machine, and obtain the corresponding data items to be read. By setting the reading task as an exclusive task or a parallel task and assigning priorities, system resources can be used more efficiently and the efficiency of data acquisition can be improved.

Description

Technical Field

[0001] The invention proposes a concentrator efficient data collection optimization method and system, belonging to the technical field of power communication. Background Art

[0002] The concentrator plays an important role in the data acquisition system, responsible for collecting data from multiple collection points and uploading the data to the main station for further processing. However, the traditional concentrator data collection method has the problem of low collection efficiency, especially when the data volume is large and the update frequency is high, the concentrator cannot handle it, which easily causes data loss or delay. Therefore, how to improve the data collection efficiency of the concentrator has become an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a concentrator efficient data collection optimization method and system to solve the problems mentioned in the above background technology:

[0004] The present invention provides a concentrator efficient data collection optimization method, the method comprising:

[0005] S1, setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station;

[0006] S2. After the system is started, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority;

[0007] S3. After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained through the task scheduling mode of the state machine;

[0008] S4. Analyze the historical data collected based on the correlation factors of the collection points and the machine learning algorithm to predict the future data change trend;

[0009] S5. Preprocess the collected data, encrypt the processed data through an encryption algorithm and send it to the cloud server for storage.

[0010] The present invention proposes a concentrator efficient data collection optimization system, the system comprising:

[0011] Task setting module: setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station;

[0012] Exclusive task module: After the system starts, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority;

[0013] Reading task module: After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained through the task scheduling mode of the state machine.

[0014] Trend prediction module: Analyzes historical data according to the correlation factors of the collection points and based on machine learning algorithms to predict future data change trends;

[0015] Data storage module: pre-processes the collected data, encrypts the processed data through encryption algorithm and sends it to the cloud server for storage.

[0016] The beneficial effects of the present invention are as follows: by setting the copying task as an exclusive task or a parallel task and performing priority allocation, the system resources can be used more efficiently and the efficiency of data collection can be improved; by using the priority evaluation model and the priority conflict resolution algorithm, the task priority can be dynamically adjusted, the system resources can be reasonably allocated and resource waste can be avoided; by introducing the state machine and the machine learning algorithm, the future data change trend can be predicted based on the collected historical data, so that the system can adapt to the changing data collection needs; by preprocessing and encrypting the collected data, the accuracy and security of the data can be ensured, and the data can be prevented from being tampered with or leaked during the transmission process; by real-time monitoring of the task execution and resource occupancy, abnormalities can be discovered and handled in a timely manner situation, improving the stability and reliability of the system; through data cleaning algorithms and data encryption algorithms, the quality and security of data processing can be improved, ensuring the accuracy and integrity of data during storage and analysis; by optimizing task scheduling and resource allocation, the delay in task execution can be reduced and the real-time performance of data acquisition can be improved; this method can dynamically adjust the scanning cycle and resource allocation strategy according to the system load and task characteristics, so that the system can adapt to different workloads and has good scalability; by optimizing task scheduling and resource allocation, unnecessary calculations and data transmission can be reduced, thereby reducing the energy consumption of the system; through the application of machine learning algorithms, the system can learn and adapt autonomously, improve the level of intelligence, and reduce human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a step diagram of the method of the present invention;

[0018] Figure 2 This is a system module diagram of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0020] One embodiment of the present invention, as Figure 1 As shown, a concentrator efficient data collection optimization method, the method comprising:

[0021] S1, setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station;

[0022] S2. After the system is started, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority;

[0023] S3. After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained through the task scheduling mode of the state machine;

[0024] S4. Analyze the historical data collected based on the correlation factors of the collection points and the machine learning algorithm to predict the future data change trend;

[0025] S5. Preprocess the collected data, encrypt the processed data through an encryption algorithm and send it to the cloud server for storage.

[0026] The working principle of the above technical solution is as follows: First, the data acquisition tasks (i.e., data reading tasks) are divided into two categories: exclusive tasks and parallel tasks. Exclusive tasks refer to those tasks that need to monopolize system resources during execution and cannot be interrupted by other tasks; while parallel tasks can be executed simultaneously without interfering with each other; a priority is assigned to each data reading task so that the system can arrange the execution order according to importance or urgency when facing multiple tasks; a state machine mechanism is introduced to manage the execution process of the first data reading task (i.e., the task to be read data configured by the acquisition master station). The state machine can clearly define the different execution stages of the task and its conversion conditions to ensure that the task is executed in a predetermined logical order; after the system is started, first check whether there is an exclusive task in the task pool; if there is an exclusive task, execute these tasks in order from high to low priority. This step ensures that critical tasks can obtain system resources first and avoid being delayed by other tasks; after all exclusive tasks are executed, the system obtains all parallel first data reading tasks from the task pool; using the task scheduling mode of the state machine, these parallel tasks will be started at the same time and executed in parallel. The state machine can monitor the execution status of each task to ensure synchronization and coordination between tasks; through parallel execution, the system can simultaneously obtain data items to be copied from multiple collection points to improve the efficiency of data collection; dynamically adjust the collection cycle according to the data volume, data update frequency, network transmission speed and other related factors of the collection point. This helps to balance the real-time nature of data collection and the consumption of system resources; use machine learning algorithms to analyze the collected historical data and predict future data change trends. Based on these prediction results, the system can further optimize the task scheduling strategy, for example, increase the collection frequency during periods of frequent data changes, and reduce the collection frequency when the data is stable; pre-process the collected data, including data cleaning, format conversion, etc., to ensure the accuracy and consistency of the data; encrypt the processed data through encryption algorithms to protect the privacy and security of the data; send the encrypted data to the cloud server for storage. The cloud server provides a highly available and scalable storage solution that can ensure the security and long-term preservation of data.

[0027] The effects of the above technical solution are as follows: by setting the copying task as an exclusive task or a parallel task, and assigning priorities according to the importance or urgency of the task, it can ensure that key tasks are handled first, avoid mutual waiting and conflicts between tasks, and thus improve the overall efficiency of data collection; the introduction of a state machine mechanism to manage the execution process of tasks can clearly define the different execution stages of tasks and their conversion conditions, ensure that tasks are efficiently executed in a predetermined logical order, and reduce waiting time and resource waste between tasks; the system first executes all exclusive tasks to ensure that these tasks are completed without interference. Subsequently, the remaining tasks are executed in parallel to make full use of system resources, improve resource utilization and system throughput; dynamically adjust the collection cycle according to the associated factors of the collection point (such as data volume, data update frequency, network transmission speed, etc.), balance the real-time nature of data collection and the consumption of system resources, and avoid waste of resources caused by excessive or insufficient collection; analyze the historical data collected based on the machine learning algorithm, predict future data change trends, and optimize the task scheduling strategy accordingly. It helps to discover data anomalies or change trends in advance, adjust the collection strategy in time, and ensure that the collected data is more accurate and reliable; pre-processing the collected data, including data cleaning and format conversion, can ensure the accuracy and consistency of the data. At the same time, the processed data is encrypted and stored through encryption algorithms to protect the privacy and security of the data and prevent data leakage or tampering; the system can dynamically adjust the collection cycle and task scheduling strategy according to the actual situation of the collection point and the data change trend, and enhance the flexibility and adaptability of the system. This helps to cope with the changes in demand in different collection scenarios and improve the scalability and maintainability of the system; sending the encrypted data to the cloud server for storage can take advantage of the elasticity and scalability of cloud services to provide the system with high availability and disaster recovery backup capabilities. At the same time, the cloud server can also provide convenient data access and management functions to reduce the operation and maintenance costs of the system.

[0028] In one embodiment of the present invention, the S1 includes:

[0029] S11. Determine the specific type of the exclusive task, distinguish different categories of parallel tasks, and set specific attributes for each task;

[0030] S12, dynamically adjust the priority according to the feedback during task execution through a multi-dimensional priority evaluation model;

[0031] S13, based on the priority conflict resolution algorithm, when multiple high-priority tasks request resources at the same time, intelligent decision-making is made according to the specific attributes of the tasks and the current state of the system;

[0032] S14. Design the state transition diagram and state behavior of the state machine, record the life cycle of the task, integrate the state monitoring module, track the task status in real time, and provide feedback to the task scheduler.

[0033] The working principle of the above technical solution is as follows: First, the system needs to clarify which tasks are exclusive tasks. These tasks usually have extremely high real-time requirements, such as power grid fault detection or emergency event reporting. They need to be executed immediately and monopolize system resources to ensure the accuracy and timeliness of data. The system sets special execution priorities and resource allocation strategies for these tasks by identifying the specific types of these tasks; for parallel tasks, the system classifies them according to their characteristics and requirements. For example, periodic data collection tasks (such as daily electricity consumption and monthly meter reading data) are usually executed at fixed time intervals, while non-real-time data collection (such as user feedback and equipment status monitoring) may be more flexible and do not require strict real-time requirements. The system sets specific attributes for each task, including task name, execution cycle, data source, and target data format, which help the system better understand and manage tasks; suppose there is a power grid monitoring system in which power grid fault detection is set as an exclusive task because it requires immediate response and fault handling. Daily electricity consumption data collection is a periodic parallel task that is executed once a day at a fixed time; the system adopts a priority evaluation model based on multiple dimensions, including the urgency, importance, and timeliness of the data. Urgency refers to the degree to which a task needs to be executed immediately; importance reflects the impact of a task on the overall performance of the system or user satisfaction; timeliness focuses on how long a task needs to be completed; the system dynamically adjusts the priority of tasks based on feedback during task execution (such as data change rate, network delay, etc.). For example, if the data change rate of a task suddenly accelerates, the system may automatically increase the priority of the task to ensure the real-time and accuracy of the data; when multiple high-priority tasks request resources at the same time, the system uses a priority conflict resolution algorithm to make intelligent decisions. The algorithm comprehensively considers the specific attributes of the task (such as urgency, importance, timeliness) and the current state of the system (such as resource occupancy, task queue length, etc.) to determine which task should be executed first; based on the results of intelligent decision-making, the system allocates necessary resources to the task, such as CPU time, memory space, network communication bandwidth, etc. This helps ensure that high-priority tasks can be executed smoothly while minimizing the impact on low-priority tasks; the system designs the state transition diagram and state behavior of the state machine for the task. The state machine is used to record the life cycle of a task, including states such as waiting, executing, pausing, resuming, and completing. Each state has a corresponding behavior definition. For example, the execution state may involve operations such as data reading, processing, and network transmission. The system integrates a state monitoring module to track the state changes of tasks in real time. When the task state changes (for example, from the waiting state to the execution state), the state monitoring module will provide feedback to the task scheduler. The task scheduler adjusts the task execution plan in a timely manner based on this feedback to ensure the smooth execution of tasks and the effective use of resources.

[0034] The effect of the above technical solution is: the system can clearly identify the specific types of exclusive tasks and parallel tasks, and set detailed attributes for each task. This not only helps the system to better understand and manage tasks, but also provides an accurate information basis for subsequent task scheduling and resource allocation; the multi-dimensional priority evaluation model comprehensively considers the urgency, importance and timeliness of the data, so that the system can dynamically adjust the priority of tasks according to the actual situation and needs of the task. This flexibility helps the system to make more reasonable scheduling decisions when facing complex and changeable acquisition tasks; the system can dynamically adjust the priority of tasks according to feedback during task execution (such as data change rate, network delay, etc.). This mechanism helps the system to prioritize the execution of key tasks when facing resource constraints or emergencies, thereby improving resource utilization and overall system performance; the priority conflict resolution algorithm can make intelligent decisions based on the specific attributes of the task and the current state of the system when multiple high-priority tasks request resources at the same time. It helps to avoid resource conflicts and waiting between tasks, and improve the response speed and resource utilization efficiency of the system; the state machine design provides a clear structure and process for the life cycle management of tasks. By recording different states of tasks (such as waiting, executing, pausing, resuming, completing, etc.), the system can track the state changes of tasks in real time and provide feedback to the task scheduler. This mechanism helps the system to adjust the task execution plan in time to ensure the smooth execution of tasks and the effective use of resources; the design of the state machine and the integration of the state monitoring module enable the system to clearly display the execution status and state changes of tasks, and improve the transparency and traceability of task execution. This helps system administrators or users to better understand the execution progress and results of tasks, so as to make more accurate decisions and evaluations; the technical solution enables the system to adapt to complex and changing task requirements through fine task classification, priority evaluation and dynamic adjustment mechanism, as well as intelligent conflict resolution algorithm and state machine design. This helps the system to quickly make adjustments and optimizations when facing new acquisition tasks or changes in requirements; the design ideas and implementation methods of the technical solution are universal and scalable, and can support the expansion and upgrade of the system. With the continuous advancement of technology and the continuous increase in acquisition needs, the system can continuously improve the performance and functions of the system by adding new task types, optimizing priority evaluation models, and improving conflict resolution algorithms.

[0035] In one embodiment of the present invention, the S12 includes:

[0036] Assign an urgency level to each task, based on the nature of the task and the importance of the data;

[0037] During the execution of tasks, the system collects feedback data in real time during the execution of tasks. Based on the collected feedback data, the system applies the priority dynamic adjustment algorithm to calculate the current priority of each task.

[0038] When multiple high-priority tasks request resources at the same time, the priority conflict resolution algorithm is applied to make intelligent decisions, dynamically adjust the calculation results of the algorithm according to the priority, and update the priority of the task in real time;

[0039] In the process of implementing the priority adjustment strategy, the system continuously monitors the execution status of tasks and the usage of resources. If it is found that the execution progress of a task is seriously lagging behind or the resource usage is too high, an alarm will be issued and feedback will be provided.

[0040] The working principle of the above technical solution is as follows: the system first assigns an urgency level to each task according to the nature of the task and the importance of the data. For example, the urgency level of the power grid fault detection task is set to the highest because it is directly related to the stable operation of the power system. For non-real-time data collection tasks such as user feedback or equipment status monitoring, the urgency level is relatively low; in addition to the urgency, the system also needs to consider the importance of the task. It is usually related to the degree of influence of the data associated with the task on the system or business decision. For example, daily electricity consumption data is crucial to the operational decision of the power company, so its importance level is set to a high level. For some non-critical equipment status monitoring data, its importance level is relatively low; for tasks with extremely high real-time requirements, such as power grid fault detection, its timeliness level is set to the highest to ensure that the data can be collected and processed in time. For periodic data collection tasks, such as monthly meter reading data, its timeliness level is relatively low because such data usually does not need to be updated in real time; in the process of executing the task, the system collects feedback data in real time during the execution of the task, including data change rate, network delay, etc. These data provide an important basis for the subsequent dynamic adjustment of priorities. Based on the collected feedback data, the system applies the priority dynamic adjustment algorithm to calculate the current priority of each task. For example, when the urgency level of a task increases, its priority should also increase accordingly. When the network delay increases and the task execution speed slows down, the priority of the task may need to be lowered to release resources for other more urgent tasks. When multiple high-priority tasks request resources at the same time, the system applies the priority conflict resolution algorithm to make intelligent decisions. The algorithm comprehensively considers the specific attributes of the task (such as urgency, importance, timeliness) and the current state of the system (such as resource occupancy, task execution progress, etc.) to make the best decision and ensure that resources can be efficiently and fairly allocated to each task. According to the calculation results of the priority dynamic adjustment algorithm, the system updates the priority of the task in real time and synchronizes the newly calculated priority with the priority in the task scheduler to ensure that the task scheduler can schedule tasks based on the latest priority information. In the process of implementing the priority adjustment strategy, the system continuously monitors the execution of tasks and the occupancy of resources. This helps the system to promptly identify problems or resource bottlenecks during task execution; if a task is found to be seriously behind schedule or using too many resources, the system will issue an alarm and provide feedback. This helps system administrators or users to promptly understand the status of task execution and take appropriate measures to intervene or optimize.

[0041] The effect of the above technical solution is: by assigning urgency, importance and timeliness levels to each task, refined classification and management of tasks are achieved. This classification method not only takes into account the direct nature of the task, but also deeply analyzes the impact of the data associated with the task on the system or business decision, thereby ensuring the comprehensiveness and accuracy of task management; the system can collect feedback data in the task execution process in real time, such as data change rate and network delay, and apply the priority dynamic adjustment algorithm based on these data to calculate the current priority of each task. This mechanism enables the system to flexibly respond to various changes in the task execution process and ensure that key tasks can be handled in a timely and effective manner. When multiple high-priority tasks request resources at the same time, the technical solution applies a priority conflict resolution algorithm for intelligent decision-making. The algorithm comprehensively considers the specific attributes of the task and the current state of the system to ensure that resources can be efficiently and fairly allocated to each task. This mechanism avoids task execution delays or failures caused by resource competition and improves the overall performance and stability of the system. In the process of implementing the priority adjustment strategy, the system continuously monitors the execution of tasks and the occupancy of resources. Once it is found that the execution progress of a task is seriously lagging behind or the resource occupancy is too high, the system can immediately issue an alarm and provide feedback. This real-time monitoring and feedback mechanism helps system administrators or users to find problems in time and take appropriate measures to intervene or optimize. By dynamically adjusting the priority of tasks and efficiently allocating resources, this technical solution significantly improves the flexibility and response speed of the system. This enables the system to better adapt to complex and changing task requirements, ensuring that key tasks can be processed in the first place, thereby improving the overall efficiency of the system and user satisfaction.

[0042] In one embodiment of the present invention, the S2 includes:

[0043] S21, periodically scan the task pool, identify all exclusive tasks to be executed, and obtain the current resource occupancy of the concentrator; the resource occupancy is calculated by the following formula:

[0044]

[0045] in, Indicates the occupied amount of the i-th type of resources; represents the total amount of resources of the i-th category; represents the importance value weight of the i-th type of resources; represents the utilization efficiency factor of the i-th type of resource; n represents the total number of resource types;

[0046] S22, applying a sorting algorithm to prioritize the exclusive tasks, and reserving necessary system resources for the exclusive tasks to be executed;

[0047] S23. Start the exclusive task based on the priority order, monitor the task execution progress in real time, and handle the exceptions that may occur during the execution process.

[0048] The working principle of the above technical solution is as follows: the system sets a fixed time interval to periodically scan the task pool. This step is intended to identify all currently pending exclusive tasks; during the scanning process, the system simultaneously obtains the current resource occupancy of the concentrator (or task execution unit), including CPU usage, memory usage, network bandwidth usage, etc.; by analyzing the resource occupancy, the system determines which resources are unnecessary or can be released to ensure that system resources are in the best utilization state before executing the exclusive task; necessary resource release operations may include terminating non-critical tasks, reclaiming unused memory space, optimizing network connections, etc.; the system applies an efficient sorting algorithm (such as quick sort, heap sort, etc.) to prioritize the identified exclusive tasks. The sorting can be based on factors such as the urgency, importance, and expected execution time of the task; after the sorting is completed, the system reserves the necessary system resources for the exclusive task to be executed. These resources include CPU time slices, memory space, network bandwidth, etc., to ensure that the task will not be disturbed by insufficient resources during execution; resource reservation operations may involve adjusting system resource allocation strategies, creating resource pools, setting resource access permissions, etc.; the system will also dynamically adjust resource allocation according to the execution requirements of the task and the resource reservation situation to optimize resource utilization and task execution efficiency; the system will start exclusive tasks in order of priority. Before the task is started, the system will conduct a final resource check and preparation to ensure that the task can be executed in the best state; during the task execution, the system monitors the execution progress and resource usage of the task in real time. This includes monitoring each stage of task execution, recording key events, detecting abnormal situations, etc.; once an abnormality is detected (such as data reading failure, network interruption, etc.), the system will immediately trigger the corresponding exception handling mechanism. This may include retrying operations, rolling back transactions, recording error logs, notifying administrators, etc.; the system will also decide whether to suspend or terminate the task, or adjust the execution plan of subsequent tasks based on the results of exception handling and the importance of the task.

[0049] The effect of the above technical solution is: by periodically scanning the task pool, the system can timely understand the current exclusive tasks to be executed, and release unnecessary resource occupation in combination with the resource occupation of the concentrator. This step ensures that the system is in the best utilization state of resources before executing the exclusive task, avoiding resource waste and conflict; the system reserves necessary system resources for the exclusive task to be executed, such as CPU time slices, memory space, network bandwidth, etc., to ensure that the task will not be disturbed due to insufficient resources during execution. At the same time, the system can also make dynamic adjustments according to the execution requirements of the task and the resource reservation situation to further optimize resource utilization; the sorting algorithm is applied to prioritize the exclusive tasks, so that the system can start the tasks in order of priority. This mechanism ensures that key tasks can be executed first, improving the efficiency and quality of task execution; the system monitors the execution progress and resource usage of the task in real time, and immediately triggers the corresponding exception handling mechanism once an abnormality is detected. It ensures that tasks are executed in an undisturbed environment, and can respond to and handle abnormal situations in a timely manner to avoid task failure or system crashes; through periodic scanning and resource release, the system reduces the possibility of resource conflicts and avoids task execution failure or system crashes due to insufficient resources; the system has a complete exception handling mechanism that can respond to and handle various abnormal situations, such as data reading failure, network interruption, etc. This enhances the stability and reliability of the system and ensures that tasks can be completed smoothly; by optimizing resource utilization and priority sorting, the system improves the efficiency and quality of task execution, allowing users to obtain the required results and services faster; the system monitors the execution progress and abnormal situations of tasks in real time, and can promptly feedback relevant information to users. This enhances users' trust and satisfaction with the system and improves the user experience. The above formula calculates the total resource usage by accumulating different types of resources (such as CPU, memory, disk I / O, etc.), which helps to fully understand the current load status of the system; through Parameters can assign different weights to different types of resources, reflecting their relative importance in the system. For example, in some applications, CPU usage may be more important than memory usage, so the CPU can be given a higher weight. The factor allows the model to take into account the efficiency of resource use. Even if resource utilization is high, if it is used inefficiently, the actual impact may not be severe. This mechanism helps to more accurately assess the actual burden of resources; the factor in the formula The formula introduces a nonlinear effect, which means that as the resource occupancy ratio increases, its impact on the total occupancy will gradually increase. This reflects that when resources are tight in reality, an extra bit of occupancy may lead to a greater performance degradation. Because the formula is calculated based on real-time data, the resource occupancy can be dynamically updated according to changes in the system's operating status, which is very important for real-time scheduling and resource management. By providing a quantitative resource occupancy indicator, the formula can help system administrators or automated systems make more informed decisions, such as when to start new tasks or how to reallocate resources to optimize system performance. When resource occupancy reaches a certain threshold, the results of the formula can be monitored to trigger alarms or take automatic measures to prevent system overload or performance degradation. During task scheduling, resources can be reasonably reserved based on the calculated resource occupancy to ensure that high-priority tasks have sufficient resources available, thereby improving the overall system's response speed and service quality.

[0050] In one embodiment of the present invention, the S21 includes:

[0051] S211, based on a preset time interval, regularly perform a comprehensive scan on the task pool, and dynamically adjust the time interval according to changes in system load and task characteristics;

[0052] S212, during the scanning process, obtaining the current status of each task, and monitoring the resource usage of the concentrator in real time through the resource detector;

[0053] S213, based on the acquired resource occupancy data, evaluate the current resource usage; according to the result of the resource occupancy evaluation, formulate and implement a resource optimization and release strategy;

[0054] S214, identifying all exclusive tasks to be executed in the task pool, and reserving necessary system resources for the exclusive tasks to be executed according to demand;

[0055] S215: Before executing the exclusive task, perform a preprocessing operation, and in the process of executing resource optimization release and exclusive task identification, perform exception handling through an exception handling mechanism.

[0056] The working principle of the above technical solution is as follows: a preset time interval is set inside the system to perform a full scan of the task pool at a regular interval. This time interval is not fixed, but is dynamically adjusted according to changes in system load and task characteristics. For example, when the system load is high or the task complexity increases, the scanning interval may be shortened to ensure that the system can respond to task changes in a timely manner; during the scanning process, the system obtains the current status of each task through an internal interface or API, including key information such as whether the task is in a pending state, the priority of the task, and the estimated execution time. At the same time, the system uses a resource detector to monitor the resource usage of the concentrator in real time, such as CPU usage, memory usage, network bandwidth usage, etc., to ensure comprehensive control of system resources; based on the obtained resource usage data, the system conducts an in-depth evaluation of the current resource usage. This step aims to identify which resources are occupied by unnecessary tasks or processes, and determine whether these resources can be released to support the execution of exclusive tasks. According to the evaluation results, the system formulates and implements resource optimization and release strategies, such as terminating unnecessary processes, reclaiming unused memory, adjusting network bandwidth allocation, etc., to ensure that system resources are in the best utilization state before executing exclusive tasks; in the task pool, the system identifies all exclusive tasks to be executed and reserves necessary system resources for them according to the needs of these tasks. These resources include CPU time slices, memory space, and network bandwidth, etc., to ensure that exclusive tasks will not be disturbed by insufficient resources during execution. The quantity and quality of reserved resources should be reasonably configured according to the characteristics of the task and the expected load to ensure that the task can be executed smoothly; before executing the exclusive task, the system performs a series of preprocessing operations, such as loading necessary library files, initializing necessary data structures, etc., to ensure that the task can be started smoothly. At the same time, in the process of executing resource optimization and release and identifying exclusive tasks, the system handles exceptions through the exception handling mechanism. This includes recording exception information, triggering alarms, taking remedial measures, etc., to ensure that the system can respond quickly and resume normal operation when encountering abnormal situations. For example, suppose that the current task pool of the system contains multiple tasks, among which task A and task B are exclusive tasks. According to the workflow of S21, the system will first perform a periodic scan to identify Task A and Task B as exclusive tasks to be executed. Then, the system monitors the resource usage of the current concentrator through the resource detector and finds that the CPU usage is high and the memory usage is close to the upper limit. Based on this information, the system evaluates that the current resources are insufficient to support the execution of Task A and Task B at the same time. Therefore, the system decides to terminate some unnecessary processes, recycle unused memory, and adjust network bandwidth allocation to free up sufficient resources. Next, the system reserves the necessary system resources for Task A and Task B and performs preprocessing operations. During the execution process, if the system encounters any abnormal situation, such as data reading failure or network interruption, it will immediately trigger the exception handling mechanism, record the exception information, and take corresponding remedial measures.

[0057] The effect of the above technical solution is: through timed scanning based on preset time intervals and dynamic adjustment according to changes in system load and task characteristics, the system can manage resources more effectively. This flexibility ensures that the frequency of resource scanning matches the actual needs of the system, and neither consumes system resources too frequently nor misses important tasks or resource changes due to long scanning intervals; the system can monitor resource usage in real time and evaluate the current resource usage. This evaluation not only helps to identify which resources are occupied by unnecessary tasks or processes, but also formulates and implements resource optimization and release strategies based on the evaluation results. This ensures the effective use of resources, reduces resource waste, and provides necessary resource support for the execution of exclusive tasks; by fully scanning the task pool, the system can accurately identify all exclusive tasks to be executed and reserve necessary system resources according to task requirements. This reservation mechanism ensures that exclusive tasks will not be disturbed due to insufficient resources during execution, thereby improving the efficiency and success rate of task execution; before executing exclusive tasks, the system performs preprocessing operations, such as loading necessary library files and initializing data structures. These operations help reduce delays and errors during task execution, and improve the efficiency and stability of task execution. In the process of optimizing resource release and identifying exclusive tasks, the system handles exceptions through an exception handling mechanism. This includes recording exception information, triggering alarms, and taking remedial measures. This mechanism ensures that the system can respond quickly when encountering abnormal situations, avoiding task failures or system crashes, thereby enhancing the stability and reliability of the system. By optimizing resource management and improving task execution efficiency, the system can respond to user requests faster and provide a smoother user experience. Users do not need to wait too long to see the results of task execution, which improves user satisfaction and loyalty.

[0058] In one embodiment of the present invention, the S211 includes:

[0059] The system sets a basic scanning cycle, which is the initial time interval for the system to perform a comprehensive scan of the task pool; the system monitors and analyzes the current system load in real time, and analyzes the characteristics of the tasks in the task pool;

[0060] Based on the analysis results of system load and task characteristics, evaluate whether the current scanning cycle is feasible; if the system load is high or there are a large number of high-priority and resource-intensive tasks in the task pool, shorten the scanning cycle;

[0061] According to the analysis results of system load and task characteristics, the system dynamically adjusts the scanning cycle; if the system load is light and there are fewer tasks in the task pool or the resource demand is low, the scanning cycle is extended;

[0062] On the contrary, if the system is heavily loaded or there are a large number of high-priority and resource-demanding tasks in the task pool, the scanning period is shortened;

[0063] Through the preset scanning cycle adjustment threshold, when the change of system load or task characteristics exceeds this threshold, the scanning cycle adjustment is triggered.

[0064] The working principle of the above technical solution is as follows: the system first sets a basic scanning cycle, which is the initial time interval for the system to conduct a comprehensive scan of the task pool. This basic cycle is determined by the system based on experience or preset configuration, and is intended to provide a reasonable and balanced scanning frequency, which is neither too frequent to avoid unnecessary system overhead nor too sparse to avoid missing changes in the status of key tasks; next, the system monitors the current system load in real time. This includes monitoring of key indicators such as CPU usage, memory occupancy, and disk I / O, which can reflect the overall operation status and resource utilization of the system; at the same time, the system analyzes the characteristics of tasks in the task pool. This includes information such as the priority of the task, the expected execution time, and resource requirements, which helps the system understand the importance of the task and the degree of dependence on resources; based on the analysis results of the system load and task characteristics, the system evaluates whether the current scanning cycle is still feasible. If the system load is high, or there are a large number of high-priority and resource-demanding tasks in the task pool, this means that the system needs to monitor the task status more frequently to ensure the timely execution of key tasks. In this case, the system will shorten the scanning cycle to respond to changes in task status more promptly; on the contrary, if the system load is light, and there are fewer tasks in the task pool or lower resource requirements, this means that the current pressure on the system is small, and the scanning cycle can be extended to reduce unnecessary system overhead; in order to avoid frequent changes in the scanning cycle causing system instability, the system sets an adjustment threshold. This threshold defines the extent to which the system load or task characteristics change before the adjustment of the scanning cycle is triggered. The system will only adjust the scanning cycle when the change in the system load or task characteristics exceeds this threshold. This mechanism ensures that the adjustment of the scanning cycle is orderly and controllable, avoiding system fluctuations caused by frequent adjustments.

[0065] The effect of the above technical solution is: by real-time monitoring of system load and task characteristics, and dynamically adjusting the scanning cycle according to the analysis results of this information, the system can adapt to different operating conditions more flexibly. When the system load is heavy or there are a large number of high-priority and resource-demanding tasks in the task pool, shortening the scanning cycle can ensure that the system responds to changes in task status more promptly, thereby improving the responsiveness and execution efficiency of the system; when the system load is light and there are fewer tasks in the task pool or the resource demand is low, extending the scanning cycle can reduce unnecessary scanning operations, thereby reducing the resource consumption of the system. This resource optimization not only helps to save energy, but also reduces the system overhead caused by frequent scanning and improves the overall performance of the system; through the preset scanning cycle adjustment threshold, the system can avoid frequent changes in the scanning cycle. This mechanism ensures that the system is orderly and controllable when adjusting the scanning cycle, avoiding system instability or performance fluctuations caused by frequent adjustments; when the system detects that there are a large number of high-priority and resource-demanding tasks in the task pool, by shortening the scanning cycle, the system can identify and process these tasks more quickly. This helps ensure that high-priority tasks are executed in a timely manner, improving the reliability and satisfaction of task execution; this technical solution realizes intelligent analysis of system load and task characteristics, and automatically adjusts the scanning cycle based on the analysis results. This intelligent management method not only improves the automation level of the system, but also enables the system to make the best decision based on actual conditions, thereby improving the overall management level and operation efficiency of the system.

[0066] In one embodiment of the present invention, the S23 includes:

[0067] Assign a unique task identifier (TaskID) to the exclusive task to be started, initialize its status information, and monitor the execution progress of the exclusive task in real time;

[0068] Based on real-time monitoring data, dynamically adjust resource allocation during task execution; identify and report various anomalies that may occur during task execution through anomaly detection mechanisms;

[0069] When an anomaly is detected, the system responds and takes appropriate measures, and when the exclusive task is successfully executed, the completion status of the task is identified and confirmed;

[0070] After confirming that the task is completed, the system releases all resources previously allocated for the task, including closing processes or threads, reclaiming memory, releasing network interfaces, etc. At the same time, the system should also update the task status information so that the administrator can understand the final execution result of the task.

[0071] The working principle of the above technical solution is as follows: the system assigns a unique TaskID to the exclusive task to be started, ensuring the uniqueness and traceability of the task during the entire execution process; the system initializes the task status information, including task name, execution time, expected completion time, current execution stage, etc., to provide basic data for subsequent task management and monitoring; the system monitors the execution progress of the exclusive task in real time, paying attention to key performance indicators such as CPU usage, memory usage, network transmission speed, etc., to evaluate the execution efficiency and resource requirements of the task; based on the monitoring data, the system dynamically adjusts the resource allocation during the task execution process. For example, when it is detected that insufficient resources cause slow task execution, the system attempts to allocate more resources to it; conversely, when the task is executed too quickly and the resource usage is too high, the system reduces resource allocation to optimize resource usage; through the anomaly detection mechanism, the system identifies and reports various anomalies that may occur during task execution, such as data reading failure, network interruption, resource competition conflict, etc.; when an anomaly is detected, the system takes appropriate response measures, such as restarting the task, rolling back to the previous stable state, releasing occupied resources, etc., to restore the normal execution of the task or minimize the loss; the system records the anomaly information and provides it to the administrator for further analysis, helping the administrator to understand the cause of the anomaly, optimize the system configuration and task management strategy; when the exclusive task is successfully executed, the system identifies and confirms the completion status of the task. This includes checking the correctness and completeness of the task output, and verifying whether the task has achieved the expected execution effect; after confirming that the task is completed, the system releases all resources previously allocated to the task, including closing processes or threads, reclaiming memory, releasing network interfaces, etc. At the same time, the system updates the task status information to ensure that the administrator can understand the final execution result of the task and the release of resources.

[0072] The effects of the above technical solution are: assigning a unique TaskID to the exclusive task to be started ensures the uniqueness and traceability of the task throughout its life cycle, making it easier for the system to manage the task accurately; by initializing the task status information, including the task name, execution time, expected completion time, current execution stage, etc., the system can fully understand the initial status of the task, providing a basis for subsequent task monitoring and management; the system monitors the execution progress and key performance indicators of the exclusive task in real time, such as CPU usage, memory usage, network transmission speed, etc., and dynamically adjusts resource allocation based on monitoring data. This flexibility ensures that the task can be executed efficiently when resources are sufficient and can get the necessary support when resources are tight, thereby improving the overall performance and resource utilization of the system; through the anomaly detection mechanism, the system can promptly discover and report various anomalies that may occur during the execution of the task, such as data reading failure, network interruption, resource competition conflict, etc. This timeliness helps the system to respond to anomalies quickly and reduce the risk of task failure; the system can take appropriate measures to deal with anomalies, such as restarting the task, rolling back to the previous stable state, releasing occupied resources, etc. These measures help to restore the normal execution of tasks and reduce the impact of exceptions on the system; when the exclusive task is successfully executed, the system identifies and confirms the completion status of the task, including checking the correctness and completeness of the task output, and verifying whether the task has achieved the expected execution effect. This confirmation mechanism ensures the integrity and accuracy of the task; after confirming that the task is completed, the system releases all resources previously allocated to the task and updates the task status information. This resource release and status update mechanism helps the system recycle resources, reduce resource waste, and provide administrators with the final execution results of the task; through the beneficial effects of the above aspects, technical solution S23 improves the stability and reliability of the system. The system can manage tasks more accurately, allocate resources more flexibly, and handle exceptions more promptly, thereby ensuring the normal operation of the system and the smooth execution of tasks.

[0073] In one embodiment of the present invention, S3 includes:

[0074] S31, retrieving all parallel first copy-reading tasks from the task pool, and grouping the parallel tasks according to the attributes of the tasks;

[0075] S32, based on the state machine scheduling strategy, the parallel tasks are executed simultaneously according to the preset priorities and resource allocation schemes, and the execution status of the parallel tasks is monitored in real time;

[0076] S33. Acquire the data items to be copied during the execution of the parallel tasks through the data acquisition strategy.

[0077] The working principle of the above technical solution is as follows: the system first retrieves all parallel first copying tasks from the task pool, which usually have similar execution requirements, such as data source, data format and execution cycle; the system performs attribute analysis on the retrieved tasks, including the reliability of the data source, the compatibility of the data format and the stability of the execution cycle; based on the analysis results of the task attributes, the system groups the parallel tasks. The purpose of grouping is to classify tasks with similar attributes or requirements together for more efficient resource allocation and task scheduling. The grouping strategy may include grouping by data source, grouping by data format or grouping by execution cycle; the system sets a preset priority and resource allocation scheme for each group. The priority and resource allocation scheme may be based on factors such as the urgency of the task, resource requirements and the current load of the system; the system adopts a scheduling strategy based on a state machine to enable parallel tasks to be executed simultaneously according to the preset priority and resource allocation scheme. The state machine scheduling strategy can ensure that tasks are executed in the correct state and avoid conflicts and resource competition between tasks; the system monitors the execution status of parallel tasks in real time, including task progress and resource usage. By monitoring data, the system can timely discover potential problems in the task execution process, such as insufficient resources, slow task execution or task failure; based on real-time monitoring data, the system can dynamically adjust the scheduling strategy, such as reallocating resources, adjusting task priorities or rearranging the order of task execution, to ensure efficient task execution and resource optimization; during the execution of parallel tasks, the system identifies the data items to be copied according to the task requirements. The data items to be copied may include data from different data sources, data with a specific format or data that needs to be updated regularly; the system adopts appropriate data acquisition strategies, such as data reading, data conversion or data synchronization, to obtain the data items to be copied from the data source. The selection of data acquisition strategies may be based on factors such as data availability, data real-time and data security; after acquiring the data, the system verifies the data to ensure the correctness and integrity of the data. Verification may include data format verification, data range verification or data consistency verification, etc.; retrieve parallel tasks from the task pool and group them according to task attributes; based on the preset priority and resource allocation plan, adopt the state machine scheduling strategy to make parallel tasks execute simultaneously, and monitor the task execution status in real time; during the execution of parallel tasks, identify and obtain the data items to be copied according to the task requirements, and perform data verification.

[0078] The effect of the above technical solution is: by grouping parallel tasks, the system can more accurately identify the similarities and differences between tasks, so as to make more reasonable resource allocation. This helps to avoid the situation where resources are over-allocated to some tasks while other tasks are under-resourced, thereby improving the overall resource utilization; the grouped tasks have clearer attributes and requirements, which makes task scheduling simpler and more efficient. The system can formulate a specific scheduling strategy for each group to reduce the complexity and uncertainty in the scheduling process; the scheduling strategy based on the state machine can ensure that the task is executed in the correct state and avoid conflicts and resource competition between tasks. This helps to reduce errors and failures in the task execution process and improve the reliability of task execution; by real-time monitoring of the execution status of parallel tasks, the system can timely discover potential problems in the task execution process and take corresponding adjustment measures. This helps to ensure that tasks can be efficiently executed according to the preset priority and resource allocation scheme, reducing delays and waiting time in the task execution process; the scheduling strategy based on the state machine also allows the system to dynamically adjust the scheduling strategy according to the real-time monitoring data to adapt to changes in the task execution process. This helps the system to better deal with emergencies and uncertainties and improve the flexibility and adaptability of the system; through the data acquisition strategy, the system can accurately obtain the data items to be copied during the execution of parallel tasks. This helps ensure the correctness and integrity of the data and reduce the risk of data errors and omissions; the selection and optimization of data acquisition strategies can significantly improve the efficiency of data acquisition. The system can select the most appropriate data acquisition method based on factors such as data availability, real-time and security, thereby speeding up data acquisition and reducing delays and waiting time during data acquisition; through the comprehensive application of task grouping, state machine-based scheduling strategies and data acquisition strategies, the system can handle parallel reading tasks more efficiently and improve the system's throughput. This helps the system handle more tasks in a short period of time and meet higher business needs; by optimizing resource allocation and task scheduling, the system can reduce unnecessary resource waste and duplication of work, thereby reducing the operating costs of the system. At the same time, improving task execution efficiency and data acquisition efficiency also helps reduce the workload of system maintenance and management, further reducing system costs.

[0079] In one embodiment of the present invention, the S31 includes:

[0080] The system performs a comprehensive scan of the task pool, retrieves all the first copying tasks to be executed, and analyzes the properties of each retrieved task;

[0081] Based on the analysis results of task attributes, tasks are classified. Based on the task classification, parallel tasks are further grouped according to the execution mode and resource requirements of the tasks;

[0082] When grouping, the grouping is based on reference factors, and after the grouping is completed, the grouped tasks are further optimized.

[0083] The working principle of the above technical solution is as follows: First, the system performs a comprehensive and in-depth scanning operation on the task pool, aiming to retrieve all the first reading tasks currently to be executed. These tasks may originate from different business needs or data sources, but they are all in a waiting state; for each retrieved task, the system will record its key attribute information in detail, including but not limited to data sources (such as electricity meters, water meters, gas meters and other specific devices), data formats (such as text, binary, JSON, etc.), execution cycles (such as real-time, daily, weekly and other periodic requirements) and possible dependencies between tasks; the system conducts an in-depth analysis of the extracted task attributes to identify the commonalities and differences between tasks. This step is crucial to understanding the characteristics and requirements of tasks; based on the results of attribute analysis, the system has formulated a series of classification criteria. For example, tasks can be preliminarily divided into different categories based on the similarity of data sources or data formats; on the basis of task classification, the system further considers the execution mode and resource requirements of tasks to establish the basic principles for parallel task grouping. Parallel tasks refer to a collection of tasks that can be executed simultaneously without interfering with each other. When grouping, the system comprehensively considers multiple key reference factors, such as task priority, resource requirements (such as the number of CPU cores, memory size, network bandwidth, etc.), execution cycle, and dependencies between tasks. These factors jointly determine the rationality and efficiency of task grouping. According to the established grouping principles and reference factors, the system performs detailed grouping operations on parallel tasks. For example, tasks with high priority and similar resource requirements are grouped into the same group so that they can be allocated the same resources and executed simultaneously. After the grouping is completed, the system will further optimize the grouped tasks. For example, according to the execution cycle and expected execution time of the task, the system can intelligently adjust the execution order of the task to reduce resource competition and waiting time and improve the overall execution efficiency. In addition, the system will reserve corresponding resources for each group of tasks based on the grouping results and resource requirements, and make full preparations for subsequent task scheduling.

[0084] The effect of the above technical solution is: through a comprehensive scan of the task pool and a detailed attribute analysis of each task, the system can clearly understand the characteristics of each task, including data source, data format, execution cycle and dependency, etc. This helps the system to grasp the overall situation of the task more accurately and provide a solid foundation for subsequent task classification and grouping; based on the analysis results of task attributes, the system can reasonably classify and group tasks. Classification helps to classify tasks with similar attributes into one category, which is convenient for resource allocation and task scheduling optimization; grouping further considers the execution mode and resource requirements of tasks to ensure that parallel tasks can be executed efficiently and without interference; in the grouping process, the system fully considers the resource requirements of tasks, such as CPU cores, memory, network bandwidth, etc. By grouping tasks with similar resource requirements into the same group, the system can allocate resources more reasonably and avoid resource waste and excessive competition; after grouping, the system will further optimize the tasks, such as adjusting the order of task execution according to the execution cycle and expected execution time. This helps to reduce resource competition and waiting time and improve the overall utilization efficiency of resources; parallel task grouping ensures that tasks that can be executed simultaneously without interfering with each other can be executed simultaneously. This greatly improves the efficiency of task execution and shortens the overall task completion time; through reasonable task grouping and execution order adjustment, the system can handle more tasks at the same time, thereby enhancing the system's throughput and processing capacity; this technical solution has high flexibility and scalability. As task attributes change or new tasks are added, the system can reclassify and group tasks to adapt to new needs; because tasks are clearly classified and grouped, the system can locate problems more quickly and take corresponding solutions during maintenance and management; through efficient task management and resource allocation, the system can complete tasks more quickly, thereby meeting users' timeliness needs; the system can clearly display the allocation and use of resources, so that users have a more intuitive understanding of the use of system resources.

[0085] In one embodiment of the present invention, the S4 includes:

[0086] S41. Continuously monitor key indicators of data collection points, including data volume, data update frequency, and network transmission speed;

[0087] S42. Conduct in-depth analysis of monitoring data based on data analysis algorithms and predict future data change trends and collection needs;

[0088] S43, based on the data analysis result, setting a dynamic adjustment strategy for the acquisition period, wherein the dynamic adjustment strategy includes adjusting the length of the period, the frequency of adjustment, and the direction of adjustment;

[0089] S44. By adjusting the collection cycle strategy, feedback data is continuously collected, and the task scheduling algorithm is improved according to the dynamic adjustment result of the collection cycle; and the allocation of system resources is dynamically adjusted according to the result of task scheduling.

[0090] The working principle of the above technical solution is as follows: the system continuously monitors the key indicators of data collection points, including data volume, data update frequency, and network transmission speed. These indicators can reflect the activity of data collection points and the efficiency of data transmission; through sensors or log recording systems deployed at data collection points, the values ​​of key indicators are collected in real time and transmitted to the central processing unit for analysis; the system uses data analysis algorithms (such as time series analysis, machine learning models, etc.) to conduct in-depth analysis of monitoring data to reveal the laws and trends of data changes; based on the trend of historical data and current data changes, the system builds a prediction model to predict future data change trends and collection needs. The prediction results include the increase or decrease in data volume, changes in data update frequency, and fluctuations in network transmission speed; based on the data analysis results, the system formulates a dynamic adjustment strategy for the collection cycle. The adjustment strategy includes the length of the adjustment cycle (such as adjusting from once an hour to once every half an hour), the frequency of adjustment (such as adjusting the frequency of adjustment according to the speed of data change), and the direction of adjustment (increasing or decreasing the number of collection times); the system applies the adjustment strategy to the data collection point, and adjusts the execution frequency and time interval of the collection task to adapt to the data change trend and changes in collection needs; after implementing the collection cycle adjustment strategy, the system continuously collects feedback data, including the adjusted data collection efficiency, resource usage, and task execution status; based on the feedback data, the system improves the task scheduling algorithm to optimize the task execution order, reduce resource competition and waiting time. The improved algorithm can schedule data collection tasks more efficiently and improve the overall data collection efficiency; according to the results of task scheduling, the system dynamically adjusts the allocation of system resources. For example, when the amount of data at a data collection point surges, the system can automatically increase the computing resources and network bandwidth allocated to the point to ensure the smooth progress of data collection.

[0091] The effects of the above technical solutions are as follows: by continuously monitoring key indicators such as the data volume, data update frequency, and network transmission speed of data collection points, the system can understand the operating status and performance bottlenecks of data collection points in real time; based on the data analysis algorithm, the system can accurately predict future data change trends and collection needs through in-depth analysis of monitoring data, thereby avoiding blindness and redundancy in data collection; based on the prediction results, the system can dynamically adjust the length, frequency, and direction of the collection cycle to ensure the timeliness and accuracy of data collection, while reducing unnecessary resource consumption; through the continuous implementation of the collection cycle adjustment strategy, the system can collect rich feedback data, which provides strong support for the improvement of the task scheduling algorithm; based on the feedback data, the system can continuously optimize the task scheduling algorithm, improve the efficiency and accuracy of task execution, and reduce resource competition and waiting time; according to the results of task scheduling, the system can dynamically adjust the system resources. The allocation of sources ensures efficient use of resources and load balancing, avoiding resource waste and bottlenecks; the technical solution is highly flexible and scalable, and can be adjusted and optimized accordingly with the increase in data collection points, the growth of data volume, and changes in business needs; since the system can continuously collect feedback data and improve and optimize accordingly, maintenance and upgrade work becomes simpler and more efficient; by optimizing the data collection cycle and task scheduling algorithm, the system can provide more accurate and timely data, thereby improving user experience and satisfaction; the system can clearly display the usage and allocation strategy of resources, so that users have a more intuitive understanding of the operating status of the system; by dynamically adjusting the collection cycle and resource allocation strategy, the system can reduce unnecessary resource consumption and waste, thereby reducing operating costs; the optimized task scheduling algorithm and resource allocation strategy can ensure efficient use of resources and further improve the overall performance and cost performance of the system.

[0092] In one embodiment of the present invention, the S44 includes:

[0093] According to the collection cycle dynamic adjustment strategy formulated in S43, implement specific collection cycle adjustments, including increasing or decreasing the length of the collection cycle, adjusting the adjustment frequency of the collection cycle, and making corresponding adjustments according to the data change trend and the change direction (increase or decrease) of the collection demand;

[0094] While implementing the collection cycle adjustment, continuously monitor the key indicators of the data collection points, and collect feedback data after the collection cycle adjustment strategy is implemented;

[0095] Conduct in-depth analysis of the collected feedback data, evaluate the effectiveness of the collection cycle adjustment strategy, and optimize the task scheduling algorithm based on the analysis results of the feedback data;

[0096] Based on the results of task scheduling and analysis of feedback data, the allocation of system resources is dynamically adjusted, and in the process of adjusting the allocation of system resources, the usage of resources is monitored in real time.

[0097] The working principle of the above technical solution is as follows: according to the dynamic adjustment strategy of the collection cycle formulated in S43, the system implements specific collection cycle adjustment, including increasing or decreasing the length of the collection cycle, adjusting the adjustment frequency of the collection cycle, and making corresponding adjustments according to the data change trend and the change direction (increase or decrease) of the collection demand; while implementing the collection cycle adjustment, the system continuously monitors the key indicators of the data collection points, such as data volume, data update frequency and network transmission speed. These monitoring data are used to ensure that the adjusted collection cycle can meet the actual data collection needs and avoid adverse effects on system performance; after the collection cycle adjustment strategy is implemented, the system collects feedback data, including the actual data volume, data update frequency, network transmission speed and system resource usage of each data collection point under the adjusted collection cycle; the collected feedback data is deeply analyzed to evaluate the effect of the collection cycle adjustment strategy. This includes comparing the data differences before and after the adjustment, analyzing the impact of the adjustment strategy on system performance and data collection quality, and identifying possible problems and improvement points; based on the analysis results of the feedback data, the system optimizes the task scheduling algorithm. This includes optimizing task scheduling strategies to improve the execution efficiency and accuracy of data collection tasks; adjusting task priorities to ensure that key data collection tasks are given priority; and introducing new task scheduling algorithms or models to better adapt to changes in data collection needs; through algorithm adjustment, the system can more reasonably arrange the execution order and time of data collection tasks, reduce resource competition and waiting time, and improve overall data collection efficiency; based on the results of task scheduling and analysis of feedback data, the system dynamically adjusts the allocation of system resources. This includes increasing or decreasing the allocation of resources such as the number of CPU cores, memory size, and network bandwidth to better meet the needs of data collection tasks; in the process of adjusting system resource allocation, the system monitors resource usage in real time. This helps to promptly discover and solve problems of unreasonable resource allocation or resource bottlenecks, ensuring stable and efficient system performance.

[0098] The effects of the above technical solution are: dynamically adjusting the length and frequency of the collection cycle according to the data change trend and the direction of change of the collection demand can ensure the timeliness and accuracy of data collection, avoid data redundancy or omission, and improve the overall efficiency of data collection; while implementing the collection cycle adjustment, continuously monitor the key indicators of the data collection point, such as data volume, data update frequency and network transmission speed, to ensure that the adjusted collection cycle can meet the actual needs and avoid adverse effects on system performance. By collecting feedback data, the effectiveness of the adjustment strategy can be further verified; based on the analysis results of the feedback data, the task scheduling algorithm is optimized, such as optimizing the task scheduling strategy, adjusting the task priority, etc., which can significantly improve the execution efficiency and accuracy of the data collection task and ensure that the key data collection tasks are given priority; introducing a new task scheduling algorithm or model can better adapt to the changes in data collection needs and improve the flexibility and scalability of the system; according to the results of task scheduling and the analysis of feedback data, dynamically adjusting the allocation of system resources, such as increasing or decreasing the number of CPU cores, memory size, network bandwidth, etc., can ensure the efficient use of system resources and avoid resource waste; in the process of adjusting the allocation of system resources , real-time monitoring of resource usage can promptly discover and resolve resource bottlenecks to ensure stable and efficient system performance; by continuously monitoring key indicators, dynamically adjusting collection cycles and task scheduling algorithms, and real-time monitoring of resource usage, it can promptly discover and resolve potential problems, reduce system failure rates, and improve system stability and reliability; optimized data collection and system resource allocation strategies can provide users with more accurate and timely data, improve user experience and satisfaction; by dynamically adjusting system resource allocation and optimizing task scheduling algorithms, it can reduce unnecessary resource and energy consumption and reduce operating costs; this technical solution can improve the overall performance and cost-effectiveness of the system, bringing greater economic benefits to the enterprise.

[0099] In one embodiment of the present invention, S5 includes:

[0100] S51. Remove invalid, duplicate or abnormal data records through data cleaning algorithms, convert data into a unified format, and verify the integrity, accuracy and consistency of the data;

[0101] S52, encrypting the preprocessed data using a data encryption algorithm, and compressing the encrypted data using a compression algorithm;

[0102] S53, transmitting the compressed data to the cloud server through a multi-channel transmission protocol, and storing the received compressed data through the cloud server.

[0103] The working principle of the above technical solution is as follows: through the data cleaning algorithm, invalid, duplicate or abnormal data records are removed, the data is converted into a unified format, and the data is checked for integrity, accuracy and consistency; the pre-processed data is encrypted using a data encryption algorithm (such as AES, RSA, etc.). AES is a symmetric encryption algorithm that provides high encryption strength and fast encryption speed; RSA is an asymmetric encryption algorithm commonly used for digital signatures and key exchange; the purpose of encryption is to protect the confidentiality of data and prevent unauthorized access to data during transmission or storage; the encrypted data is compressed to reduce the size of the data and the transmission time; the compression algorithm may include lossless compression (such as Huffman coding, Lempel-Ziv-Welch algorithm) or lossy compression (such as JPEG image compression), and the specific choice depends on the type and requirements of the data; the data is transmitted to the cloud server in parallel using a multi-channel transmission protocol. This can significantly improve the throughput and reliability of data transmission; multi-channel transmission protocols may include HTTP / 2, QUIC or other protocols that support parallel connections and flow control; after receiving the compressed data, the cloud server stores it in the cloud storage system; the cloud storage system provides high availability and scalability, and can ensure the security and persistence of data; data may be further organized, indexed, and managed in cloud storage to facilitate subsequent data analysis and access.

[0104] The effects of the above technical solutions are as follows: through the data cleaning algorithm, invalid and duplicate data records can be accurately identified and eliminated, thereby purifying the data set and reducing noise interference; data from different sources and formats can be converted into a unified format to facilitate subsequent data processing and analysis, and improve data consistency and availability; strict data verification is performed to ensure data integrity (no omissions), accuracy (no errors) and consistency (logical consistency), providing reliable protection for subsequent data analysis; advanced encryption algorithms such as AES and RSA are used to encrypt pre-processed data to effectively prevent data from being stolen or tampered with during transmission and storage, and to protect the confidentiality and integrity of the data; data is compressed based on encryption, which not only reduces the size of the data and reduces storage and transmission costs, but also further enhances data security, because compressed data is more difficult to be identified and parsed by unauthorized personnel; using a multi-channel transmission protocol, multiple data transmission channels can be opened at the same time to achieve data Parallel transmission can significantly improve the throughput and efficiency of data transmission; the transmission of compressed data reduces the size of data, reduces the occupancy of network bandwidth, and speeds up data transmission, especially when the network conditions are poor, this advantage is more obvious; cloud servers provide flexible storage resources, which can dynamically adjust the storage space according to the growth of data to meet the growing demand for data storage; cloud storage systems usually adopt a distributed architecture with high availability and fault tolerance, which can ensure normal access and storage of data even if some nodes fail; compared with traditional local storage, cloud storage has lower maintenance costs and higher resource utilization, saving enterprises a lot of storage and management costs; high-quality data provides strong support for enterprise decision-making, making decisions more scientific, accurate and efficient; by optimizing data processing and storage processes, enterprises can respond to market changes more quickly and improve business processing efficiency, thereby occupying an advantageous position in the fierce market competition.

[0105] One embodiment of the present invention, as Figure 2 As shown, a concentrator efficient data collection optimization system, the system comprises:

[0106] Task setting module: setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station;

[0107] Exclusive task module: After the system starts, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority;

[0108] Reading task module: After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained through the task scheduling mode of the state machine.

[0109] Trend prediction module: Analyzes historical data based on the correlation factors of the collection points and machine learning algorithms to predict future data change trends;

[0110] Data storage module: pre-processes the collected data, encrypts the processed data through encryption algorithm and sends it to the cloud server for storage.

[0111] The working principle of the above technical solution is as follows: First, the data acquisition tasks (i.e., data reading tasks) are divided into two categories: exclusive tasks and parallel tasks. Exclusive tasks refer to those tasks that need to monopolize system resources during execution and cannot be interrupted by other tasks; while parallel tasks can be executed simultaneously without interfering with each other; a priority is assigned to each data reading task so that the system can arrange the execution order according to importance or urgency when facing multiple tasks; a state machine mechanism is introduced to manage the execution process of the first data reading task (i.e., the task to be read data configured by the acquisition master station). The state machine can clearly define the different execution stages of the task and its conversion conditions to ensure that the task is executed in a predetermined logical order; after the system is started, first check whether there is an exclusive task in the task pool; if there is an exclusive task, execute these tasks in order from high to low priority. This step ensures that critical tasks can obtain system resources first and avoid being delayed by other tasks; after all exclusive tasks are executed, the system obtains all parallel first data reading tasks from the task pool; using the task scheduling mode of the state machine, these parallel tasks will be started at the same time and executed in parallel. The state machine can monitor the execution status of each task to ensure synchronization and coordination between tasks; through parallel execution, the system can simultaneously obtain data items to be copied from multiple collection points to improve the efficiency of data collection; dynamically adjust the collection cycle according to the data volume, data update frequency, network transmission speed and other related factors of the collection point. This helps to balance the real-time nature of data collection and the consumption of system resources; use machine learning algorithms to analyze the collected historical data and predict future data change trends. Based on these prediction results, the system can further optimize the task scheduling strategy, for example, increase the collection frequency during periods of frequent data changes, and reduce the collection frequency when the data is stable; pre-process the collected data, including data cleaning, format conversion, etc., to ensure the accuracy and consistency of the data; encrypt the processed data through encryption algorithms to protect the privacy and security of the data; send the encrypted data to the cloud server for storage. The cloud server provides a highly available and scalable storage solution that can ensure the security and long-term preservation of data.

[0112] The effects of the above technical solution are as follows: by setting the copying task as an exclusive task or a parallel task, and assigning priorities according to the importance or urgency of the task, it can ensure that key tasks are handled first, avoid mutual waiting and conflicts between tasks, and thus improve the overall efficiency of data collection; the introduction of a state machine mechanism to manage the execution process of tasks can clearly define the different execution stages of tasks and their conversion conditions, ensure that tasks are efficiently executed in a predetermined logical order, and reduce waiting time and resource waste between tasks; the system first executes all exclusive tasks to ensure that these tasks are completed without interference. Subsequently, the remaining tasks are executed in parallel to make full use of system resources, improve resource utilization and system throughput; dynamically adjust the collection cycle according to the associated factors of the collection point (such as data volume, data update frequency, network transmission speed, etc.), balance the real-time nature of data collection and the consumption of system resources, and avoid waste of resources caused by excessive or insufficient collection; analyze the historical data collected based on the machine learning algorithm, predict future data change trends, and optimize the task scheduling strategy accordingly. It helps to discover data anomalies or change trends in advance, adjust the collection strategy in time, and ensure that the collected data is more accurate and reliable; pre-processing the collected data, including data cleaning and format conversion, can ensure the accuracy and consistency of the data. At the same time, the processed data is encrypted and stored through encryption algorithms to protect the privacy and security of the data and prevent data leakage or tampering; the system can dynamically adjust the collection cycle and task scheduling strategy according to the actual situation of the collection point and the data change trend, and enhance the flexibility and adaptability of the system. This helps to cope with the changes in demand in different collection scenarios and improve the scalability and maintainability of the system; sending the encrypted data to the cloud server for storage can take advantage of the elasticity and scalability of cloud services to provide the system with high availability and disaster recovery backup capabilities. At the same time, the cloud server can also provide convenient data access and management functions to reduce the operation and maintenance costs of the system.

[0113] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A concentrator efficient data collection optimization method, characterized in that: The method comprises: S1, setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station; S2. After the system is started, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority; S3. After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained by executing them in parallel through the task scheduling mode of the state machine; S4. Analyze the historical data collected based on the correlation factors of the collection points and the machine learning algorithm to predict the future data change trend; S5, pre-processing the collected data, encrypting the processed data through an encryption algorithm and sending it to a cloud server for storage; Said S1 comprises: S11. Determine the specific type of the exclusive task, distinguish different categories of parallel tasks, and set specific attributes for each task; S12, dynamically adjust the priority according to the feedback during task execution through a multi-dimensional priority evaluation model; S13, based on the priority conflict resolution algorithm, when multiple high-priority tasks request resources at the same time, decisions are made based on the specific attributes of the tasks and the current state of the system; S14. Design the state transition diagram and state behavior of the state machine, record the life cycle of the task, integrate the state monitoring module, track the task status in real time, and provide feedback to the task scheduler; The S2 comprises: S21, periodically scan the task pool, identify all exclusive tasks to be executed, and obtain the current resource occupancy of the concentrator; S22, applying a sorting algorithm to prioritize the exclusive tasks and reserving system resources for the exclusive tasks to be executed; S23, starting the exclusive task based on the priority order, monitoring the task execution progress in real time, and handling any exceptions that may occur during the execution process; The S21 includes: S211, based on a preset time interval, regularly perform a comprehensive scan on the task pool, and dynamically adjust the time interval according to changes in system load and task characteristics; S212, during the scanning process, obtaining the current status of each task, and monitoring the resource usage of the concentrator in real time through the resource detector; S213, based on the acquired resource occupancy data, evaluate the current resource usage; according to the result of the resource occupancy evaluation, formulate and implement a resource optimization and release strategy; S214, identifying all exclusive tasks to be executed in the task pool, and reserving necessary system resources for the exclusive tasks to be executed according to demand; S215: Before executing the exclusive task, perform a preprocessing operation, and in the process of executing resource optimization release and exclusive task identification, perform exception handling through an exception handling mechanism.

2. According to claim 1, a concentrator efficient data collection optimization method is characterized in that: The S12 comprises: Assign an urgency level to each task, based on the nature of the task and the importance of the data; During the execution of tasks, the system collects feedback data in real time during the execution of tasks. Based on the collected feedback data, the system applies the priority dynamic adjustment algorithm to calculate the current priority of each task. When multiple high-priority tasks request resources at the same time, the priority conflict resolution algorithm is applied to make intelligent decisions, dynamically adjust the calculation results of the algorithm according to the priority, and update the priority of the task in real time; In the process of implementing the priority adjustment strategy, the system continuously monitors the execution status of tasks and the usage of resources. If it is found that the execution progress of a task is seriously lagging behind or the resource usage is too high, an alarm will be issued and feedback will be provided.

3. According to claim 1, a concentrator efficient data collection optimization method is characterized in that: The S211 includes: The system sets a basic scanning cycle, which is the initial time interval for the system to perform a comprehensive scan of the task pool; the system monitors and analyzes the current system load in real time, and analyzes the characteristics of the tasks in the task pool; Based on the analysis results of system load and task characteristics, evaluate whether the current scanning cycle is feasible; if the system load is high or there are a large number of high-priority and resource-intensive tasks in the task pool, shorten the scanning cycle; According to the analysis results of system load and task characteristics, the system dynamically adjusts the scanning cycle; if the system load is light and there are fewer tasks in the task pool or the resource demand is low, the scanning cycle is extended; On the contrary, if the system is heavily loaded or there are a large number of high-priority and resource-demanding tasks in the task pool, the scanning period is shortened; Through the preset scanning cycle adjustment threshold, when the change of system load or task characteristics exceeds this threshold, the scanning cycle adjustment is triggered.

4. The method for optimizing the efficient data collection of a concentrator according to claim 1, characterized in that: The S3 includes: S31, retrieving all parallel first copy-reading tasks from the task pool, and grouping the parallel tasks according to the attributes of the tasks; S32, based on the state machine scheduling strategy, the parallel tasks are executed simultaneously according to the preset priorities and resource allocation schemes, and the execution status of the parallel tasks is monitored in real time; S33. Acquire the data items to be copied during the execution of the parallel tasks through the data acquisition strategy.

5. The method for optimizing concentrator efficient data collection according to claim 1, characterized in that: The S4 comprises: S41. Continuously monitor key indicators of data collection points, including data volume, data update frequency, and network transmission speed; S42. Conduct in-depth analysis of monitoring data based on data analysis algorithms and predict future data change trends and collection needs; S43, based on the data analysis result, setting a dynamic adjustment strategy for the acquisition period, wherein the dynamic adjustment strategy includes adjusting the length of the period, the frequency of adjustment, and the direction of adjustment; S44. By adjusting the collection cycle strategy, feedback data is continuously collected, and the task scheduling algorithm is improved according to the dynamic adjustment result of the collection cycle; and the allocation of system resources is dynamically adjusted according to the result of task scheduling.

6. The method for optimizing concentrator efficient data collection according to claim 1, characterized in that: The S5 comprises: S51. Remove invalid, duplicate or abnormal data records through data cleaning algorithms, convert data into a unified format, and verify the integrity, accuracy and consistency of the data; S52, encrypting the preprocessed data using a data encryption algorithm, and compressing the encrypted data using a compression algorithm; S53, transmitting the compressed data to the cloud server through a multi-channel transmission protocol, and storing the received compressed data through the cloud server.

7. A system for executing the concentrator efficient data collection optimization method according to claim 1, characterized in that: The system comprises: Task setting module: setting each reading task as an exclusive task or a parallel task, assigning priorities to the reading tasks, and introducing a state machine for executing the first reading task, that is, the task of collecting the data to be read configured by the master station; Exclusive task module: After the system starts, it first determines whether there are exclusive tasks in the task pool. If so, all exclusive tasks are executed in order of priority; Reading task module: After the exclusive task is executed, all the parallel first reading tasks in the task pool are obtained, and the corresponding data items to be read are obtained through the task scheduling mode of the state machine. Trend prediction module: Analyzes historical data based on the correlation factors of the collection points and machine learning algorithms to predict future data change trends; Data storage module: pre-processes the collected data, encrypts the processed data through encryption algorithm and sends it to the cloud server for storage.

Images