Distributed tower monitoring scheduling system and scheduling method for multi-stage task

Abstract

This invention discloses a distributed pole monitoring and scheduling system and method for multi-stage tasks, comprising multiple device terminals, a gateway, an edge monitoring platform, edge clients, a distributed backend server, and distributed server nodes. The device terminals collect BeiDou data for pole positioning, and each terminal transmits data back to the edge clients via the gateway. The edge clients submit BeiDou data to the distributed backend server in multiple batches at different stages. The distributed backend server distributes tasks to the distributed server nodes using Nginx load balancing. Each distributed server node's task is handled by multiple worker nodes, which use distributed locks to preempt tasks. The gateway transmits signals to the edge monitoring platform, which initiates tasks and uses asynchronous polling to distribute tasks to the distributed backend server for processing. This invention effectively improves throughput and task processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of pole monitoring technology, specifically to a distributed pole monitoring scheduling system and scheduling method for multi-stage tasks. Background Technology

[0002] With the exponential increase in the number of mobile monitoring devices connected to the power grid, the development of smart grid-related technologies has become a key focus. Transmission towers, as one of the most important base station facilities, are particularly vulnerable to tilting, which can cause power transmission failures and even serious accidents. Intelligent monitoring of transmission towers still faces many technical bottlenecks. On the one hand, transmission towers are generally located in remote mountainous areas, and traditional manual monitoring is not only costly but also suffers from limitations such as personnel safety and low timeliness in extreme weather conditions. On the other hand, considering intelligent tower monitoring, there is the problem of difficulty in transmitting data from towers in remote areas, requiring a focus on low power consumption and long-distance transmission capabilities of data acquisition equipment. Furthermore, taking BeiDou as a technology for handling tower tilting as an example, the limited bandwidth of wireless data transmission between nodes due to the need for long-distance transmission leads to long processing times for the backend BeiDou processing module. Additionally, centralized servers suffer from resource waste due to prolonged blocking when processing such tasks, resulting in poor system disaster recovery.

[0003] Against this backdrop, distributed asynchronous scheduling systems have emerged. This system, through a distributed concurrent processing architecture, distributes time-consuming processing tasks across various server nodes, effectively solving the problems of large data processing volume and long processing times under multi-tower node monitoring. Simultaneously, it improves system resource utilization through reasonable load balancing strategies. The distributed architecture also gives the overall monitoring system better scalability and disaster recovery capabilities; when a server node fails, other nodes can continue to handle the tasks without affecting the operation of the entire system. This distributed data management approach can effectively mitigate problems such as system crashes and network congestion that arise from centralized management. However, in practical applications, the following technical issues still need to be further addressed:

[0004] Question 1: In the data acquisition of pole towers, when designing terminal equipment, it is necessary to ensure low power consumption and long-distance transmission of BeiDou data and other sensor data, and design the data packet protocol format for interaction with the management platform;

[0005] Question 2: The visual management platform needs to have functions such as data transfer, log system, alarm system, equipment management and fully automatic control;

[0006] Question 3: When designing load balancing strategies and providing interfaces for the backend servers of a distributed system, how can we ensure the idempotency of the interfaces in the multi-worker working mode and the performance under high traffic?

[0007] Question 4: The worker design of a distributed system needs to consider solving the competition problem, avoiding the waste of resources caused by duplicate task consumption, and at the same time realizing Beidou processing and sensor data processing. Summary of the Invention

[0008] To overcome the defects and shortcomings of existing technologies, this invention provides a distributed pole monitoring and scheduling system for multi-stage tasks, which effectively improves throughput and task processing efficiency.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention provides a distributed pole monitoring and scheduling system for multi-stage tasks, comprising: multiple device terminals, a gateway, an edge monitoring platform, an edge client, a distributed backend server, and distributed server nodes;

[0011] The device terminal is used to collect BeiDou data for tower positioning. Each device terminal interacts with data through a LoRa self-organizing network and transmits the data back to the edge client through a gateway.

[0012] The edge client submits BeiDou data to the distributed backend server in stages and batches. The distributed backend server has multiple distributed server nodes. The task is distributed to the distributed server nodes through Nginx load balancing. Each distributed server node's task is handled by multiple worker nodes. The worker nodes use distributed locks to preempt tasks, and the multiple coroutines inside the worker nodes handle concurrent processing.

[0013] The gateway transmits signals to the edge monitoring platform, which then hands over the task processing to the distributed backend server through task initiation and asynchronous task polling.

[0014] As a preferred technical solution, the device terminal includes a dual power supply module, an MCU microcontroller, a LoRa radio frequency module, a Beidou processor, a gyroscope, a smoke sensor, and a temperature and humidity sensor;

[0015] The gyroscope collects tower attitude information, the BeiDou processor collects data from multiple satellites for tower positioning, and performs joint calculations based on the tower attitude information collected by the gyroscope. The smoke sensor collects ambient smoke concentration, the temperature and humidity sensor collects ambient temperature and humidity, the LoRa RF module provides a LoRa protocol interface for LoRa communication with the gateway, and the dual power supply module provides both solar and battery power to the BeiDou processor and MCU microcontroller. The MCU microcontroller controls the dual power supply module to switch power modes and controls the gyroscope, smoke sensor, and temperature and humidity sensor to collect data, and encrypts and decrypts the data.

[0016] As a preferred technical solution, the distributed backend server is equipped with a task management module. This module categorizes tasks into completed "cold" tasks and incomplete "hot" tasks. Based on a location information table, it periodically performs table partitioning operations, marking the table position `begin_pos` occupied by the task in the next scheduling iteration and the table position `end_pos` for newly created tasks in the location information table. Tasks within the specified interval are considered hot tasks.

[0017] As a preferred technical solution, the distributed backend server is equipped with a task retry module. This module constructs a uniform retry mechanism and a progressive retry mechanism. In the progressive retry mechanism, the progressive retry time is determined by the maximum number of retries (max_retry_num), the current number of retries (crt_retry_num), and the maximum retry interval (max_retry_interval). The progressive retry time is:

[0018] retry_interval min(1< <crt_retry_num,max_retry_interval)。

[0019] This invention also provides a scheduling method for a distributed pole monitoring and scheduling system for multi-stage tasks, comprising the following steps:

[0020] Data collection from poles and towers is performed based on device terminals, and joint calculation is achieved by combining the pole and tower attitude information collected by gyroscopes. Each device terminal interacts with data through a LoRa self-organizing network and transmits the data back to the edge client through a gateway.

[0021] The edge client submits BeiDou data to the distributed backend server in stages and batches. The distributed backend server has multiple distributed server nodes. The task is distributed to the distributed server nodes through Nginx load balancing. Each distributed server node's task is handled by multiple worker nodes. The worker nodes use distributed locks to preempt tasks, and the multiple coroutines inside the worker nodes handle concurrent processing.

[0022] The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform sends tasks to the distributed backend server for processing by initiating tasks and asynchronous polling.

[0023] The edge monitoring platform receives the message content sent by the gateway, parses the message content based on the set protocol format, sends multiple BeiDou data in batches to request updates to the context information of the current task, initiates a task creation request to the distributed backend server, and fills the task priority field of the BeiDou data with the highest priority.

[0024] As a preferred technical solution, the method of distributing tasks to distributed server nodes through Nginx load balancing specifically includes:

[0025] The distributed backend server manages all tasks through a task information table, a task configuration table, and a task location table. The task information table includes the actual task information pulled by the worker nodes, the task configuration table includes the pulled task configuration information, and the task location table includes table partitioning logic information.

[0026] Load balancing is achieved through a destination address hashing scheduling algorithm, a task registration interface is provided, and the processing logic is embedded in the Worker, thus decoupling task scheduling from task execution.

[0027] As a preferred technical solution, the actual task information pulled by the worker node is prioritized. When task creation and single-stage task execution are successful, the task scheduling time is determined according to the priority, as shown below:

[0028] order_time = modify_time - priority

[0029] When a task fails or is found to be stuck, the next round of task scheduling time is determined based on the retry time, expressed as follows:

[0030] order_time = modify_time + retry_interval

[0031] Where order_time represents the task scheduling time, modify_time represents the task update time, priority represents the task priority, and retry_interval represents the task retry time;

[0032] Adding an MQ message queue decouples task retrieval from task ownership.

[0033] As a preferred technical solution, the server-side interface of the distributed backend server is called to perform database operations, build a database connection pool, and optimize it according to the maximum number of long connections in the database connection pool and the maximum number of connections including long connections and short connections.

[0034] As a preferred technical solution, the Worker node implements task preemption through a distributed lock, and concurrent processing is performed by multiple coroutines within the Worker node, specifically including:

[0035] The task is preempted based on Redis distributed lock, and the Beidou processing logic and sensor data task processing logic are built in the worker program.

[0036] Configure watchdog and lock expiration mechanisms;

[0037] In the BeiDou processing logic, the Worker node obtains the MSM message in the RTCM frame data from the context of the task as observation data, obtains the positioning estimate through the single-point positioning algorithm and sends it to the CORS center. Based on the positioning estimate and the correction data calculated by the CORS center, the positioning result is obtained in the carrier relative positioning algorithm. The positioning result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds the threshold, the processing result that the tower is collapsing is sent.

[0038] In the sensor data task processing logic, the tilt result obtained by the Worker node in processing the sensor data is used as the BeiDou polling frequency switch. When an anomaly is detected, the BeiDou polling frequency is increased to monitor whether the tower is tilted using BeiDou positioning.

[0039] As a preferred technical solution, multiple BeiDou data streams are sent in batches to request updates to the context information of the current task, specifically including:

[0040] Multiple BeiDou data are used in a new BeiDou data processing phase. The first phase requests a task ID from the distributed backend server and calls the interface to notify the distributed backend server to modify the context information.

[0041] The distributed backend server updates the context information to the context corresponding to the requested task, which serves as the raw data for the next stage of task scheduling.

[0042] When a distributed server node is scheduled by a worker in the current stage, the worker retrieves the intermediate values ​​of the historical stages in the context and the new data packets of the current stage to perform single-point coarse positioning, and hands them over to the CORS center to calculate the carrier relative positioning result. If the calculation result cannot be fixed, it is returned to update the distributed backend server again. If the calculation result is successfully fixed, it is determined whether the tower is tilted, and the task status is reset to the final state, indicating that the multi-stage task of the distributed server node has ended.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) This invention solves the limitation of blockage and waste when processing Beidou data by a single machine through a distributed architecture. The terminal adopts a method of mounting multiple nodes on a single network in a multi-network to reduce network transmission pressure.

[0045] (2) The present invention designs a general back-end server framework, which is applicable not only to the tower monitoring scenario of this project, but also to the scheduling of any multi-stage task scenario, effectively improving throughput and task processing efficiency. Its framework has strong scalability and applicability.

[0046] (3) The present invention uses a dynamic rolling table method to distinguish between hot and cold data, which solves the problem of low performance caused by excessive accumulation of cold data. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall architecture of the distributed pole monitoring and scheduling system for multi-stage tasks according to the present invention.

[0048] Figure 2 This is a schematic diagram of the architecture of each module of the tower data acquisition device of the present invention;

[0049] Figure 3 This is a schematic diagram of the overall architecture of the management platform of the present invention;

[0050] Figure 4 This is a schematic diagram of the data processing flow of the management platform of the present invention;

[0051] Figure 5 This is a schematic diagram of the task scheduling flow of the distributed backend server of the present invention;

[0052] Figure 6 This is a schematic diagram of the table partitioning and rolling strategy of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Example

[0054] like Figure 1 As shown, this embodiment provides a distributed pole monitoring and scheduling system for multi-stage tasks, which can effectively improve the ability of distributed concurrent processing and scheduling tasks, and save server resources. The system includes:

[0055] The gateway and device terminals (i.e., tower data acquisition devices) at the data acquisition end are used to collect BeiDou data. The various device terminals interact with each other via LoRa self-organizing network and transmit the data back to the edge client through a secure intranet dedicated line at the gateway. The edge client submits BeiDou data to the distributed backend server in multiple batches in stages. The distributed backend server distributes the data to a node through load balancing and persists the data in the database, waiting for worker scheduling. The distributed backend server has a task retry module and a task management module. The worker schedules and pulls a batch of tasks from the server at regular intervals. Since the BeiDou data of the same node is implemented asynchronously in multiple stages, the intermediate BeiDou calculation results of the node are returned after each scheduling and the results are persisted in the context field of the database. In particular, when the BeiDou calculation reaches the persistence threshold, the task status corresponding to the node is set to the final state.

[0056] The device terminal and gateway serve as data aggregation access points. The device terminal is equipped with a Beidou monitoring module and transmits signals based on LoRa communication.

[0057] The monitoring and management are based on a visual edge monitoring platform. The gateway and the platform achieve unified access through UDP communication. The platform sends tasks to a distributed backend server for processing by initiating tasks and asynchronous polling.

[0058] The distributed backend server has multiple scheduling servers. The monitoring platform uses Nginx load balancing to distribute tasks to a specific server, i.e., managing multiple distributed server nodes through Nginx. A connection pool design restricts user access and reduces the overhead of creating connections. Each server's tasks are handled by multiple workers. Workers use distributed locks to preempt tasks, and concurrent processing is handled by multiple coroutines within the worker, incorporating BeiDou task processing logic and sensor data storage and query logic.

[0059] like Figure 2As shown, the pole data acquisition device includes a dual power supply module equipped with a solar panel and a battery, an MCU microcontroller, a LoRa radio frequency module, a Beidou processor, a gyroscope, a smoke sensor, and a temperature and humidity sensor. Compared with other terminal devices, the gateway in this embodiment has an additional W5500 Ethernet module for accessing the intranet monitoring platform at the edge. The gateway also has pole monitoring and aggregation functions.

[0060] This embodiment also provides a scheduling method for a distributed pole monitoring and scheduling system for multi-stage tasks, including the following steps:

[0061] S1: In tower data acquisition, construct a low-power, long-distance transmission terminal device equipped with a Beidou processor, build a proprietary datagram protocol format, and interact with the edge monitoring platform via UDP through a gateway;

[0062] The data format of the proprietary protocol is shown in Table 1 below:

[0063] Table 1. Data Format Diagram of Proprietary Agreements

[0064]

[0065] In this embodiment, the BeiDou processor collects data from multiple satellites for tower positioning and combines it with gyroscope-collected tower attitude information for joint calculation. Since BeiDou calculation consumes significant wireless mesh network resources and can cause network congestion in narrowband transmission, and because gyroscope data is extremely limited, BeiDou polling positioning is triggered periodically every 5 hours. During other time periods, the gyroscope module roughly calculates the tower's offset. At 05:00, the BeiDou positioning module is activated. During BeiDou data acquisition, after demodulation processing by the BeiDou antenna, the MSM message in the RTCM frame data is parsed and obtained. This message includes the message header, navigation data, and observation data, which are used as task context information as discussed below. In addition, the device terminal is also equipped with temperature and humidity sensors, smoke sensors, etc., as daily sensing and monitoring targets. Under severe conditions such as heavy rain, the priority of BeiDou calculation tasks is increased. In particular, the number of satellites affects positioning accuracy; therefore, the device terminal needs to be placed in an open area with good signal coverage whenever possible.

[0066] In this embodiment, the distance between adjacent towers is on the order of kilometers, so a communication module for long-distance transmission is required; LoRa is selected as the radio frequency networking transmission module, and the module uses the SX1278 chip with integrated LoRa protocol interface.

[0067] In this embodiment, the MCU microcontroller uses an STM32F103ZET6 as the main control processor and a FreeRTOS operating system as its kernel. It implements AES data encryption / decryption and sensor data acquisition tasks in each node; additionally, it implements network management and timed task execution in each aggregation gateway. Specifically, under low power consumption requirements, the MCU microcontroller automatically switches between sleep mode and operating mode to save power consumption and switches between the solar panel and the 3.3V auxiliary power supply to ensure continuous power supply.

[0068] S2: The visual edge monitoring platform is equipped with a two-way communication protocol with multiple terminal devices and multiple distributed systems, and has functions such as data relay, log system, alarm system, device management and fully automatic control.

[0069] In this embodiment, the visual management platform uses QT for desktop development. The edge management platform receives information sent by the gateway aggregation node and parses the message payload content based on its own protocol format to obtain the RootAddr, NodeAddr, and DataBuffer data in the aforementioned proprietary protocol packet. Preferably, in implementing the fully automatic control function, the instructions in the message packet include autonomous route modification, sensor data polling, BeiDou polling, and device status information polling. Terminal data is received through a reliable wired intranet transmission method. It also includes a log operation recording management module, a user management module, a data analysis and ledger display module, a login and registration module, and a bidirectional data transmission module that enables bidirectional data transmission with the data acquisition device and the backend server via HTTP.

[0070] In this embodiment, the edge monitoring platform achieves bidirectional communication with the aggregation gateway in an intranet environment via a reliable UDP wired transmission method. Data packets follow a custom protocol format, and the message includes information such as the gateway node number, monitoring terminal number, function-code, and data. The function-code consists of 1 byte and is divided into 256 types ranging from 0 to 127 and 128 to 255, respectively used to identify the function-codes on the device and the area for function-codes issued by the management platform, facilitating the addition and management of functions.

[0071] Specifically, for BeiDou data transmission, there is a bottleneck of low bandwidth between nodes, resulting in limited data transmission speed and a significant idle period between adjacent data from the same node. To address this, a batch processing concept is proposed, representing multiple batches of data from the same node as multiple stages. This allows Workers to asynchronously execute tasks from other nodes while waiting for the next stage of data during the idle period. Therefore, when the edge monitoring platform receives BeiDou data, it sends requests to update the context information of the current task in batches of n data. When initiating a task creation request to the distributed backend server, the BeiDou task priority field is filled with the highest priority.

[0072] Preferably, to achieve automated management of tower monitoring and terminal equipment, this embodiment provides various control commands for remote control, including but not limited to commands for BeiDou polling, sensor data polling, equipment status information polling, remote equipment reset, and autonomous node routing modification. The commands follow the aforementioned proprietary protocol packaging method, with FunctionCode representing different command types, and are sent to the designated node via RootAddr and NodeAddr.

[0073] like Figure 3 As shown, the designed management platform includes a data acquisition and transmission module, a log management module, a user login and registration module, and a data display and analysis management module.

[0074] like Figure 4 As shown, for the data processing flow, taking BeiDou data as an example, when a node initiates the BeiDou module, it collects a batch of data for a new BeiDou data processing stage. In the first stage, it requests a task ID from the distributed backend server and calls an interface to notify the distributed backend server to modify the context information, where the context contains 5 BeiDou data entries. Further, the distributed backend server updates the context to the context corresponding to the previously requested task, serving as the raw data for the next stage of task scheduling. When the distributed server node is scheduled by a worker in the current stage, the worker retrieves the intermediate values ​​from previous stages in the context and the new data packets of the current stage for single-point coarse positioning, and hands them over to the CORS center to calculate the carrier relative positioning result. If the calculation result is still not fixed, the intermediate result of this stage is returned to update the distributed backend server. Conversely, if the positioning is successfully fixed, it checks for tilt and resets the task status to the final state, indicating that the multi-stage task of the distributed server node has ended.

[0075] S3: The distributed backend server design includes load balancing strategies, provides HTTP interfaces with the management platform and Workers, and features interfaces for task registration, task query, task preemption, and task creation. It implements modules for scheduling, task retries, and task governance, and ensures interface idempotency and performance under high traffic in a multi-Worker working mode through unique indexes and distributed scheduling. Notably, this server is versatile, allowing it to be integrated as a general framework to meet other task requirements; specific task processing is handled by Workers.

[0076] In this embodiment, the distributed backend server is allowed to receive at least 3000 task requests per second. Tasks are persisted in the master-slave database, and all tasks are managed through a task information table, a task configuration table, and a task location table. The task information table is partitioned and rolled over with a limit of 5 million requests per second. Different processing priorities are assigned to two main task categories: BeiDou data processing and sensor data processing. Optionally, BeiDou data processing has time-sensitive requirements, so its priority is much higher than that of sensor data processing tasks. Preferably, the backend server provides interfaces for task registration, task query, task preemption, and task creation, implementing modules such as scheduling, task retry, and task governance, and achieving load balancing through a destination address hash scheduling algorithm. Specifically, the designed backend server can serve as a general-purpose framework, receiving and scheduling any task and providing a task registration interface, while the actual task processing logic is persisted in the Worker by the business side, achieving decoupling between task scheduling and task execution. Preferably, the backend server improves its response capability by optimizing the number of connection pools.

[0077] The distributed backend server in this embodiment uses a destination address hashing scheduling algorithm to achieve route load balancing, which can solve the problems of poor disaster recovery performance and low efficiency caused by centralized servers.

[0078] In this embodiment, the backend server provides multiple interfaces to the management platform and the Worker, consisting of a task information table, a task configuration table, and a location information table used for table partitioning logic. The task information table contains the actual task information that the worker needs to retrieve; the task configuration table includes configurations such as the number of tasks retrieved each time, the task retrieval interval, and the maximum number of retries for the task; the location information table is used for table partitioning and rolling, i.e. Figure 6 The table partitioning logic is as follows: tasks are persisted in a task information table in a MySQL database. This table contains a unique task ID index to ensure interface idempotency. The unique task ID is generated by concatenating the table number using the snowflake algorithm. The table number is used by the management platform for task status queries and task context updates. In the interface design, application / json is selected as the data transmission format for content-type, and encapsulation and parsing are performed through serialization and deserialization. The interface design is shown in Table 2 below.

[0079] Table 2 Interface Design Diagram

[0080]

[0081] like Figure 5 As shown, considering the characteristics of task execution in this embodiment, tasks are divided into completed "cold tasks" and incomplete "hot tasks." Therefore, the table design also adopts a rolling table mode for cold and hot tables, as shown below. Figure 6As shown, the table partitioning is implemented by the task governance module in conjunction with the location information table, executed periodically. Therefore, the table partitioning design has a delay, allowing the number of tasks to exceed the threshold of 5 million within a short period. The location information table's `begin_pos` marks the table position occupied by the task in the next scheduling, and `end_pos` marks the table position of the newly created task. The tasks within the interval are the aforementioned hot tasks.

[0082] Combination Figure 5 As shown, to ensure successful task execution, the task governance module periodically monitors stuck tasks and restores them to a pending state. A retry mechanism is also designed. Optionally, the general task scheduling server framework in this embodiment provides a uniform retry mechanism and a progressive retry mechanism. In the progressive retry mechanism, the retry time is determined by the maximum number of retries (max_retry_num), the current number of retries (crt_retry_num), and the maximum retry interval (max_retry_interval). The retry time starts from 1 second, and the progressive retry time is:

[0083] retry_interval min(1< <crt_retry_num, max_retry_interval)

[0084] Specifically, during BeiDou data transmission, there is a significant idle period between adjacent data pairs. To address this, the management platform assembles n data pairs into a batch. The scheduling system is designed to process these batches of data in a multi-stage manner. The processing results of the previous stage are stored in the context, and other tasks are asynchronously scheduled and executed during the gaps between the current task's multiple stages. This improves scheduling efficiency and reduces resource waste.

[0085] Preferably, task retrieval requires determining which tasks to retrieve first. Task retrieval depends entirely on the task's execution status; tasks to be retrieved for the next batch of scheduling must be in a pending execution state. Simultaneously, the order in which tasks are retrieved is related to the task update time (modify_time), task priority, and task retry time (retry_interval). In this embodiment, these three are coupled into order_time. When retrieving tasks, the first n tasks in ascending order_time are selected, reducing the query time for task retrieval by decreasing the number of composite indexes. The task update time follows a first-in, first-out (FIFO) principle, ensuring new tasks are queued to the end. This embodiment abstracts task priority as the time required to execute tasks ahead of priority; higher-priority tasks have more advance time. For the BeiDou processing task in this embodiment, it has the highest priority, so its priority is 1 year, which can be considered infinite priority. In this embodiment, the retry logic is reflected in order_time, which accumulates the retry time to ensure that tasks within the retry time cannot be scheduled, thus preventing abnormal tasks from blocking the server. Based on the above constraints, order_time will be updated in the following stages:

[0086] 1) Upon successful task creation and execution of a single-stage task, the task scheduling time is determined based on priority:

[0087] order_time = modify_time - priority

[0088] 2) When a task fails or the governance module detects that a task is stuck, the next scheduling time is determined based on the retry time mentioned above, ensuring that tasks cannot be scheduled within the retry time:

[0089] order_time = modify_time + retry_interval

[0090] Preferably, in order to further solve the problem of wasted worker resources when multiple workers compete to occupy tasks, it is allowed to add an MQ message queue to decouple task retrieval and task occupation, and pull a batch of tasks into the message queue in advance to reduce the time spent on task occupation and improve the response speed of task occupation.

[0091] Preferably, in order to further prevent server port exhaustion and reduce the connection creation cost when receiving requests, in this embodiment, database operations are required after the server interface is called. In order to reduce the consumption caused by frequent database creation and destruction, this example optimizes the design by designing a database connection pool. When the server receives a request from the client or worker, it needs to establish a connection with the database. At this time, maintaining a batch of database connections for reuse can reduce consumption and improve performance. Based on experimental optimization of the maximum number of long connections in the pool and the maximum number of connections including long connections and short connections, the maximum number of long connections in the connection pool is selected as 1000 and the maximum number of connections is 2000.

[0092] S4: The Worker in the distributed system acts as the main part of task processing. It uses a distributed lock to preempt tasks and realizes BeiDou processing and sensor data processing.

[0093] In this embodiment, multiple workers resolve task contention issues using Redis distributed locks, and a watchdog mechanism and lock expiration mechanism are introduced to ensure the non-repeated consumption of preempted tasks. Specifically, based on the nature of this project, BeiDou processing logic and sensor data task processing logic are uniquely designed. BeiDou positioning employs high-precision retrieval and positioning based on network RTK.

[0094] Specifically, in the design of Worker task execution:

[0095] Multiple workers effectively solve the task contention problem using Redis distributed locks. In distributed systems, multiple workers may attempt to process the same task simultaneously, leading to duplicate task execution and wasted resources. Redis distributed locks provide a reliable mechanism to ensure that only one worker can process a specific task at any given time.

[0096] To further ensure the correct execution of tasks, a watchdog mechanism and a lock expiration mechanism are introduced. The watchdog mechanism monitors the running status of Workers and takes timely measures, such as releasing locks, when a Worker malfunctions, allowing other Workers to take over the task. The lock expiration mechanism prevents tasks from being occupied for an extended period due to a Worker's inability to release the lock after acquiring it. By setting a reasonable lock expiration time, it ensures that locks are automatically released when a Worker fails, thereby guaranteeing task executability and system stability.

[0097] In this embodiment, BeiDou processing logic and sensor data task processing logic are designed. BeiDou positioning adopts high-precision differential positioning based on network RTK. It utilizes observation data from multiple reference stations and eliminates or reduces errors in the satellite signal propagation process through differential algorithms, thereby improving positioning accuracy. In RTK high-precision differential positioning, the worker obtains the MSM message from the aforementioned RTCM frame data as observation data from the task context, obtains a positioning estimate through a single-point positioning algorithm, and sends it to the CORS center. Based on the positioning estimate and the correction data calculated by the CORS center, a high-precision positioning result is further obtained through a carrier relative positioning algorithm. This result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds a threshold of 20cm, the processing result that the tower is collapsing is sent to the management system.

[0098] For the sensor data processing logic, corresponding processing algorithms and processes were designed based on the characteristics of different types of sensor data. For example, for gyroscope and vibration sensor data, anomalies in the tower can be detected in a timely manner by analyzing their changing trends. Since a single BeiDou polling operation occupies the network between terminal nodes for a long time, and because BeiDou data packets are large, frequent BeiDou polling can cause rapid power consumption and prolonged bandwidth occupation. Therefore, this embodiment provides sensors with smaller data volumes, such as gyroscopes, as a high-frequency tilt monitoring method. The tilt result obtained by the Worker from processing the sensor data serves as the BeiDou polling frequency switch. When an anomaly is detected, the BeiDou polling frequency is increased to monitor whether the tower is tilted using BeiDou's high-precision positioning. In addition, to meet the management platform's need for ledger analysis and display of historical sensor data, the Worker provides filtering and querying capabilities for historical data in the database.

[0099] This invention fully considers the multi-stage, time-consuming task types in pole monitoring and designs a distributed system suitable for multi-pole monitoring. This system collects data through designed monitoring terminal nodes, and multiple nodes access the monitoring center for processing via a shared gateway. Specifically, the processing system uses a distributed multi-server architecture as the task management center, responsible for providing interfaces to clients to initiate task creation and querying, and providing task retrieval interfaces for workers. Specific time-consuming tasks are handled concurrently by the workers. The design focuses on task governance modules, multi-mode retry mechanisms, and task scheduling processes. A visualization platform is also designed to intuitively display the monitoring and processing results, facilitating user monitoring and management.

[0100] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A distributed pole monitoring and scheduling method for multi-stage tasks, characterized in that, A distributed pole monitoring and scheduling system for multi-stage tasks is characterized by the following steps: the scheduling system includes multiple device terminals, a gateway, an edge monitoring platform, an edge client, a distributed backend server, and distributed server nodes. Data collection from poles and towers is performed based on device terminals, and joint calculation is achieved by combining the pole and tower attitude information collected by gyroscopes. Each device terminal interacts with data through a LoRa self-organizing network and transmits the data back to the edge client through a gateway. The edge client submits BeiDou data to the distributed backend server in stages and batches. The distributed backend server has multiple distributed server nodes. The task is distributed to the distributed server nodes through Nginx load balancing. Each distributed server node's task is handled by multiple worker nodes. The worker nodes use distributed locks to preempt tasks, and the multiple coroutines inside the worker nodes handle concurrent processing. The process of distributing tasks to distributed server nodes via Nginx load balancing specifically includes: The distributed backend server manages all tasks through a task information table, a task configuration table, and a task location table. The task information table includes the actual task information pulled by the worker nodes, the task configuration table includes the pulled task configuration information, and the task location table includes table partitioning logic information. Load balancing is achieved through a destination address hashing scheduling algorithm strategy. A task registration interface is provided, and the processing logic is embedded in the Worker to achieve decoupling between task scheduling and task execution. The actual task information pulled by the worker node has a priority. When a task is created and a single-stage task is successfully executed, the task scheduling time is determined according to the priority, as shown below: order_time = modify_time - priority When a task fails or is found to be stuck, the next round of task scheduling time is determined based on the retry time, expressed as follows: order_time = modify_time + retry_interval Where order_time represents the task scheduling time, modify_time represents the task update time, priority represents the task priority, and retry_interval represents the task retry time; Adding an MQ message queue decouples task retrieval from task ownership; The Worker node implements task preemption through a distributed lock, and concurrent processing is performed by multiple coroutines within the Worker node, specifically including: The task is preempted based on Redis distributed lock, and the Beidou processing logic and sensor data task processing logic are built in the worker program. Configure watchdog and lock expiration mechanisms; In the BeiDou processing logic, the Worker node obtains the MSM message in the RTCM frame data from the context of the task as observation data, obtains the positioning estimate through the single-point positioning algorithm and sends it to the CORS center. Based on the positioning estimate and the correction data calculated by the CORS center, the positioning result is obtained in the carrier relative positioning algorithm. The positioning result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds the threshold, the processing result that the tower is collapsing is sent. In the sensor data task processing logic, the tilt result obtained by the Worker node in processing the sensor data is used as the BeiDou polling frequency switch. When an anomaly is detected, the BeiDou polling frequency is increased to monitor whether the tower is tilted using BeiDou positioning. The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform sends tasks to the distributed backend server for processing by initiating tasks and asynchronous polling. The edge monitoring platform receives the message content sent by the gateway, parses the message content based on the set protocol format, sends multiple BeiDou data in batches to request updates to the context information of the current task, initiates a task creation request to the distributed backend server, and fills the task priority field of the BeiDou data with the highest priority.

2. The scheduling method according to claim 1, characterized in that, After the server-side interface of the distributed backend server is called, database operations are performed, a database connection pool is built, and optimization is carried out based on the maximum number of long connections in the database connection pool and the maximum number of connections including long connections and short connections.

3. The scheduling method according to claim 1, characterized in that, Multiple BeiDou data points are sent in batches to request updates to the context information of the current task, specifically including: Multiple BeiDou data are used in a new BeiDou data processing phase. The first phase requests a task ID from the distributed backend server and calls the interface to notify the distributed backend server to modify the context information. The distributed backend server updates the context information to the context corresponding to the requested task, which serves as the raw data for the next stage of task scheduling. When a distributed server node is scheduled by a worker in the current stage, the worker retrieves the intermediate values ​​of the historical stages in the context and the new data packets of the current stage to perform single-point coarse positioning, and hands them over to the CORS center to calculate the carrier relative positioning result. If the calculation result cannot be fixed, it is returned to update the distributed backend server again. If the calculation result is successfully fixed, it is determined whether the tower is tilted, and the task status is reset to the final state, indicating that the multi-stage task of the distributed server node has ended.

Images