Distributed tower monitoring scheduling system for multi-stage tasks and scheduling method thereof

Abstract

The invention discloses a distributed tower monitoring scheduling system for multi-stage tasks and a scheduling method thereof. The distributed tower monitoring scheduling system comprises a plurality of equipment terminals, a gateway, an edge monitoring platform, an edge client, a distributed background server and a distributed server node, the equipment terminals are used for collecting Beidou data and positioning a tower, and each equipment terminal transmits the data back to the edge client through the gateway; an edge client submits Beidou data to a distributed background server in a staged and multi-batch manner, the distributed background server distributes tasks to distributed server nodes through Nginx load balancing, the tasks of each distributed server node are processed by a plurality of Worker nodes, and the Worker nodes preempt the tasks through a distributed lock; signal transmission is carried out between the gateway and the edge monitoring platform, and the edge monitoring platform carries out task processing on the distributed background server in a task initiating and task asynchronous polling mode. The throughput rate and the task processing efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pole tower monitoring, and in particular to a distributed pole tower monitoring and dispatching system for multi-stage tasks and a dispatching method thereof. Background Art

[0002] As the number of mobile monitoring devices connected to the power grid increases exponentially, the development of smart grid related technologies has also become one of the key focuses. Among them, transmission towers are one of the most important base station facilities. The tilt of the towers will cause power transmission terminals and even serious accidents. In the intelligent monitoring of transmission towers, there are still many technical bottlenecks. On the one hand, transmission towers are generally distributed in remote mountainous areas. Traditional manual monitoring is not only costly, but also has limitations such as personnel safety and low timeliness in extreme weather. On the other hand, considering the intelligent tower monitoring, there is a problem that tower data in remote areas is difficult to transmit back, and it is necessary to focus on the low power consumption and long-distance transmission capabilities of data acquisition equipment. In addition, taking Beidou as a tower tilt processing technology as an example, due to the need for long-distance transmission, there is a bottleneck of limited bandwidth for wireless transmission of data between nodes, which also leads to the problem of long-term processing time of the back-end Beidou processing module, and the centralized server has a problem of long-term blocking and waste of resources in processing such tasks, resulting in poor disaster tolerance of the system;

[0003] In this context, the distributed asynchronous scheduling system came into being. Through the distributed concurrent processing architecture, the system distributes the time-consuming processing task load to each server node, effectively solving the problem of large data processing volume and long processing time under multi-tower node monitoring; at the same time, it improves the resource utilization of the system through a reasonable load balancing strategy. The use of a distributed architecture also makes the overall monitoring system have better scalability and disaster tolerance. When a server node fails, other nodes can continue to take on tasks without affecting the operation of the entire system. Through this distributed data management method, problems such as system crashes and network congestion caused by centralized management can be effectively solved, but in actual application, the following technical problems still need to be further solved:

[0004] Question 1: In tower data collection, when designing terminal equipment, it is necessary to ensure low-power long-distance transmission of Beidou data and other sensor data, and design the datagram protocol format to interact with the management platform;

[0005] Question 2: The visual management platform needs to have functions such as data transfer, log system, alarm system, equipment management and full-automatic control;

[0006] Question 3: When designing load balancing strategies and providing interfaces, the backend server of the distributed system must ensure the idempotence of the interfaces in the multi-Worker working mode and the performance in the face of high traffic.

[0007] Question 4: The Worker design of a distributed system needs to consider solving competition problems, avoid repeated task consumption and waste of resources, and realize Beidou processing and sensor data processing at the same time. Summary of the invention

[0008] In order to overcome the defects and shortcomings of the prior art, the present invention provides a distributed tower monitoring and scheduling system for multi-stage tasks, which effectively improves the throughput rate and task processing efficiency.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention provides a distributed tower monitoring and dispatching system for multi-stage tasks, including: multiple equipment terminals, a gateway, an edge monitoring platform, an edge client, a distributed background server, and a distributed service end node;

[0011] The device terminal is used to collect Beidou data for positioning the tower. Each device terminal exchanges data through the LoRa self-organizing network and transmits the data back to the edge client through the gateway;

[0012] The edge client submits Beidou data to the distributed background server in multiple batches in stages. The distributed background server is equipped with multiple distributed server nodes. The tasks are distributed to the distributed server nodes through Nginx load balancing. Each distributed server node task is handled by multiple Worker nodes. The Worker nodes implement task preemption through distributed locks, and multiple coroutines inside the Woker node perform concurrent processing.

[0013] The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform transmits tasks to the distributed background server for task processing by initiating tasks and asynchronously polling tasks.

[0014] As a preferred technical solution, the device terminal includes a dual power supply module, an MCU microcontroller, a LoRa radio frequency module, a Beidou processor, a gyroscope, a smoke sensor, and a temperature and humidity sensor;

[0015] The gyroscope collects tower attitude information, the Beidou processor is used to collect data from multiple satellites to perform data positioning of the tower, and the tower attitude information collected by the gyroscope is jointly solved. The smoke sensor is used to collect environmental smoke concentration, the temperature and humidity sensor is used to collect environmental temperature and humidity, the LoRa radio frequency module is used to provide a LoRa protocol interface, and perform LoRa communication with the gateway. The dual power supply module is used to provide dual power supply of solar energy and battery to the Beidou processor and MCU microcontroller, and the MCU microcontroller is used to control the dual power supply module to switch the power supply mode, and control the gyroscope, smoke sensor and temperature and humidity sensor to collect data, and encrypt and decrypt the data.

[0016] As an optimal technical solution, the distributed background server is provided with a task management module, which divides tasks into completed cold tasks and uncompleted hot tasks, and periodically performs table division operations based on the location information table, marking the table position begin_pos occupied by the task in the next scheduling in the location information table, marking the table position end_pos of the newly created task, and treating tasks in the interval [begin_pos, end_pos] as hot tasks.

[0017] As a preferred technical solution, the distributed background server is provided with a task retry module, which constructs a uniform retry mechanism and a progressive retry mechanism. In the progressive retry mechanism, the maximum number of retries max_retry_num, the current number of retries crt_retry_num and the maximum retry time interval max_retry_interval are used to determine the progressive retry time. The progressive retry time is:

[0018] retry_interval = min(1 < <crt_retry_num,max_retry_interval)。

[0019] The present invention also provides a dispatching method for a distributed tower monitoring dispatching system for multi-stage tasks, comprising the following steps:

[0020] The tower data is collected based on the device terminal, and the tower attitude information collected by the gyroscope is combined to achieve joint solution. Each device terminal exchanges data through the LoRa self-organizing network, and transmits the data back to the edge client through the gateway;

[0021] The edge client submits Beidou data to the distributed background server in multiple batches in stages. The distributed background server is equipped with multiple distributed server nodes. The tasks are distributed to the distributed server nodes through Nginx load balancing. Each distributed server node task is handled by multiple Worker nodes. The Worker nodes implement task preemption through distributed locks, and multiple coroutines inside the Woker node perform concurrent processing.

[0022] The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform sends tasks to the distributed background server for task processing by initiating tasks and asynchronous task polling;

[0023] The edge monitoring platform receives the message content sent by the gateway, parses the message content based on the set protocol format, sends multiple Beidou data in batches to request to update the context information of the current task, initiates a task creation request to the distributed background server, and fills the task priority field of the Beidou data with the highest level.

[0024] As a preferred technical solution, the task is distributed to the distributed server nodes through Nginx load balancing, specifically including:

[0025] The distributed backend server manages all tasks through a task information table, a task configuration table and a task location table. The task information table includes the actual task information pulled by the worker node, the task configuration table includes the pulled task configuration information, and the task location table includes the sub-table logic information;

[0026] Load balancing is achieved through the destination address hash scheduling algorithm strategy, a task registration interface is provided, the processing logic is solidified in the Worker, and the task scheduling and task execution are decoupled.

[0027] As a preferred technical solution, the actual task information pulled by the worker node is set with a priority. When the task is created and the single-stage task is executed successfully, the task scheduling time is determined according to the priority, which is expressed as:

[0028] order_time=modify_time-priority

[0029] When a task fails to execute or is found to be stuck, the next round of task scheduling time is determined based on the retry time, expressed as:

[0030] order_time=modify_time+retry_interval

[0031] Among them, order_time indicates the task scheduling time, modify_time indicates the task update time, priority indicates the task priority, and retry_interval indicates the task retry time;

[0032] Adding MQ message queue decouples task pulling and task possession.

[0033] As an optimal technical solution, after the server interface of the distributed background server is called, database operations are performed, a database connection pool is constructed, and optimization is performed based on the maximum number of long connections in the database connection pool and the maximum number of connections including long connections and short connections.

[0034] As a preferred technical solution, the Worker node implements task preemption through distributed locks, and multiple coroutines within the Woker node perform concurrent processing, specifically including:

[0035] Based on Redis distributed lock preemption tasks, Beidou processing logic and sensor data task processing logic are built in the worker program;

[0036] Set up watchdog mechanism and lock expiration mechanism;

[0037] In the Beidou processing logic, the Worker node obtains the MSM message in the RTCM frame data from the task context as observation data, obtains the positioning estimate through the single-point positioning algorithm and sends it to the CORS center. Based on the positioning estimate and the correction data calculated by the CORS center, the positioning result is obtained in the carrier relative positioning algorithm. The positioning result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds the threshold, the processing result that the tower is collapsing is sent.

[0038] In the sensor data task processing logic, the tilt result obtained by processing the sensor data with the Worker node is used as the Beidou polling frequency switch. When an abnormality is determined, the Beidou polling frequency is increased to monitor whether the tower is tilted by Beidou positioning.

[0039] As a preferred technical solution, multiple Beidou data are sent in batches to request to update the context information of the current task, specifically including:

[0040] Multiple Beidou data are used as a new Beidou data processing stage. The data in the first stage applies for a task ID from the distributed background server and calls the interface to notify the distributed background server to modify the context information;

[0041] The distributed backend server updates the context information to the context corresponding to the requested task, which serves as the original data for the next stage of scheduling of the task.

[0042] When the node is scheduled by the worker in the current stage, the worker takes out the intermediate value of the historical stage in the context and the new data packet of the current stage for single-point coarse positioning, and sends it to the CORS center to calculate the relative positioning result of the carrier. If the calculation result cannot be fixed, it is returned to the distributed background server for update. If the calculation result is successfully fixed, it is determined whether the tower is tilted, and the task status is reset to the final state, indicating that the multi-stage task of the node is completed.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) The present invention solves the limitation of blocking and waste when a single machine processes Beidou data through a distributed architecture, in which the terminal uses a single network in multiple networks to mount multiple nodes to reduce network transmission pressure.

[0045] (2) The present invention designs a universal background server framework, which is applicable to the scheduling of any multi-stage task scenario in addition to the tower monitoring scenario of this project, effectively improving the throughput and task processing efficiency, and its framework has strong scalability and applicability.

[0046] (3) The present invention uses a dynamic rolling table method to distinguish between cold and hot data, thereby solving the problem of low performance caused by excessive accumulated cold data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the overall architecture of the distributed tower monitoring and dispatching system for multi-stage tasks of the present invention;

[0048] Figure 2 It is a schematic diagram of the architecture of each module of the tower data acquisition device of the present invention;

[0049] Figure 3 It is a schematic diagram of the overall architecture of the management platform of the present invention;

[0050] Figure 4 This is a schematic diagram of the data processing flow of the management platform of the present invention;

[0051] Figure 5 A schematic diagram of task scheduling flow of a distributed backend server of the present invention;

[0052] Figure 6 It is a schematic diagram of the table partitioning and rolling strategy flow of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Example

[0055] like Figure 1 As shown, this embodiment provides a distributed tower monitoring and scheduling system for multi-stage tasks, which can effectively improve the ability of distributed concurrent processing and scheduling tasks and save server resources. The system includes:

[0056] The gateway and device terminals (i.e., tower data acquisition devices) at the data collection end are used to collect Beidou data. Data is exchanged between the various device terminals through the LoRa self-organizing network, and is transmitted back to the edge client through a secure intranet dedicated line at the gateway; the edge client submits Beidou data to the distributed background server in stages and batches, and the distributed background server solidifies the data into the database by allocating it to a certain node through load balancing and waiting for worker scheduling. The distributed background server is equipped with a task retry module and a task management module; the worker regularly schedules and pulls a batch of server-side tasks. Since the Beidou data of the same node is implemented asynchronously in multiple stages, the Beidou intermediate calculation result of the node will be returned after each scheduling, and the result will be re-solidified in the context field of the database. In particular, when the Beidou calculation reaches the solidification threshold, the task state corresponding to the node is set to the final state.

[0057] The device terminal and gateway are used as data aggregation access points. The device terminal is equipped with a Beidou monitoring module and transmits signals based on LoRa communication.

[0058] Monitoring and management are performed based on a visual edge monitoring platform. The gateway and the platform are connected in a unified manner through UDP communication. The platform initiates tasks and asynchronously polls tasks, which are then processed by distributed backend servers.

[0059] The distributed backend server has multiple scheduling servers. The monitoring platform distributes tasks to a server through Nginx load balancing, that is, multiple distributed server nodes are managed through Nginx. The connection pool design limits user access and reduces the extra overhead of creating connections. The tasks of each server are handled by multiple workers. Workers use distributed locks to implement task preemption, and multiple coroutines inside Woker perform concurrent processing, equipped with Beidou task processing logic and sensor data storage query logic.

[0060] like Figure 2As shown, the tower data acquisition equipment includes a dual power supply module equipped with solar panels and batteries, an MCU microcontroller, a LoRa radio frequency module, a Beidou processor, a gyroscope, a smoke sensor, and a temperature and humidity sensor. Compared with other terminal devices, the gateway of this embodiment has an additional W5500 Ethernet module for accessing the intranet monitoring platform at the edge. The gateway also has tower monitoring and aggregation functions.

[0061] This embodiment also provides a dispatching method for a distributed tower monitoring dispatching system for multi-stage tasks, comprising the following steps:

[0062] S1: In the tower data collection, a low-power long-distance transmission terminal device equipped with a Beidou processor is constructed, and its own datagram protocol format is constructed, and data is exchanged through the gateway and the edge monitoring platform through UDP;

[0063] The data format of the proprietary protocol is shown in Table 1 below:

[0064] Table 1 Schematic diagram of the data format of the proprietary protocol

[0065] FunctionCode RootAddr NodeAddr Len DataBuffer End 1 Bytes 2Bytes 2Bytes 2Bytes nBytes 2Bytes

[0066] In this embodiment, the Beidou processor is used to collect data from multiple satellites for data positioning of the tower, and combines the gyroscope to collect the tower attitude information to achieve joint solution. Since the Beidou solution occupies a large amount of wireless Mesh network resources and in the case of narrowband transmission, it will obviously cause network congestion. The amount of gyroscope data is extremely small, so the Beidou polling positioning is set to be triggered once every 5h. In other time periods, the gyroscope module is required to roughly calculate the offset of the tower. When the offset α≥0.5, the Beidou positioning module is awakened. In Beidou data acquisition, after demodulation processing by the Beidou antenna, the MSM message in the RTCM frame data is parsed and obtained, including the message header, navigation data, and observation data, which are used as the task context information mentioned below. In addition, the device terminal is also equipped with temperature and humidity sensors, smoke sensors, etc. as daily sensing monitoring objects. Under the harsh conditions of monitoring heavy rain, the priority of the Beidou solution task is increased. In particular, the number of satellites affects the positioning accuracy. Therefore, the device terminal needs to be placed in an open link with good signal as much as possible.

[0067] In this embodiment, the distance between adjacent towers is at the kilometer level, so a long-distance transmission communication module is required; LoRa is selected as the radio frequency networking transmission module, and the module uses the SX1278 model chip with an integrated LoRa protocol interface.

[0068] In this embodiment, the MCU microcontroller uses STM32F103ZET6 as the main control processor, transplants the FreeRTOS operating system as the kernel, implements AES data encryption and decryption, and sensor data collection tasks in each node; and additionally implements network management and timing task execution in each aggregation gateway. Specifically, under low power consumption requirements, the MCU microcontroller implements automatic switching between sleep mode and working mode to save power consumption, and implements switching between solar cells and 3.3v auxiliary power supplies to ensure long-term power supply.

[0069] S2: The visual edge monitoring platform is equipped with a two-way communication protocol with multiple terminal devices and multiple distributed systems, and has functions such as data transfer, log system, alarm system, equipment management and fully automatic control.

[0070] In this embodiment, the visual management platform uses QT for desktop development, the edge management platform receives the information sent by the gateway aggregation node, parses the message payload content based on the proprietary protocol format, and obtains the RootAddr, NodeAddr and DataBuffer data in the above proprietary protocol package. Preferably, in the implementation of the fully automatic control function, the message packaging instructions include autonomous route modification, sensor data polling, Beidou polling, device status information polling, etc. Receive terminal data through a reliable intranet wired transmission method, and at the same time have a log operation record management module, a user management module, a data analysis and ledger display, a login registration module, and a data two-way transmission module that realizes two-way data transmission with the data acquisition device and the background server through HTTP;

[0071] In this embodiment, the edge monitoring platform realizes two-way communication with the aggregation gateway in the intranet environment through a reliable UDP wired transmission method. The data packet follows the custom protocol format, and the message includes information such as the gateway node number, the monitoring terminal number, the function-code instruction code, and data. The instruction code consists of 1 Byte and is partitioned into 256 types of instructions ranging from 0 to 127 and 128 to 255, which are used to identify the instruction code above the device and the instruction code area issued by the management platform, so as to increase and manage instructions.

[0072] In particular, for Beidou data transmission, there is a bottleneck of low bandwidth between nodes, which leads to limited data transmission speed. There is a large idle period between two adjacent data of the same node. Based on this, the concept of batch processing is proposed, and multiple batches of data of the same node represent multiple stages, allowing Workers to asynchronously execute tasks of other nodes while waiting for the next stage of data during the idle period. Therefore, when receiving Beidou data, the edge monitoring platform will send a request to update the context information of the current task in batches of n data, and when initiating a task creation request to the distributed background server, fill in the Beidou task priority field with the highest level.

[0073] Preferably, in order to realize the automatic management of tower monitoring and terminal equipment, this embodiment provides a variety of control instructions for remote control, including but not limited to Beidou polling, sensor data polling, device status information polling, device remote reset, node routing autonomous change, etc. Among them, the instructions follow the above-mentioned proprietary protocol packaging method, FunctionCode represents different instruction types, and sends instructions to the specified node through RootAddr and NodeAddr.

[0074] like Figure 3 As shown, the designed management platform includes a data acquisition and transmission module, a log management module, a user login and registration module, and a data display and analysis management module.

[0075] like Figure 4 As shown, for the data processing flow, taking Beidou data as an example, when a node starts the Beidou module, after collecting batch data as a new Beidou data processing stage, the data of the first stage will apply for a task ID to the distributed background server, and call the interface to notify the distributed background server to modify the context information, where the context contains 5 Beidou data. Further, the distributed background server updates the context to the context corresponding to the previously applied task as the original data for the next stage scheduling of the task. When the node is scheduled by the worker in the current stage, the worker takes out the intermediate value of the historical stage in the context and the new data packet of the current stage for single-point coarse positioning, and hands it over to the CORS center to calculate the relative positioning result of the carrier. When the calculation result cannot be fixed, the intermediate result of this stage is returned again to update the distributed background server. On the contrary, if it is successfully fixed, it is determined whether it is tilted, and the task state is reset to the final state, indicating that the multi-stage task of the node is over.

[0076] S3: The design of the distributed backend server includes load balancing strategies, provides HTTP interfaces with the management platform and Worker, has interfaces such as task registration, task query, task preemption, and task creation, implements modules such as scheduling, task retry, and task governance, and ensures the idempotence of interfaces and high traffic performance in the multi-Worker working mode through unique indexes and distributed scheduling. In particular, the Sever is versatile and allows other task requirements to be accessed as a general framework, and the specific task processing is implemented by the Worker.

[0077] In this embodiment, the distributed background server is allowed to receive at least 3000 task requests per second, and the tasks are solidified into the master-slave database. All tasks are managed through the task information table, the task configuration table and the task location table, wherein the task information table is divided and rolled with 500W as the limit, and has different processing priorities for two major types of tasks such as Beidou data processing and sensor data processing tasks. Optionally, Beidou data processing has timeliness requirements, so its task priority is much higher than the sensor data processing task. Preferably, the background server provides interfaces such as task registration, task query, task preemption, and task creation, realizes scheduling, task retry, task governance and other modules, and realizes load balancing through the destination address hash scheduling algorithm strategy. In particular, the designed background server can be used as a universal framework to receive and schedule any task, provide a task registration interface, and the processing logic of the actual task is independently solidified in the Worker by the business party, realizing the decoupling of task scheduling and task execution. Preferably, the background server improves the responsiveness by optimizing the number of connection pools.

[0078] The distributed background server of this embodiment adopts the destination address hash scheduling algorithm to realize the routing load balancing distribution, which can solve the problems of poor disaster recovery performance and low efficiency caused by the centralized server.

[0079] In this embodiment, the background server provides multiple interfaces to the management platform and Worker, which are composed of a task information table, a task configuration table, and a location information table for table partitioning logic. The task information table is the actual task information to be pulled by the worker; the task configuration table includes the configuration of each task pull amount, task pull interval, and maximum task retry times; the location information table is used for rolling table partitioning, that is, Figure 6 The task is persisted in the task information table of the MySQL database. The task information table contains a unique task ID index to ensure the idempotence of the interface. The unique task ID is generated by splicing the table number based on the snowflake algorithm. The table number is used by the management platform to query the task status and update the task context. In the interface design, the content-type selects application / json as the data transmission format, which is encapsulated and parsed through serialization and deserialization. The interface design is shown in Table 2 below:

[0080] Table 2 Interface design diagram

[0081]

[0082] like Figure 5 As shown, according to the characteristics of task execution in this embodiment, tasks are divided into completed cold tasks and uncompleted hot tasks, so the table design also adopts the rolling table mode of cold and hot tables, such as Figure 6As shown in the figure, the implementation of the sub-table is executed regularly by the task management module in combination with the location information table. Therefore, there is a delay in the design of the sub-table, which allows the number of tasks to be greater than the threshold of 5 million in a short period of time. The begin_pos of the location information table marks the table position occupied by the task in the next scheduling, and the end_pos marks the table position of the newly created task. The tasks in the interval [begin_pos, end_pos] are recorded as the above hot tasks.

[0083] Combination Figure 5 As shown, in order to ensure that the task is successfully executed, the task management module regularly monitors the stuck task and restores it to the pending state. At the same time, a retry mechanism is designed. Optionally, the general task scheduling server framework of this embodiment provides a uniform retry mechanism and a progressive retry mechanism. In the progressive retry mechanism, the maximum number of retries max_retry_num, the current number of retries crt_retry_num and the maximum retry time interval max_retry_interval are determined. The retry time starts from 1s, and the progressive retry time is:

[0084] retry_interval = min(1 < <crt_retry_num,max_retry_interval)

[0085] In particular, during the Beidou data transmission process, there is a large idle period between two adjacent data. Based on this, the management platform assembles n data as a batch of data. When designing the scheduling system, multiple batches of data are converted into multi-stage concepts for processing. The processing results of the previous stage are stored in the context. During the gap period between the multiple stages of the current task, other tasks will be asynchronously scheduled for execution. This improves scheduling efficiency and reduces resource waste.

[0086] Preferably, task pulling needs to confirm which tasks are pulled first, and the pulling of tasks absolutely depends on the execution status of the tasks, that is, the tasks pulled as the next batch of scheduling must be in a waiting state. At the same time, the order of pulling tasks is related to the task update time modify_time, the task priority priority and the task retry time retry_interval. In the design of this embodiment, the three are coupled as order_time. When pulling tasks, the first n tasks in ascending order of order_time are selected to reduce the query time of task pulling by reducing the number of joint indexes. Among them, the task update time follows the first-in-first-out principle to ensure that new tasks are placed at the end of the queue; in this embodiment, the task priority level is abstracted as the time to execute the task in advance of priority, that is, the tasks with higher priority have more advance time. For this embodiment, the Beidou processing task has the highest priority, so its priority is 1 year, which can be considered as infinite priority; in this embodiment, the retry logic of the task is reflected in order_time, and the retry time is accumulated in order_time to ensure that the tasks within the retry time cannot be scheduled, so as to prevent abnormal tasks from blocking the server. According to the above constraints, order_time will be updated in the following stages:

[0087] 1) When the task is created and the single-stage task is executed successfully, the task scheduling time is determined according to the priority:

[0088] order_time=modify_time-priority

[0089] 2) When the task execution fails and the governance module finds that the task is stuck, the next round of scheduling time is determined according to the above retry time to ensure that the task within the retry time cannot be scheduled:

[0090] order_time=modify_time+retry_interval

[0091] Preferably, in order to further solve the problem of worker resource waste caused by multiple workers competing for tasks, it is allowed to join the MQ message queue to decouple task pulling and task possession, and pull a batch of tasks into the message queue in advance to reduce the time consumption of task possession and improve the response speed of task possession.

[0092] Preferably, in order to further prevent the exhaustion of the server port and reduce the link creation cost when receiving requests, in this embodiment, database operations need to be performed after the server interface is called. In order to reduce the consumption caused by frequent creation and destruction of the database, this example optimizes the design of the database connection pool. When the server receives a request from a client or a worker, it needs to establish a connection with the database. At this time, a batch of database connections are maintained for reuse, which can reduce consumption and improve performance. According to the maximum number of long connections in the pool and the maximum number of connections including long connections and short connections, the maximum number of long connections in the connection pool is selected as 1000 and the maximum number of connections is 2000.

[0093] S4: The Worker of the distributed system is the main part of task processing. It preempts tasks through distributed locks and implements Beidou processing and sensor data processing.

[0094] In this embodiment, multiple workers solve the task competition problem through Redis distributed locks, and introduce watchdog mechanism and lock expiration mechanism to ensure the non-repeatable consumption of preempted tasks. In particular, according to the nature of this project, Beidou processing logic and sensor data task processing logic are uniquely designed. Beidou positioning adopts high-precision point-checking positioning based on network RTK.

[0095] Specifically, in the design of Worker execution tasks:

[0096] Multi-worker effectively solves the problem of task competition through Redis distributed locks. In a distributed system, multiple workers may try to process the same task at the same time, which will lead to repeated execution of tasks and waste of resources. Redis distributed locks provide a reliable mechanism to ensure that only one worker can process a specific task at any given time.

[0097] In order to further ensure the correct execution of tasks, a watchdog mechanism and a lock expiration mechanism are introduced. The watchdog mechanism can monitor the running status of the Worker. When an abnormality occurs in the Worker, timely measures can be taken, such as releasing the lock so that other Workers can take over the task. The lock expiration mechanism can prevent the Worker from being unable to release the lock normally for some reason after acquiring the lock, resulting in the task being occupied for a long time. By setting a reasonable lock expiration time, it can be ensured that the lock can be automatically released when a Worker fails, thereby ensuring the executability of the task and the stability of the system.

[0098] In this embodiment, Beidou processing logic and sensor data task processing logic are designed. Beidou positioning adopts high-precision differential positioning based on network RTK, using observation data from multiple base stations, and eliminating or weakening errors in the satellite signal propagation process through differential algorithms, thereby improving positioning accuracy. In RTK high-precision differential positioning, the Worker obtains the MSM message in the above RTCM frame data from the context of the task as observation data, obtains the positioning estimate through the single-point positioning algorithm and sends it to the CORS center. According to the positioning estimate and the correction data calculated by the CORS center, a high-precision positioning result is further obtained in the carrier relative positioning algorithm. The result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds the threshold of 20cm, the processing result that the tower is collapsing is sent to the management system.

[0099] For the sensor data task processing logic, corresponding processing algorithms and processes are designed according to the characteristics of different types of sensor data. For example, for sensor data such as gyroscopes and vibrations, abnormal conditions of poles and towers can be discovered in time by analyzing their changing trends. Since a Beidou polling will occupy the network between terminal nodes for a long time, and since the Beidou data packets are large, frequent Beidou polling will cause the terminal to consume power quickly and the bandwidth to be occupied for a long time. Based on this, this embodiment provides sensors with smaller data volumes such as gyroscopes as high-frequency tilt monitoring means, and the tilt result obtained by Worker processing sensor data is used as the Beidou polling frequency switch. When an abnormality is found, the Beidou polling frequency is increased, and Beidou's high-precision positioning is used to monitor whether the pole tower is tilted. In addition, combined with the management platform's needs for record analysis and display of historical sensor data, Worker provides the ability to filter and query database historical data.

[0100] The present invention fully considers the multi-stage time-consuming task types in pole tower monitoring, and designs a distributed system suitable for multi-tower monitoring. The system collects data through the designed monitoring terminal nodes, and accesses the monitoring center for processing through multiple nodes sharing the same gateway. Specifically, the design of the processing system uses distributed multi-servers as the task management center to provide interfaces to the client to initiate task creation and query, and provide task pull interfaces for workers. Specific time-consuming tasks are handed over to workers for concurrent processing. And the design focuses on the task governance module, multi-mode retry mechanism, task scheduling process and other parts. And design a visualization platform to intuitively display the monitoring and processing results, which is convenient for users to monitor and manage.

[0101] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A distributed tower monitoring and dispatching system for multi-stage tasks, characterized in that: include: Multiple device terminals, gateways, edge monitoring platforms, edge clients, distributed backend servers, and distributed server nodes; The device terminal is used to collect Beidou data and locate the tower. Each device terminal interacts with data through the LoRa self-organizing network and transmits the data back to the edge client through the gateway; The edge client submits Beidou data to the distributed background server in multiple batches in stages. The distributed background server is equipped with multiple distributed server nodes. The tasks are distributed to the distributed server nodes through Nginx load balancing. Each distributed server node task is handled by multiple Worker nodes. The Worker nodes implement task preemption through distributed locks, and multiple coroutines inside the Woker node perform concurrent processing. The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform transmits tasks to the distributed background server for task processing by initiating tasks and asynchronously polling tasks.

2. The distributed tower monitoring and dispatching system for multi-stage tasks according to claim 1 is characterized in that: The device terminal includes a dual power supply module, an MCU microcontroller, a LoRa radio frequency module, a Beidou processor, a gyroscope, a smoke sensor, and a temperature and humidity sensor; The gyroscope collects tower attitude information, the Beidou processor is used to collect data from multiple satellites to perform data positioning of the tower, and the tower attitude information collected by the gyroscope is jointly solved. The smoke sensor is used to collect environmental smoke concentration, the temperature and humidity sensor is used to collect environmental temperature and humidity, the LoRa radio frequency module is used to provide a LoRa protocol interface, and perform LoRa communication with the gateway. The dual power supply module is used to provide dual power supply of solar energy and battery to the Beidou processor and MCU microcontroller, and the MCU microcontroller is used to control the dual power supply module to switch the power supply mode, and control the gyroscope, smoke sensor and temperature and humidity sensor to collect data, and encrypt and decrypt the data.

3. The distributed tower monitoring and dispatching system for multi-stage tasks according to claim 1 is characterized in that: The distributed background server is provided with a task management module, which divides tasks into completed cold tasks and uncompleted hot tasks, and performs table division operations based on the location information table at regular intervals, marking the table position begin_pos occupied by the task in the next scheduling in the location information table, marking the table position end_pos of the newly created task, and treating the tasks in the interval [begin_pos, end_pos] as hot tasks.

4. The distributed tower monitoring and dispatching system for multi-stage tasks according to claim 1 is characterized in that: The distributed background server is provided with a task retry module, which constructs a uniform retry mechanism and a progressive retry mechanism. In the progressive retry mechanism, the maximum number of retries max_retry_num, the current number of retries crt_retry_num and the maximum retry time interval max_retry_interval are used to determine the progressive retry time. The progressive retry time is: retry_interval = min(1 < <crt_retry_num,max_retry_interval)。 5. The dispatching method of the distributed tower monitoring and dispatching system for multi-stage tasks according to claims 1-4, characterized in that: The steps include: The tower data is collected based on the device terminal, and the tower attitude information collected by the gyroscope is combined to achieve joint solution. Each device terminal exchanges data through the LoRa self-organizing network, and transmits the data back to the edge client through the gateway; The edge client submits Beidou data to the distributed background server in multiple batches in stages. The distributed background server is equipped with multiple distributed server nodes. The tasks are distributed to the distributed server nodes through Nginx load balancing. Each distributed server node task is handled by multiple Worker nodes. The Worker nodes implement task preemption through distributed locks, and multiple coroutines inside the Woker node perform concurrent processing. The gateway transmits signals to the edge monitoring platform, and the edge monitoring platform sends tasks to the distributed background server for task processing by initiating tasks and asynchronous task polling; The edge monitoring platform receives the message content sent by the gateway, parses the message content based on the set protocol format, sends multiple Beidou data in batches to request to update the context information of the current task, initiates a task creation request to the distributed background server, and fills the task priority field of the Beidou data with the highest level.

6. The dispatching method of the distributed tower monitoring and dispatching system for multi-stage tasks according to claim 5, characterized in that: The task is distributed to the distributed server nodes through Nginx load balancing, specifically including: The distributed backend server manages all tasks through a task information table, a task configuration table and a task location table. The task information table includes the actual task information pulled by the worker node, the task configuration table includes the pulled task configuration information, and the task location table includes the sub-table logic information; Load balancing is achieved through the destination address hash scheduling algorithm strategy, a task registration interface is provided, the processing logic is solidified in the Worker, and the task scheduling and task execution are decoupled.

7. The dispatching method of the distributed tower monitoring and dispatching system for multi-stage tasks according to claim 5, characterized in that: The actual task information pulled by the worker node is set with a priority. When the task is created and the single-stage task is executed successfully, the task scheduling time is determined according to the priority, which is expressed as: order_time=modify_time-priority When a task fails to execute or is found to be stuck, the next round of task scheduling time is determined based on the retry time, expressed as: order_time=modify_time+retry_interval in, order_time indicates the task scheduling time, modify_time indicates the task update time, priority indicates the task priority, and retry_interval indicates the task retry time; Adding MQ message queue decouples task pulling and task possession.

8. The dispatching method of the distributed tower monitoring and dispatching system for multi-stage tasks according to claim 5, characterized in that: After the server-side interface of the distributed background server is called, database operations are performed, a database connection pool is constructed, and optimization is performed based on the maximum number of long connections in the database connection pool and the maximum number of connections including long connections and short connections.

9. The dispatching method of the distributed tower monitoring and dispatching system for multi-stage tasks according to claim 5, characterized in that: The Worker node implements task preemption through distributed locks, and multiple coroutines within the Woker node perform concurrent processing, specifically including: Based on Redis distributed lock preemption tasks, Beidou processing logic and sensor data task processing logic are built in the worker program; Set up watchdog mechanism and lock expiration mechanism; In the Beidou processing logic, the Worker node obtains the MSM message in the RTCM frame data from the task context as observation data, obtains the positioning estimate through the single-point positioning algorithm and sends it to the CORS center. Based on the positioning estimate and the correction data calculated by the CORS center, the positioning result is obtained in the carrier relative positioning algorithm. The positioning result is compared with the original tower positioning to obtain the tower offset. When the offset exceeds the threshold, the processing result that the tower is collapsing is sent. In the sensor data task processing logic, the tilt result obtained by processing the sensor data with the Worker node is used as the Beidou polling frequency switch. When an abnormality is determined, the Beidou polling frequency is increased to monitor whether the tower is tilted by Beidou positioning.

10. The dispatching method of the distributed tower monitoring dispatching system for multi-stage tasks according to claim 5, characterized in that: Send multiple Beidou data in batches to request to update the context information of the current task, including: Multiple Beidou data are used as a new Beidou data processing stage. The data in the first stage applies for a task ID from the distributed background server and calls the interface to notify the distributed background server to modify the context information; The distributed backend server updates the context information to the context corresponding to the requested task, which serves as the original data for the next stage of scheduling of the task. When the node is scheduled by the worker in the current stage, the worker takes out the intermediate value of the historical stage in the context and the new data packet of the current stage for single-point coarse positioning, and sends it to the CORS center to calculate the relative positioning result of the carrier. If the calculation result cannot be fixed, it is returned to the distributed background server for update. If the calculation result is successfully fixed, it is determined whether the tower is tilted, and the task status is reset to the final state, indicating that the multi-stage task of the node is completed.

Images