A drone cooperation system

Abstract

The application provides a UAV cooperation system, comprising a UAV, an edge computing node, a cloud and at least two operation terminals; the edge computing node is in communication connection with the UAV and the cloud respectively, and the cloud is also in communication connection with each operation terminal; the UAV and the edge computing node, the edge computing node and the cloud, and the cloud and the operation terminal all adopt a first communication protocol to transmit flight data; the UAV, the edge computing node, the cloud and each operation terminal adopt a second communication protocol to transmit video data; wherein the first communication protocol and the second communication protocol are independent of each other, the system adopts double communication protocols to respectively transmit flight control instructions and video data, which helps to reduce the interference of video data on the transmission of flight control instructions, improves the transmission delay and reliability of flight control instructions in a weak network environment to some extent, and improves the safety and real-time control performance of the UAV in flight.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV collaborative system. Background Technology

[0002] In recent years, drone technology has been widely used in emergency rescue, security patrols, power line inspections, and forest fire prevention. In these complex application scenarios, multiple operators often need to coordinate the control of the same or multiple drones through their respective operating terminals to achieve wide-area coverage, parallel task execution, and real-time information sharing.

[0003] However, existing systems typically transmit flight control commands and video data transmitted back by the UAV using the same communication protocol or transmission channel. Because video data is large in volume and continuously consumes bandwidth, when video data transmission consumes significant bandwidth resources, the transmission of flight control commands will be severely interfered with, causing delays or even loss of control commands, threatening the flight safety of the UAV. Summary of the Invention

[0004] This application provides a drone collaborative system that uses dual communication protocols to transmit flight control commands and video data separately. This helps reduce the interference caused by video data to the transmission of flight control commands, improves the transmission delay and reliability of flight control commands in weak network environments to a certain extent, and enhances the safety and real-time control of drones during flight.

[0005] In a first aspect, embodiments of this application provide a drone collaboration system, comprising: a drone, an edge computing node, a cloud, and at least two operating terminals; the edge computing node is communicatively connected to both the drone and the cloud, and the cloud is also communicatively connected to each of the operating terminals; the drone and the edge computing node transmit flight data using a first communication protocol, the edge computing node and the cloud transmit the flight data using the first communication protocol, and the cloud and the operating terminals transmit the flight data using the first communication protocol; the drone transmits video data to the edge computing node via a second communication protocol, the edge computing node transmits the video data to the cloud via the second communication protocol, and the cloud transmits the video data to each of the operating terminals via the second communication protocol; wherein the first communication protocol and the second communication protocol are independent of each other.

[0006] In one or more embodiments, the edge computing node is configured to: detect the communication link quality parameters between the edge computing node and the cloud in real time; and when the communication link quality parameters meet the delay condition, process the flight control commands in a hierarchical manner based on the priority of the flight control commands.

[0007] In one or more embodiments, the step of processing the flight control commands in a hierarchical manner based on their priority includes: dividing the flight control commands into regular flight control commands and emergency flight control commands, caching the regular flight control commands locally, and sending the emergency flight control commands to the UAV.

[0008] In one or more embodiments, the cloud is configured to: when receiving flight control commands from at least two of the operating terminals within the same time window, perform conflict arbitration based on the role priority of each operating terminal to determine the flight control command to be executed; and send the flight control command to be executed to the UAV via the edge computing node.

[0009] In one or more embodiments, the cloud is further configured to broadcast conflict arbitration results to each of the operating terminals.

[0010] In one or more embodiments, the cloud is further configured to: upon receiving a control transfer request from an operating terminal to which control is to be transferred, send a hovering command to the drone via the edge computing node, and send a takeover confirmation request to the operating terminal to which control is to be transferred; receive confirmation takeover information sent by the operating terminal to which control is to be transferred, and complete the control transfer.

[0011] In one or more embodiments, the cloud is further configured to broadcast the control transfer result to each of the operating terminals after the control transfer is completed.

[0012] In one or more embodiments, the cloud is further configured to: in response to user input, divide the search area into multiple search sub-areas; distribute the search tasks for each of the search sub-areas to the corresponding operating terminals, so that the operating terminals control the corresponding drones to perform search tasks in the corresponding search sub-areas; and display the search results on the collaborative map based on the target data reported by each of the drones.

[0013] In one or more embodiments, the cloud is further configured to: receive video image annotation data sent by the operating terminal, the video image annotation data including pixel coordinates of annotated areas in the video image returned by the user from the drone and corresponding video frame information; receive pose data of the drone, the pose data including the drone's geographical location, flight attitude and flight altitude; calculate the three-dimensional geographic coordinates corresponding to the annotated areas based on the video image annotation data and the pose data; and generate abnormal annotation anchor points on the collaborative map based on the video image annotation data and the three-dimensional geographic coordinates.

[0014] In one or more embodiments, the cloud is further configured to: employ a conflict-free copy data type algorithm to merge and process the collaborative map data submitted concurrently by each of the operating terminals, so that the collaborative map displayed by each of the operating terminals and the collaborative map displayed by the cloud are consistent.

[0015] The beneficial effects of this application are as follows: This application provides a drone collaborative system, including: a drone, an edge computing node, a cloud, and at least two operating terminals; the edge computing node is communicatively connected to both the drone and the cloud, and the cloud is also communicatively connected to each operating terminal; the drone and the edge computing node transmit flight data using a first communication protocol, the edge computing node and the cloud transmit flight data using a first communication protocol, and the cloud and the operating terminals transmit flight data using a second communication protocol; the drone transmits video data to the edge computing node via a second communication protocol, the edge computing node transmits video data to the cloud via a second communication protocol, and the cloud transmits video data to each operating terminal via a second communication protocol; wherein, the first communication protocol and the second communication protocol are independent of each other, and the system uses dual communication protocols to transmit flight control commands and video data respectively, which helps to reduce the interference caused by video data to the transmission of flight control commands, improves the transmission delay and reliability of flight control commands in weak network environments to a certain extent, and improves the safety and real-time control of the drone during flight. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0017] Figure 1 An exemplary block diagram of a drone collaborative system is shown; Figure 2 An example is shown in the timing diagram of a drone collaboration system; Figure 3 An example diagram of a collaborative map is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0020] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, the terms "first" and "second" used herein do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0021] With the rapid development of drone technology, its application scenarios have gradually expanded from the traditional single-person, single-drone operation mode to multi-drone collaboration and multi-department joint operations. In complex mission scenarios such as large-scale search and rescue and cross-regional industrial inspection, multiple operators, industry experts, and commanders are often required to collaborate on the real-time control and data analysis of drones, which places high demands on the system's multi-terminal concurrent access capabilities and real-time collaboration efficiency.

[0022] However, existing UAV control systems mostly adopt a "one-to-one" local remote control or single-terminal control mode, which is difficult to effectively support the application requirements of multi-terminal real-time collaboration. Especially in scenarios with weak networks or network fluctuations, existing systems generally mix video stream data and high-frequency flight control commands in the same transmission channel. Due to the large data volume and continuous bandwidth consumption of video stream data, it is very easy to cause transmission channel congestion, which in turn leads to delays or even loss of critical flight control commands, posing a serious threat to the flight safety of UAVs.

[0023] In view of this, this application provides a drone collaboration system. The system constructs a dual-channel hybrid transmission architecture in which the control command channel and the video data channel are independent of each other, and introduces edge computing nodes to perform localized priority processing of control commands. This helps to reduce the interference of video data on the transmission of flight control commands, and to a certain extent improves the transmission delay and reliability of key flight control commands in weak network environments, thereby improving flight safety and real-time control performance in multi-terminal collaboration scenarios.

[0024] See Figure 1The drone collaboration system 100 provided in this application includes: a drone 10, an edge computing node 20, a cloud platform 30, and at least two operating terminals 40. The edge computing node 20 is communicatively connected to both the drone 10 and the cloud platform 30, and the cloud platform 30 is also communicatively connected to each operating terminal 40. The drone 10 transmits flight data to the edge computing node 20 using a first communication protocol, the edge computing node 20 transmits flight data to the cloud platform 30 using a second communication protocol, and the cloud platform 30 transmits flight data to each operating terminal 40 using a second communication protocol. The drone 10 transmits video data to the edge computing node 20 via a second communication protocol, the edge computing node 20 transmits video data to the cloud platform 30 via a second communication protocol, and the cloud platform 30 transmits video data to each operating terminal 40 via a second communication protocol. The flight data includes flight control commands, telemetry data, and heartbeat packets. The first and second communication protocols are independent of each other.

[0025] The UAV 10 refers to an unmanned aerial vehicle equipped with a flight control unit, sensors, and a communication module. The flight control unit, mounted on the UAV 10, is the core control module responsible for parsing flight control commands and driving the UAV 10 to execute corresponding flight maneuvers. Upon receiving flight control commands, the flight control unit parses the command content and generates corresponding motor drive signals to control the UAV 10's flight direction, speed, altitude, attitude, and other flight status parameters. Simultaneously, it encapsulates sensor data and transmits it externally via the communication module, allowing the operating terminal 40 to monitor the UAV 10's flight status in real time. Sensors include, but are not limited to, optical cameras, lidar, infrared sensors, and inertial navigation units, used to collect environmental data in real time, such as images, terrain, temperature, and attitude. The communication module, mounted on the UAV 10, is a wireless communication unit responsible for establishing a wireless communication link between the UAV 10 and the edge computing node 20, enabling bidirectional transmission of flight data and video data. The communication module supports concurrent transmission of the first communication protocol and the second communication protocol. Flight data is transmitted and received through the first communication protocol transmission channel, while video data is transmitted to the edge computing node 20 in real time through the second communication protocol transmission channel and distributed to each operating terminal 40 after being forwarded level by level.

[0026] Edge computing node 20 (Edge Node) is a computing device deployed close to the operational site of drone 10, acting as a regional proxy node between drone 10 and cloud 30. Edge computing node 20 can locally relay and preprocess flight control commands and video data. Furthermore, edge computing node 20 and cloud 30 employ a differential synchronization mechanism to transmit status data. Specifically, each time edge computing node 20 reports data from drone 10 to cloud 30, it only extracts and transmits the status increments (differential data) that have changed compared to the previous report, rather than the full status data. This significantly reduces the communication bandwidth usage between edge computing node 20 and cloud 30 while ensuring the real-time performance and integrity of data from cloud 30, helping to maintain data transmission stability and low latency in scenarios with limited network conditions.

[0027] Cloud 30 refers to a collaborative management platform deployed on a remote server or cloud computing platform. It is responsible for aggregating and uniformly processing flight control commands and collaborative business data sent by various operating terminals 40, including but not limited to performing multi-terminal command conflict arbitration, transferring management control, issuing task orchestration commands, and distributing video data transmitted back by UAV 10 to various operating terminals 40. After receiving differential data reported by edge computing node 20, Cloud 30 merges it with local full status data to complete status synchronization, and pushes the updated status data to the relevant operating terminals 40 when needed.

[0028] The operating terminal 40 refers to the human-machine interaction device that allows operators to access the UAV collaboration system 100, including but not limited to remote controllers, tablet computers, PC browsers, or dedicated ground station software.

[0029] In this application, at least two operation terminals 40 are included to support multi-terminal concurrent collaboration scenarios. Each operation terminal 40 sends flight control commands to the cloud 30 according to its assigned role permissions and receives video data transmitted back by the drone 10 in real time. When accessing the system, each operation terminal 40 can be uniformly assigned a unique role identifier by the cloud 30. The role identifier is used to uniquely identify the role type and corresponding permission level of the operation terminal 40 in the multi-user collaboration system. When executing business logic such as command conflict arbitration, control right management, and arbitration result broadcasting, the cloud 30 uses the role identifier as the basis for identifying the operation terminal 40.

[0030] Specifically, such as Figure 2As shown, the roles of the operating terminal 40 can be divided into Pilot-in-Command (PIC) terminal 41 and Collaborative Observer (CO) terminal 42. The role identifier of PIC terminal 41 corresponds to a higher role priority, while the role identifier of CO terminal 42 corresponds to a lower role priority. The correspondence between role identifiers and role priorities is uniformly configured by the cloud 30 during the system initialization phase. PIC terminal 41 is operated by the PIC, and CO terminal 42 is operated by the CO. In the multi-user collaborative system, PIC terminal 41 has the highest flight control authority, responsible for the direct piloting and flight safety management of UAV 10, and can send all flight control commands to the cloud 30, including regular flight control commands and emergency flight control commands. CO terminal 42, in the multi-user collaborative system, has permissions such as video viewing, gimbal control, or business annotation, mainly used by the CO for real-time monitoring of the screen and task annotation. The aforementioned role and permission system is uniformly managed and arbitrated by the cloud 30. When multiple operating terminals 40 issue conflicting flight control commands at the same time, the cloud 30 arbitrates according to the role priority rules to ensure that the command of PIC terminal 41 has the highest execution priority, thereby ensuring flight safety.

[0031] In some embodiments, the role of the operating terminal 40 also includes a command center terminal. The command center terminal is used by a user with the authority of a regional manager. Its main responsibility is to perform command and dispatch operations such as dividing the search area and assigning tasks to sub-areas through the user interface of the cloud 30, and to receive target discovery alarm notifications pushed by the cloud 30 in real time. The command center terminal does not directly participate in the flight control of the UAV 10; its operational authority focuses on task orchestration and global situational awareness. It complements the PIC terminal 41 and CO terminal 42 in terms of function, together forming a complete role and authority framework under a multi-user collaborative system.

[0032] At the data transmission level, this application employs a dual-channel approach, with independent first and second communication protocols, to transmit flight data and video data respectively. The flight data includes flight control commands, telemetry data, and heartbeat packets. Flight control commands are control commands generated by the operating terminal 40 and forwarded to the UAV 10 via the cloud 30 and edge computing nodes 20. These commands include, but are not limited to, conventional flight control commands such as flight direction, speed, altitude, and gimbal attitude adjustment, as well as emergency flight control commands such as emergency obstacle avoidance, hovering, and automatic return. Telemetry data includes flight status parameters such as the UAV 10's geographical location, flight attitude, and flight altitude. Heartbeat packets are used to maintain the communication connection between nodes and monitor the link health in real time. The aforementioned flight data is transmitted bidirectionally between the UAV 10 and the edge computing node 20, between the edge computing node 20 and the cloud 30, and between the cloud 30 and each operating terminal 40, all via the first communication protocol. The first communication protocol is a lightweight communication protocol used for transmitting flight data. In one or more embodiments of this application, the first communication protocol is the Message Queuing Telemetry Transport (MQTT) protocol. The MQTT protocol is based on the publish / subscribe model and has the characteristics of small message size, low bandwidth consumption, and support for graded quality of service. In weak network or network fluctuation environments, it helps to achieve high reliability transmission and low latency delivery of flight data and is suitable for flight control scenarios with high requirements for latency and reliability.

[0033] Video data refers to video stream data that is collected and encoded in real time by the onboard camera of the UAV 10. After being forwarded by the edge computing node 20 and the cloud 30, it is finally distributed to each operating terminal 40 for operators to view in real time. The video data is transmitted unidirectionally from the UAV 10 to the edge computing node 20 via the second communication protocol, and then from the edge computing node 20 to the cloud 30 via the second communication protocol. After receiving the video stream uploaded by the edge computing node 20, the Selective Forwarding Unit (SFU) server deployed in the cloud 30 selectively forwards the video data to each operating terminal 40 according to their subscription requirements via the second communication protocol. This eliminates the need for mixing or decoding the video stream, thereby reducing server computing overhead while supporting concurrent reception by multiple operating terminals 40. The second communication protocol is a streaming media communication protocol used to transmit real-time video data. In one or more embodiments of this application, the second communication protocol is the Web Real-Time Communication (WebRTC) protocol. The WebRTC protocol natively supports the distribution of multiple video streams via an SFU server, featuring low latency and high image quality, which helps meet the needs of multiple operating terminals (40 in total) simultaneously viewing real-time video feeds from 10 drones.

[0034] In this application, the first communication protocol and the second communication protocol are independent of each other, meaning that flight data and video data flow in their respective independent transmission channels without interfering with each other. This dual-channel separation architecture helps reduce the risk that video data, due to its large data volume, will continuously occupy bandwidth and thus cause congestion and interference to the transmission of critical flight control commands. To a certain extent, it is beneficial to achieve low latency and high reliability in the transmission of flight control commands in weak network environments.

[0035] In some embodiments, the edge computing node 20 is configured to detect the communication link quality parameters between the edge computing node 20 and the cloud 30 in real time; when the communication link quality parameters meet the delay condition, the flight control commands are processed in a hierarchical manner based on the priority of the flight control commands.

[0036] Communication link quality parameters are quantitative indicators used to characterize the communication link status between edge computing node 20 and cloud 30, including but not limited to communication link latency, packet loss rate, and link jitter. Specifically, communication link latency refers to the round-trip time for a data packet to travel from edge computing node 20 to cloud 30 and receive a response; packet loss rate refers to the proportion of data packets that fail to reach the target node within a unit of time; and link jitter refers to the fluctuation range of communication link latency over a period of time, reflecting the stability of link transmission.

[0037] The delay condition is a criterion used to indicate that the current link quality has significantly deteriorated. In one or more embodiments of this application, the delay condition includes one or more of the following: the communication link delay is greater than or equal to a preset delay threshold, the packet loss rate is greater than or equal to a preset packet loss rate threshold, or the link jitter value is greater than or equal to a preset jitter threshold. When the edge computing node 20 detects that the communication link quality parameters meet the delay condition, it determines that the current communication link between the edge computing node 20 and the cloud 30 is in a congested or high-latency state, and triggers the hierarchical processing flow of flight control commands.

[0038] The priority of flight control commands is a classification of processing levels based on the degree of impact of the commands on flight safety. Specifically, flight control commands can be divided into high priority and low priority. High priority flight control commands include, but are not limited to, emergency obstacle avoidance, hovering, and automatic return to home, which have a direct impact on flight safety; low priority flight control commands include, but are not limited to, adjustments to flight direction, speed, altitude, and gimbal attitude.

[0039] When the communication link quality parameters meet the delay condition, the edge computing node 20 performs hierarchical processing of flight control commands based on their priority. At this time, the edge computing node 20 differentiates the flight control commands to be forwarded based on their priority, in order to ensure the timely delivery of high-priority flight control commands as much as possible under limited link resources, thereby reducing the potential impact of link congestion on flight safety.

[0040] In some embodiments, flight control commands are processed in a hierarchical manner based on their priority, including: dividing flight control commands into regular flight control commands and emergency flight control commands, caching regular flight control commands locally, and sending emergency flight control commands to the UAV 10.

[0041] Routine flight control commands refer to the control commands generated by the operating terminal 40 during normal flight operations and used to adjust the flight status of the UAV 10, including but not limited to commands for adjusting flight direction, flight speed, flight altitude, and gimbal attitude. Routine flight control commands are relatively less sensitive to transmission latency and can be appropriately delayed under link congestion conditions. Emergency flight control commands refer to the control commands used during flight to deal with sudden abnormal situations and are directly related to flight safety, including but not limited to commands for emergency obstacle avoidance, hovering in place, and automatic return to home. Emergency flight control commands are highly sensitive to transmission latency and must be prioritized and forwarded when link resources are limited.

[0042] After the communication link quality parameters meet the delay condition and the edge computing node 20 triggers the hierarchical processing procedure, the edge computing node 20 identifies the type and prioritizes the flight control commands to be forwarded. For control commands identified as regular flight control commands, the edge computing node 20 temporarily stores them in a local cache queue. Once the communication link quality parameters return to the normal range and the delay condition is no longer met, the commands are read from the local cache queue and forwarded to the UAV 10. For control commands identified as emergency flight control commands, the edge computing node 20 skips the cache queuing process and forwards them directly to the UAV 10 with priority, in order to minimize the response delay of emergency flight control commands and reduce the risk of flight safety accidents caused by delayed command issuance.

[0043] This embodiment categorizes flight control commands into regular flight control commands and emergency flight control commands based on their impact on flight safety, and employs differentiated processing strategies under conditions of link congestion or high latency. This facilitates the rational allocation of limited bandwidth resources when link transmission resources are constrained. On one hand, emergency flight control commands are processed promptly through direct priority forwarding, which helps reduce the risk of delayed emergency command response in weak network environments. This allows the UAV 10 to respond more quickly to the operator's emergency control intentions when facing sudden abnormal situations, thus contributing to flight safety maintenance. On the other hand, regular flight control commands are processed using local caching, which helps prevent concurrent transmission of regular commands from further exacerbating link congestion and crowding out emergency command transmission resources. It also helps regular commands to be forwarded more orderly after the link is restored, balancing the safety and integrity of flight control and improving the overall stability and reliability of the UAV collaborative system 100 in weak network or network fluctuation environments.

[0044] In some embodiments, the cloud 30 is configured to: when receiving flight control commands from at least two operating terminals 40 within the same time window, to conduct conflict arbitration based on the role priority of each operating terminal 40, and to determine the flight control command to be executed; and to send the flight control command to be executed to the drone 10 via the edge computing node 20.

[0045] The same time window refers to a fixed time interval set by the cloud 30, used to uniformly collect and detect conflicts among multiple flight control commands arriving at the cloud 30 within this interval. Flight control commands arriving at the cloud 30 within the same time window are considered concurrent commands and must be processed through a conflict arbitration process to prevent conflicting flight control commands from being issued to the UAV 10 at the same time.

[0046] In multi-user collaborative control scenarios, each operating terminal 40 is assigned a unique role identifier by the cloud 30 upon accessing the system. The cloud 30 pre-establishes a mapping table between role identifiers and role priorities, with different role identifiers corresponding to different role priorities. Taking a two-person collaborative control scenario as an example, the roles of the operating terminals 40 can be divided into PIC terminal 41 and CO terminal 42, where the role identifier of PIC terminal 41 corresponds to a higher role priority, and the role identifier of CO terminal 42 corresponds to a lower role priority. When each operating terminal 40 sends flight control commands to the cloud 30, the command message carries the role identifier of the operating terminal 40, so that the arbitration engine module of the cloud 30 can perform identity recognition and priority lookup during conflict arbitration.

[0047] After receiving concurrent flight control commands, the arbitration engine module in the cloud 30 parses the role identifiers carried in each flight control command message. Based on the mapping table of the correspondence between role identifiers and role priorities, it queries the role priority corresponding to each command source operation terminal 40, and arbitrates the concurrent commands according to the preset role priority rules. The flight control command sent by the operation terminal 40 with the highest role priority is selected as the flight control command to be executed, and forwarded to the drone 10 for execution by the edge computing node 20. For flight control commands that are not selected in the arbitration, the cloud 30 discards or temporarily stores them and does not send them to prevent conflicting commands from interfering with the flight status of the drone 10.

[0048] Specifically, see Figure 2 At t=0ms, PIC terminal 41 sends a forward command carrying the PIC role identifier to cloud 30, and CO terminal 42 sends a left turn command carrying the CO role identifier to cloud 30 at t=2ms. Both commands fall within the same time window and are considered concurrent commands by cloud 30. The arbitration engine module in cloud 30 parses the role identifiers in the two command messages, queries the mapping table of role identifiers and role priorities, and confirms that the role priority corresponding to the PIC role identifier is higher than the role priority corresponding to the CO role identifier. Therefore, the forward command sent by PIC terminal 41 is determined as the flight control command to be executed and is sent to UAV 10 for execution via edge computing node 20. The left turn command sent by CO terminal 42 is not selected in this arbitration, and cloud 30 discards it and does not forward it, thus avoiding the risk of flight safety hazards caused by conflicting commands acting on UAV 10 simultaneously.

[0049] This embodiment introduces a role-priority-based command conflict arbitration mechanism in the cloud (30) to help resolve the problem of conflicting concurrent flight control commands in multi-user collaborative operation scenarios. In existing systems, when multiple operating terminals (40) send flight control commands simultaneously within the same time window, there is often a lack of effective priority determination and conflict handling methods, which can easily lead to the UAV (10) receiving contradictory control commands, resulting in abnormal flight attitude or even flight accidents. This embodiment pre-assigns a unique role identifier to each operating terminal (40) and establishes a mapping table between role identifiers and role priorities. When concurrent commands are detected, the arbitration engine module in the cloud (30) queries the corresponding priority based on the role identifier carried in each command message and automatically performs arbitration. This helps ensure that the UAV (10) responds to the highest-priority valid command in complex scenarios of multi-user concurrent operation, reducing the possibility of flight safety risks caused by command conflicts. In addition, the arbitration mechanism is uniformly executed in the cloud (30), which helps reduce the probability of control rights disputes between operating terminals (40) due to unclear role permissions, improves the overall orderliness and reliability of the multi-user collaborative operation system, and provides a certain degree of safety guarantee for complex flight missions involving multiple participants.

[0050] In some embodiments, the cloud 30 is also configured to broadcast conflict arbitration results to each operating terminal 40.

[0051] Specifically, the conflict arbitration result includes, but is not limited to, the role identifier of the source operating terminal 40 of the executed flight control command, the specific content of the executed flight control command, and the role identifier and processing status of the source operating terminal 40 corresponding to the unexecuted flight control command. After completing the arbitration determination, the cloud 30 simultaneously pushes the above conflict arbitration result to all operating terminals 40 currently connected to the system in the form of broadcast. After receiving the conflict arbitration result, each operating terminal 40 presents the conflict arbitration result to the corresponding operator through its corresponding user interface in the form of text prompts, status labels, or alarm notifications, so that the operator can intuitively understand the current handling status of the command conflict.

[0052] Specifically, see Figure 2After the cloud terminal 30 completes the arbitration decision and sends the forward command sent by the PIC terminal 41 to the drone 10, the cloud terminal 30 immediately broadcasts the arbitration result to all operating terminals 40 in the system. On one hand, the cloud terminal 30 sends a command execution confirmation message to the PIC terminal 41. This confirmation message includes the PIC role identifier and the content of the executed forward command. The user interface of the PIC terminal 41 can display a status prompt of "the forward command has been executed and sent for execution" based on the command execution confirmation message, so that the PIC can confirm that the forward command has been effectively responded to. At the same time, the cloud terminal 30 sends a command non-execution notification to the CO terminal 42. This notification includes the CO role identifier, the content of the non-executed left turn command, and the PIC role identifier information of the command that takes priority in this arbitration. The user interface of the CO terminal 42 can display an alarm notification of "the left turn command was not executed, and the command of the PIC terminal 41 was executed in this arbitration" based on the above notification, so that the CO can know in time that the command it sent was not executed in this arbitration, and avoid the CO from mistakenly believing that the command has been executed normally due to not receiving any feedback, and thus making incorrect subsequent control decisions.

[0053] This embodiment improves the transparency and information symmetry of the multi-user collaborative control system by broadcasting the arbitration results to all operating terminals 40 after the conflict arbitration is completed in the cloud 30. Without feedback on the arbitration results, operators of terminals 40 whose commands were not adopted often cannot promptly perceive that their flight control commands have been discarded during the arbitration process. This can easily lead operators to mistakenly believe that the commands have been correctly issued and executed, resulting in inappropriate subsequent control decisions and potentially increasing flight safety risks. This embodiment clearly marks the role of each operating terminal 40 and its corresponding command processing status in the arbitration results and simultaneously broadcasts the arbitration results to all operating terminals 40. This helps operators to perceive the current control authority and the specific handling of the command conflict in a timely and accurate manner, thereby reducing the possibility of misoperation due to information asymmetry. Furthermore, the broadcast feedback mechanism of the arbitration results helps establish a relatively clear perception of control authority among multiple operators, promoting collaboration and providing information assurance for the safe execution of flight missions in multi-user collaborative control scenarios.

[0054] In some embodiments, the cloud 30 is further configured to: upon receiving a control transfer request from the operating terminal 40 to which control is to be transferred, send a hovering command to the drone 10 via the edge computing node 20, and send a takeover confirmation request to the operating terminal 40 to which control is to be transferred. Upon receiving the takeover confirmation information from the operating terminal 40 to which control is to be transferred, the control transfer is completed.

[0055] The operating terminal 40 to be transferred refers to the operating terminal 40 that currently holds effective control of the UAV 10 and actively initiates the request for transfer of control, that is, the party currently responsible for implementing flight control of the UAV 10; the operating terminal 40 to be taken over control refers to the operating terminal 40 that has been designated as the target of control transfer, does not yet hold control but is about to take over control of the UAV 10, that is, the party that will formally assume the responsibility of flight control of the UAV 10 after the transfer of control is completed.

[0056] In its implementation, when the operating terminal 40 whose control is to be transferred needs to hand over control, it sends a control transfer request to the cloud 30. The request message contains the role identifier of the operating terminal 40 currently holding control (i.e., the role identifier of the source of the control transfer) and the role identifier of the target operating terminal 40 whose control is to be taken over (i.e., the role identifier of the target party of the control transfer). After receiving the control transfer request, the cloud 30 verifies the legality of the request content, queries the control ownership record maintained by the cloud 30 based on the source role identifier carried in the request message, confirms that the operating terminal 40 corresponding to the role identifier does indeed hold the current valid control, and confirms that the target operating terminal 40 whose control is to be taken over is in an online access state based on the target party role identifier. After the verification is passed, the control transfer process officially begins.

[0057] To prevent the UAV 10 from losing control due to the lack of effective control commands during the control transfer period, the cloud 30, after initiating the control transfer process, first performs a handshake. The cloud 30, through the edge computing node 20, issues a hover command to the UAV 10, ensuring the UAV 10 remains hovered during the control transfer and guaranteeing flight safety. Subsequently, a second handshake is performed. The cloud 30 sends a takeover confirmation request to the operating terminal 40, which is awaiting control. This request includes the role identifiers of the source and target parties of the control transfer, as well as the current flight status information of the UAV 10. This allows the operator of the operating terminal 40 to make a decision on whether to take over control based on a full understanding of the UAV 10's current status. After the operator of the operating terminal 40 manually confirms the takeover intention through the user interface, a third handshake is performed, and the operating terminal 40 returns a takeover confirmation message to the cloud 30. After receiving the confirmation of takeover information, cloud 30 completes the three-way handshake and updates the control ownership record in the system, changing the control ownership from the operating terminal 40 to the operating terminal 40 to take over the control.

[0058] For details, please refer to [link / reference]. Figure 2The PIC terminal 41 sends a control transfer request to the cloud 30, designating the CO terminal 42 as the operating terminal 40 to take over control. After the cloud 30 completes the legality verification, it sends a hovering command to the drone 10 through the edge computing node 20, and the drone 10 enters the hovering state. At the same time, the cloud 30 sends a takeover confirmation request to the CO terminal 42. After the CO confirms that it has accepted the control, the CO terminal 42 returns the takeover confirmation information to the cloud 30. The cloud 30 completes the control transfer.

[0059] This embodiment introduces a dynamic transfer of control process based on a request-confirmation mechanism in the cloud 30, which helps solve the problem that existing systems need to disconnect the current connection and re-establish a new connection when transferring control of the UAV 10. In existing systems, the latency introduced by connection disconnection and reconstruction during the transfer of control can easily lead to interruption or delay of flight control commands, causing the UAV 10 to be briefly out of control during the transfer of control, increasing flight safety risks. This embodiment, by uniformly recording the ownership of control in the cloud 30, achieves a smooth transfer of control by dynamically updating the ownership record without disconnecting any existing connection to the operating terminal 40. This helps reduce the risk of flight control command delay or loss caused by connection interruption during the transfer of control, and facilitates a smooth dynamic transfer of control with low latency. In addition, this embodiment prioritizes issuing hovering commands to the UAV 10 after the control transfer process is initiated, so that the UAV 10 maintains a relatively stable flight state during the control transfer transition, which helps reduce the risk of flight attitude loss due to the UAV 10 being in a command-unresponsive state during the transition. Meanwhile, this embodiment introduces a step of active confirmation by the target operation terminal 40, which helps to avoid the situation where the drone 10 is in an unmanned state for a long time due to the transfer of control before the target operator is in place. This is conducive to improving the standardization and reliability of the control handover process in multi-user collaborative operation scenarios.

[0060] In some embodiments, the cloud 30 is also configured to broadcast the control transfer result to each operating terminal 40 after the control transfer is completed.

[0061] In its implementation, the control transfer result includes, but is not limited to, the source party's role identifier, the target party's role identifier, the control transfer completion timestamp, and the role identifier of the current holder of the valid control after the transfer. After updating the control ownership record, the cloud 30 simultaneously broadcasts the control transfer result to all currently connected operating terminals 40. Upon receiving the control transfer result, each operating terminal 40 presents the result to the corresponding operator through its user interface using status labels, text prompts, or alarm notifications, enabling operators to intuitively and accurately perceive the current control ownership status.

[0062] Specifically, see Figure 2 After the control right is officially transferred from the PIC terminal 41 corresponding to the PIC role identifier to the CO terminal 42 corresponding to the CO role identifier in the cloud 30, the cloud 30 immediately broadcasts the result of this control right transfer to all operating terminals 40 in the system. The broadcast content includes information such as the role identifier of the source of control right (PIC), the role identifier of the target party (CO), and the role identifier of the current control right holder (CO). For PIC terminal 41, which has completed the transfer of control, its user interface displays a status message "Control has been successfully transferred to CO terminal 42" based on the source and target role identifiers in the broadcast content. This allows PIC operators to confirm that the transfer of control has been completed normally and to understand that they no longer hold valid control. For CO terminal 42, which has completed takeover, its user interface displays a status message "Control takeover successful, you currently hold control of drone 10" based on the current valid control holder role identifier in the broadcast content. This allows CO to clearly perceive that it now officially holds control of drone 10 and can effectively control drone 10. For other operating terminals 40 in the system that are under monitoring, their user interfaces are updated synchronously based on the broadcast content to display the current control ownership status, showing the current valid control holder role identifier (CO) in the status bar. This allows relevant operators to keep track of the latest control ownership status in real time and avoid misjudgments of the current control ownership status due to information lag.

[0063] This embodiment improves the transparency and consistency of information perception in multi-user collaborative control systems by broadcasting the control transfer result to all operating terminals 40 after the control transfer is completed in the cloud 30. Without feedback to each operating terminal 40, operators at each terminal 40 often struggle to perceive changes in control ownership in a timely and accurate manner. For the party that has completed the control transfer, without receiving clear feedback, they may mistakenly believe the transfer has not yet taken effect and continue to attempt to send flight control commands, resulting in invalid commands. For the party that has taken over, without receiving clear notification of successful takeover, they may delay timely control of the drone 10 due to uncertainty about whether they have officially acquired control. For other operating terminals 40 in the monitoring state, without synchronously obtaining the latest control ownership status, information lag may lead to misjudgments of the current control situation, thus affecting the accuracy of subsequent collaborative control decisions. This embodiment clearly marks the source party role identifier, target party role identifier, and current valid control rights holder role identifier in the control transfer result, and simultaneously pushes the above information to all operation terminals 40 in the form of broadcast. This helps each operator to accurately perceive the latest ownership status of control as soon as the control handover is completed. It helps to reduce the possibility of misoperation caused by information asymmetry in control ownership and provides a certain information guarantee for the safe execution of flight missions in multi-user collaborative control scenarios.

[0064] In some embodiments, the cloud 30 is further configured to: in response to user input, divide the search area into multiple search sub-areas; distribute the search tasks of each search sub-area to the corresponding operation terminal 40, so that the operation terminal 40 controls the corresponding drone 10 to perform the search task in the corresponding search sub-area; and display the search results on the collaborative map based on the target data reported by each drone 10.

[0065] Among them, input operations refer to the regional delineation and task configuration operations performed by users with the role of regional manager through the user interaction interface of the cloud 30, including but not limited to specifying the boundaries of the search area on the collaborative map displayed on the user interaction interface by means of touch, mouse selection or coordinate input, as well as setting the number of search sub-areas, sub-area priority, search route mode and other related parameters.

[0066] The search area refers to the target geographical area specified by the user on the collaborative map, which requires the drone 10 to perform the search task. The search sub-area refers to the geographical area of ​​each sub-block obtained by the cloud 30 after decomposing the search area according to a preset area division strategy, such as uniform division by grid, adaptive division by terrain features, or division by user-defined boundaries. Each search sub-area corresponds to the search responsibility area of ​​one drone 10.

[0067] Target data refers to mission-related data collected and reported to the cloud 30 by the UAV 10 during the search mission, including image data collected by airborne sensors, target identification results (including GPS coordinates and screenshot images of suspected targets), and the UAV 10's own position information.

[0068] The collaborative map is a shared map within the UAV collaborative system 100, uniformly maintained by the cloud 30 and synchronously pushed to all operating terminals 40 in real time. Based on satellite remote sensing imagery or surveying base maps, it overlays various dynamic business information in a structured layer format, including: the real-time location and flight trajectory of each UAV 10, the geographical boundaries of the search sub-areas managed by each UAV 10 and the current search coverage progress, and suspected target alarm annotations generated by the cloud 30 after the UAV 10 reports target data (such as target coordinates marked with a red asterisk and a screenshot thumbnail). To ensure consistent display of the collaborative map across the command center terminal, PIC terminal 41, and CO terminal 42, the collaborative map data status is centrally maintained by the cloud 30. When the map status changes (such as UAV 10 location updates, target annotation additions, or changes in search progress), the cloud 30 pushes the changes to all online terminals in the form of incremental update packages. Each terminal processes the changes locally and then synchronizes with the cloud 30. In addition, the cloud terminal 30 can push differentiated views based on the role permissions of each terminal. For example, the command center terminal displays the real-time status of all drones 10 and alarm markings for suspected targets, and triggers pop-up notifications when new alarms are added. The PIC terminal 41 displays information about the controlled drones 10, while the other drones 10 are presented in an auxiliary layer. The CO terminal 42 adopts a global view similar to that of the command center terminal, and integrates multiple drone 10 video transmission monitoring windows, supporting CO to perform task marking on the collaborative map. The marking results are reflected to all terminals in real time after being synchronized by the cloud terminal 30.

[0069] In practice, after receiving user input, the cloud-based 30 divides the search area into multiple sub-areas based on a preset regional division strategy. Combining the number and spatial distribution of currently online drones 10, it matches each sub-area with its corresponding operating terminal 40 and the drones 10 controlled by that terminal. This ensures that each sub-area has a corresponding drone 10 responsible for performing the search task, thus achieving parallel coverage of the search area by multiple drones 10. For example, in a search and rescue scenario, the area manager can divide the overall search area into multiple sub-areas on the collaborative map displayed on the cloud-based 30's user interface and dynamically assign search tasks for each sub-area to different drones 10 and their corresponding operating terminals 40. When the cloud-based 30 sends the search tasks for each sub-area to the corresponding operating terminal 40, the task message includes the geographic boundary coordinates of the sub-area, suggested flight paths (such as a bow-shaped coverage scan path), search altitude, and flight speed, enabling the operating terminal 40 to plan and control the drones 10 to perform coverage searches within the designated sub-area along predetermined routes. During the search mission, the UAV 10 reports target data to the cloud 30 in real time. After the cloud 30 aggregates and processes the reported target data, it updates the suspected targets to the collaborative map in the form of alarm labels, which can be viewed by all operating terminals 40 in real time, and pushes real-time alarm notifications to the command center terminal.

[0070] Specifically, the operating terminal 40 includes a command center terminal, a PIC terminal 41, and a CO terminal 42. (See also...) Figure 3The regional manager divides the overall search area into four sub-areas (a, b, c, and d) on the collaborative map S1 using the command center terminal. Search tasks for each sub-area are then assigned to the corresponding operation terminals 40, completing the allocation of drones 11 to 14 to their respective sub-areas. During task execution, the command center terminal can view the real-time location information and transmitted video of all drones and receive target detection alarms from the cloud terminal 30. On-site drone pilots directly control their respective drones to perform search tasks via their PIC terminals 41. For example, the PIC terminal corresponding to search sub-area a controls drone 11 to perform search tasks within sub-area a and transmits video data collected by drone 11 back to the cloud terminal 30 in real-time. Remote experts can simultaneously view the transmitted video from the four drones on the collaborative map displayed on the CO terminal 42 and can request control permissions for specific drones from the cloud terminal 30 when necessary to provide remote intervention and guidance for on-site execution. Under the control of their respective operating terminals, UAVs 11 to 14 perform a zigzag-shaped sweep scan in four search sub-areas (a, b, c, and d). During the search, each UAV reports the collected target data to the cloud (30) in real time. The cloud (30) aggregates and processes the data, and displays suspected targets on the collaborative map in the form of alarm markers (such as red asterisks at the location of suspected targets). For example, when UAV 11 discovers suspected target 1 in search sub-area a, the cloud (30) synchronizes the location marker of suspected target 1 to the collaborative map in real time. The command center terminal then receives an alarm notification that "suspected target 1 has been discovered." Users can perceive the suspected target through the command center terminal and dispatch rescue forces to the corresponding location accordingly.

[0071] This embodiment introduces a multi-UAV collaborative search task distribution and collaborative map sharing mechanism in the cloud 30, which helps solve the problems of low search efficiency and limited coverage in the traditional single-UAV search mode. By dividing the search area into multiple search sub-regions and scheduling corresponding UAVs 10 to execute search tasks in parallel, the search coverage efficiency of a large area can be significantly improved. At the same time, the introduction of the collaborative map enables each operating terminal 40 to share search progress and target discovery information in real time, which helps to avoid multiple UAVs 10 repeatedly searching the same search sub-region or missing key areas, and is conducive to improving the task execution efficiency and real-time information sharing in multi-UAV collaborative search scenarios.

[0072] In some embodiments, the cloud 30 is further configured to: receive video image annotation data sent by the operation terminal 40, the video image annotation data including the pixel coordinates of the annotated area in the video image returned by the user from the drone 10 and the corresponding video frame information; receive pose data of the drone 10, the pose data including the geographical location, flight attitude and flight altitude of the drone 10; calculate the three-dimensional geographic coordinates corresponding to the annotated area based on the video image annotation data and the pose data; and generate abnormal annotation anchor points on the collaborative map based on the video image annotation data and the three-dimensional geographic coordinates.

[0073] Video annotation data refers to the data generated when a user performs annotation operations on the video screen through the user interface of the terminal 40 while watching the real-time video stream transmitted by the drone 10. During the drone 10's mission, the operator can use touch or a mouse to select suspected abnormal areas on the video screen. This operation will generate video annotation data locally and upload it to the cloud 30. The video annotation data specifically includes the following information: the pixel coordinates of the annotated area in the video screen coordinate system (usually represented by the pixel coordinates of the upper left and lower right corners of a rectangular bounding box, or by the coordinates of the center point plus width and height), the annotation shape (such as a rectangle, point annotation, or free selection area), the video frame timestamp corresponding to the annotation time, and the annotation description information filled in by the user (such as defect type, anomaly level, etc.).

[0074] Pose data refers to the spatial state data collected in real time by the onboard sensors of the UAV 10 and reported to the cloud 30 during the execution of its flight mission. It is used to describe the precise spatial state of the UAV 10 and its onboard camera at the marked time. The pose data specifically includes the following information: the geographical location (latitude and longitude coordinates) of the UAV 10, its flight altitude (relative to ground altitude or elevation), its flight attitude (yaw angle, pitch angle, and roll angle), and the gimbal attitude angle of the onboard camera, etc.

[0075] The calculation process of three-dimensional geographic coordinates is essentially the process of back-projecting the pixel position in the image coordinate system to the geographic coordinate system. First, the cloud 30 extracts the pose information that strictly corresponds to the time from the pose data stream continuously reported by the UAV 10 based on the timestamp of the video frame corresponding to the labeled time. Then, the cloud 30 combines the pre-constructed coordinate transformation relationship between the video image pixel coordinate system and the three-dimensional geographic coordinate system to convert the pixel coordinates into three-dimensional geographic coordinates. The specific coordinate transformation relationship can be established according to existing technology, which will not be elaborated here.

[0076] After completing the 3D geographic coordinate calculation, the cloud-based 30 integrates the video image annotation data and geographic coordinates to generate anomaly annotation anchor points on the collaborative map. The anchor points use the calculated 3D geographic coordinates as their map display location, are marked with icons (such as red asterisks or highlighted marks) at the corresponding map positions, and include information such as the pose data of the UAV 10 at the time of annotation. After the anchor points are generated, the cloud-based 30 synchronizes them in real time to all online terminals, enabling the collaborative maps of the command center terminal, PIC terminal 41, and CO terminal 42 to display the anomaly annotation anchor points in real time, achieving precise spatial positioning of the anomaly location and multi-terminal sharing.

[0077] This embodiment deeply integrates the user's two-dimensional annotation operations on the video screen with the real-time pose data of the UAV 10, which helps to solve the problem that in traditional inspection scenarios, abnormal locations can only rely on text descriptions or manual records, making accurate positioning difficult. By automatically back-projecting the annotated coordinates on the video screen into three-dimensional geographic coordinates and generating collaborative map anchor points, it is beneficial to achieve accurate spatial calibration of inspection abnormal locations and real-time sharing across multiple terminals, which helps to improve the collaborative efficiency and anomaly positioning accuracy between remote experts and on-site operators in industrial inspection scenarios.

[0078] In some embodiments, the cloud 30 is also configured to use a conflict-free replicated data type (CRDT) algorithm to merge and process the collaborative map data submitted concurrently by each operation terminal 40, so that the collaborative map displayed by each operation terminal 40 and the collaborative map displayed by the cloud 30 are consistent.

[0079] Conflict-Free Replicated Data Type (CRDT) is a data structure specifically designed for distributed systems. Its core characteristic is that, even when multiple nodes concurrently execute data update operations, and network latency or brief offline periods exist between nodes, the system can automatically resolve conflicts between concurrent operations through mathematically provable merging rules without relying on centralized conflict arbitration mechanisms or distributed locks. This ensures that all nodes eventually converge to a consistent data state after network connectivity is restored, achieving eventual consistency of distributed data. In this embodiment, the CRDT algorithm ensures that concurrent update operations performed by multiple operating terminals 40 on collaborative map data (including grid allocation status of search sub-regions, task allocation information, suspected target annotation anchor points, inspection anomaly anchor points, and other business data) under inconsistent network latency conditions can be correctly merged and processed by the cloud 30, thereby ensuring that the collaborative map state displayed by all operating terminals 40 remains eventually consistent with the collaborative map state maintained by the cloud 30.

[0080] When each operating terminal 40 performs a collaborative map data update operation, it submits the update data, carrying a logical timestamp and terminal identifier, to the cloud 30 in real time. The cloud 30 merges the concurrently submitted update data according to the CRDT merging rules and synchronizes the merged latest collaborative map status to all online operating terminals 40 via incremental push, ensuring that the collaborative map status displayed by the command center terminal, PIC terminal 41, and CO terminal 42 is consistent with that of the cloud 30. In the event of out-of-order arrival of update data due to network latency, the cloud 30, based on the commutative and associative properties of the CRDT algorithm, ensures that the final merging result remains consistent regardless of the order in which the update data arrives, thus eliminating the impact of network out-of-order delivery on data consistency at the structural level.

[0081] Taking a large-scale wind farm cross-regional collaborative inspection scenario as an example, the specific operation process of this embodiment is explained. The on-site pilot (PIC) arrives at the wind farm operation site with a drone 10 and an edge computing node 20. The drone 10 connects to the edge computing node 20 via a local area network, and the edge computing node 20 connects to the cloud-based collaborative platform 30 via a 5G network. A remote senior engineer (CO) remotely logs into the cloud-based platform 30 via a PC browser to complete role access. The cloud-based platform 30 establishes dual channels for flight data and video data for this task. The flight data control channel uses the MQTT protocol, and the on-site pilot's control commands are transmitted to the drone 10 via the edge computing node 20. The video data channel uses the WebRTC protocol; the onboard gimbal camera footage of the drone 10 is encoded by the edge computing node 20 and then distributed in real-time by the cloud-based platform's SFU server to both the on-site pilot's terminal and the remote engineer's terminal. The end-to-end video transmission latency is less than 200ms. During an inspection of a wind turbine blade, a remote engineer (CO) discovered a suspected crack in the video feed and initiated a request to take over gimbal control from the system. The on-site pilot (PIC) received a pop-up notification and confirmed the request, completing a three-way handshake process for the transfer of control. At this point, the drone (UAV 10) hovered at its current GPS location, and gimbal control was transferred to the remote engineer's terminal. The remote engineer then zoomed in on the video feed, selecting the crack area in the browser's video stream. The system recorded the screen's 2D coordinates of the marked area and, combined with the drone's current latitude, longitude, altitude, gimbal pitch, and yaw angles, calculated the actual 3D geographic coordinates of the crack target. A corresponding "serious defect" anomaly marker was then generated on the cloud-based collaborative map (30). After the marker was completed, the remote engineer released gimbal control, and the on-site pilot regained full control, guiding the drone (UAV 10) to the next inspection target. The defect markers generated during this inspection were synchronized in real-time to all participating terminals (40) on the collaborative map using the CRDT algorithm, ensuring consistency in map information across all terminals.

[0082] This embodiment introduces a multi-terminal collaborative environment based on the CRDT algorithm in the cloud 30. Figure 1 The consistent synchronization mechanism deeply integrates business data streams such as search and rescue grid division and inspection AR annotation with the real-time spatial status of the UAV 10. This helps solve the problem of data conflicts and state inconsistencies that easily occur when multiple operating terminals 40 concurrently update collaborative map data under conditions of inconsistent network latency. Compared with traditional data synchronization schemes based on distributed locks or centralized conflict arbitration, the CRDT algorithm does not require blocking concurrent write operations, which helps reduce the risk of operation blocking caused by data synchronization conflicts in multi-terminal collaborative scenarios. By attaching a logical timestamp and terminal identifier to each operation, the cloud 30 can automatically merge the concurrent updates of each terminal into a globally consistent collaborative map state without centralized locking. This is conducive to achieving high concurrency, low latency, and eventual consistency synchronization of collaborative map data in multi-user collaborative operation scenarios, thereby improving the real-time data sharing, team collaboration efficiency, and overall system reliability when multiple operating terminals 40 are working together.

[0083] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for at least one computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A drone collaborative system, characterized in that, include: Drones, edge computing nodes, cloud computing, and at least two operating terminals; The edge computing nodes are respectively connected to the drone and the cloud, and the cloud is also connected to each of the operating terminals. The drone and the edge computing node transmit flight data using a first communication protocol. The edge computing node and the cloud transmit the flight data using the first communication protocol. The cloud and the operating terminal transmit the flight data using the first communication protocol. The flight data includes flight control commands, telemetry data, and heartbeat packets. The drone transmits video data to the edge computing node via a second communication protocol, the edge computing node transmits the video data to the cloud via the second communication protocol, and the cloud transmits the video data to each of the operating terminals via the second communication protocol. The first communication protocol and the second communication protocol are independent of each other.

2. The drone collaborative system according to claim 1, characterized in that, The edge computing node is configured as follows: Real-time detection of communication link quality parameters between the edge computing node and the cloud; When the communication link quality parameters meet the delay condition, the flight control commands are processed in a tiered manner based on their priority.

3. The drone collaborative system according to claim 2, characterized in that, The step of processing the flight control commands in a tiered manner based on their priority includes: The flight control commands are divided into regular flight control commands and emergency flight control commands. The regular flight control commands are cached locally, and the emergency flight control commands are sent to the UAV.

4. The drone collaborative system according to claim 1, characterized in that, The cloud is configured as follows: When at least two flight control commands are received from the operation terminals within the same time window, conflict arbitration is performed based on the role priority of each operation terminal to determine the flight control command to be executed. The flight control commands to be executed are sent to the UAV via the edge computing node.

5. The drone collaborative system according to claim 4, characterized in that, The cloud is also configured as follows: The conflict arbitration result is broadcast to each of the aforementioned operating terminals.

6. The drone collaborative system according to claim 1, characterized in that, The cloud is also configured as follows: Upon receiving a control transfer request from the operating terminal to which control is to be transferred, the edge computing node sends a hovering command to the drone and a takeover confirmation request to the operating terminal to which control is to be taken over. The system receives confirmation of takeover from the terminal that is about to take over control, thus completing the transfer of control.

7. The drone collaborative system according to claim 6, characterized in that, The cloud is also configured as follows: After the transfer of control is completed, the result of the transfer of control is broadcast to each of the aforementioned operating terminals.

8. The drone collaborative system according to claim 1, characterized in that, The cloud is also configured as follows: In response to user input, the search area is divided into multiple sub-search areas; The search tasks for each of the search sub-regions are respectively sent to the corresponding operation terminals, so that the operation terminals control the corresponding drones to perform search tasks in the corresponding search sub-regions; Based on the target data reported by each of the aforementioned drones, the search results are displayed on the collaborative map.

9. The drone collaborative system according to claim 1, characterized in that, The cloud is also configured as follows: The system receives video frame annotation data sent by the operation terminal. The video frame annotation data includes the pixel coordinates of the annotation area in the video frame returned by the drone and the corresponding video frame information. Receive the pose data of the UAV, the pose data including the UAV's geographical location, flight attitude and flight altitude; Calculate the three-dimensional geographic coordinates corresponding to the labeled area based on the video image annotation data and the pose data; Based on the video image annotation data and the three-dimensional geographic coordinates, anomaly annotation anchor points are generated on the collaborative map.

10. The drone collaborative system according to claim 1, characterized in that, The cloud is also configured as follows: A conflict-free copy data type algorithm is adopted to merge and process the collaborative map data submitted concurrently by each of the operation terminals, so as to ensure that the collaborative map displayed by each operation terminal and the collaborative map displayed in the cloud are consistent.

Images