A collaborative inspection and control method and system for unmanned aerial vehicles based on digital twins

Abstract

This application relates to the field of digital twin technology and discloses a method and system for collaborative inspection control of unmanned aerial vehicles (UAVs) based on digital twins. The method includes acquiring inspection data with first timestamps uploaded by multiple UAVs, using the time of receiving the inspection data as a second timestamp, and storing the inspection data in the order of the second timestamps to form a state record sequence. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is greater than a preset threshold, starting from a historical moment, the state changes of the digital twin virtual model from the historical moment to the current valid time are re-deduced according to the order of the first timestamps of each inspection data in the state record sequence. The result of the re-deduction is used as the latest current state of the digital twin virtual model. Subsequent inspection tasks are reallocated based on the latest current state. By reverting the virtual model to the correct historical moment and re-deducing, a reliable data foundation is provided for subsequent collaborative control.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, and more specifically, to a collaborative inspection and control method and system for unmanned aerial vehicles based on digital twins. Background Technology

[0002] In the inspection of large-scale facilities such as water conservancy dams, the use of multi-drone collaborative operations has become an important means to improve efficiency and ensure safety. Its core relies on a digital twin system, which aggregates inspection data uploaded by each drone to construct and update a virtual mirror of the physical world in real time. The command center uses this virtual model for task scheduling and decision-making. However, in actual operation sites, especially in complex environments like dams with terrain obstructions and electromagnetic interference, the data communication link between the drones and the ground control terminal is extremely unstable, often resulting in significant delays or even out-of-order arrival of data packets.

[0003] Existing technologies typically update digital twin models in the order data arrives, failing to adequately consider the difference between data generation and reception times. When delayed or out-of-order data is processed directly, the state presented by the digital twin model deviates from the reality of the physical world. For example, information such as the drone's position and completed task areas in the virtual model may be incorrect. Reassigning drone tasks or adjusting flight routes based on such a distorted model can lead to scheduling chaos, such as duplicate inspections, missed tasks, and even drone collisions in the air. This severely impacts inspection efficiency and safety, significantly reducing the reliability of the digital twin system in complex communication environments.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a UAV collaborative inspection control method and system based on digital twins. This method and system can solve the technical problem in existing technologies where the inconsistency between the digital twin model and the physical world state is caused by the delay and out-of-order arrival of UAV inspection data, which in turn affects the accuracy of subsequent collaborative control decisions.

[0006] Firstly, this application provides a collaborative inspection and control method for unmanned aerial vehicles (UAVs) based on digital twins, comprising: Acquire inspection data with first timestamps uploaded by multiple drones, record the time when the inspection data is received as the second timestamp, and store the inspection data in the order of the second timestamps to form a status record sequence. The first timestamp of the newly received inspection data is compared with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is greater than a preset threshold, the inspection data is determined to be delayed data, and the state of the digital twin virtual model is traced back to the historical moment corresponding to the first timestamp. Starting from a historical moment, according to the order of the first timestamp of each inspection data in the state record sequence, the state changes of the digital twin virtual model from the historical moment to the current effective time are re-deduced, and the result of the re-deduction is taken as the latest current state of the digital twin virtual model; Based on the latest current status, subsequent inspection tasks will be reassigned, drone flight routes will be adjusted, and / or shooting strategies will be adjusted.

[0007] This technical solution establishes a backtracking and replay mechanism for processing delayed data. When delayed data sufficient to affect model consistency is detected, the virtual model's state can be reverted to the correct historical moment when the data was generated, and the model can be re-enacted to the current moment based on the actual time sequence. This fundamentally ensures the spatiotemporal consistency of the digital twin model and provides a reliable data foundation for subsequent collaborative control.

[0008] Furthermore, the inspection data includes sensor-collected data, drone location information, drone flight trajectory, and first timestamp.

[0009] This technical solution clarifies the specific content that constitutes the inspection data, giving the status record and model update a clear data object. This provides the necessary data input for subsequent location calculation, status assessment, and trajectory extrapolation, and enhances the operability of the method.

[0010] Furthermore, the digital twin virtual model includes a three-dimensional geometric model of the drone, a three-dimensional geometric model of the object being inspected, a status list of the drone, and a task schedule. The current valid time is the latest time point recorded in the digital twin virtual model that reflects the state of the physical world.

[0011] This technical solution specifically defines the constituent elements of a digital twin virtual model, concretizes the concept of a virtual model, and clearly defines the meaning of the current effective time, providing a clear reference system and operational object for state backtracking and time comparison.

[0012] Furthermore, prior to the step of acquiring inspection data with first timestamps uploaded by multiple drones, the following steps are also included: Data verification and transmission are performed when the drone is in an area with good signal. Multiple sets of first timestamps and corresponding second timestamps are collected, and a time mapping function between the first timestamp and the difference between the first timestamp and the second timestamp is constructed. The steps to obtain inspection data with first-timestamps uploaded by multiple drones include: The first timestamp of the inspection data is corrected using a time mapping function to obtain the corrected first timestamp.

[0013] By learning a network latency model when communication is good, the timestamps of subsequently received data are pre-calibrated, which effectively reduces the time error introduced by network fluctuations, improves the accuracy of the first timestamp, thereby reducing unnecessary state backtracking and improving system operating efficiency.

[0014] Furthermore, the step of comparing the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model includes: If the first timestamp is not earlier than the current valid time, and the time difference between the first timestamp and the current valid time is less than or equal to a preset threshold, and the inspection data conforms to the expected behavior logic of the drone, then the state of the digital twin virtual model is updated using the inspection data, and the current valid time of the digital twin virtual model is updated to the first timestamp.

[0015] This technical solution defines a standard process for processing normal, near-real-time data, which complements the abnormal process for processing delayed data, forming a complete model update logic. Simultaneously, it adds a conformity check for expected behavior logic, filtering out some abnormal data and enhancing the robustness of model updates.

[0016] Furthermore, after the step of determining that the inspection data is delayed data, and before the step of backtracking the state of the digital twin virtual model to the historical moment corresponding to the first timestamp, the method also includes: Store the delayed data in the delayed processing buffer and set the aggregation time window based on the first timestamp of the delayed data; Within the aggregation time window, if subsequent delayed data with the first timestamp falling within the coverage area of ​​the aggregation time window is received, the subsequent delayed data will be stored together in the delayed processing buffer. When the aggregation time window is closed, all delayed data is extracted from the delayed pending buffer, and the target delayed data with the earliest first timestamp is identified.

[0017] This technical solution introduces a strategy of buffering and aggregating delayed data, avoiding the need to immediately trigger a costly backtracking operation for every single piece of sporadic delayed data. By batch processing delayed data within a time window, it significantly reduces system computational overhead and improves overall efficiency in handling bursty and continuous network latency.

[0018] Furthermore, the steps to trace the state of the digital twin virtual model back to the historical moment corresponding to the first timestamp include: Using the first timestamp of the target delayed data as the backtracking anchor point, the digital twin virtual model is backtracked to the corresponding historical moment, and all delayed data in the delayed processing buffer is integrated into the digital twin virtual model in chronological order.

[0019] By selecting the earliest timestamp as the sole backtracking anchor, only one backtracking action is needed to integrate all delayed data in the buffer into the model evolution path in the correct time sequence, making the state correction process more efficient and concise.

[0020] Furthermore, the steps for re-analyzing the state changes of the digital twin virtual model from a historical moment to the current effective time include: Real-time capture of environmental change information in the inspection environment, and assign a base timestamp based on global standard time to each environmental change information to form a global event log; Using the first timestamp as an index, search the global event log for overlapping time anchors; If overlapping time anchors are detected, and there is a logical conflict between the inspection data and the environmental change information corresponding to the overlapping time anchors, the inspection data will be labeled with attributes according to the priority of the environmental change information, and a consistency warning will be issued. The physical constraints generated by environmental change information are used as boundary conditions to correct the flight trajectory of the UAV, and the state changes of the digital twin virtual model are re-inferred based on the corrected inspection data.

[0021] This technical solution incorporates consideration of dynamic changes in the external environment during state reenactment, making the reenactment process no longer closed but synchronized with real-world emergencies (such as the addition of new no-fly zones). By detecting and resolving logical conflicts between drone data and environmental events, it ensures that the reenacted model state is not only temporally correct but also physically more reasonable and reliable.

[0022] Furthermore, the steps of using the physical constraints generated by environmental change information as boundary conditions to correct the UAV's flight trajectory include: Access distant views from nearby drones or monitoring images from ground cameras in the inspection environment at the time corresponding to the first timestamp. Based on distant views or monitoring images from ground cameras, the actual flight trajectory in inspection data with logical conflicts is recalculated so that the corrected actual flight trajectory conforms to the physical constraints generated by environmental change information.

[0023] By utilizing data sources from other perspectives, such as neighboring drones or ground monitoring, as auxiliary information, conflicting drone trajectory data can be objectively corrected. This allows the model state recovery to no longer rely solely on a single data source, but rather on multi-source data fusion, greatly improving the accuracy and reliability of the correction results.

[0024] Secondly, this application also discloses a digital twin-based UAV collaborative inspection and control system for executing any of the aforementioned digital twin-based UAV collaborative inspection and control methods. The system includes: The data acquisition and storage module is used to acquire inspection data with a first timestamp uploaded by multiple drones, record the time of receiving the inspection data as a second timestamp, and store the inspection data in the order of the second timestamps to form a status record sequence. The time comparison and backtracking module is used to compare the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is not within the preset range, the inspection data is determined to be delayed data, and the state of the digital twin virtual model is backtracked to the historical moment corresponding to the first timestamp. The state deduction module is used to re-deduce the state changes of the digital twin virtual model from the historical moment to the current effective time, according to the order of the first timestamp of each inspection data in the state record sequence, starting from the historical moment, and use the re-deduction result as the latest current state of the digital twin virtual model; The inspection control module is used to perform subsequent inspection task reallocation, drone flight route adjustment, and / or shooting strategy adjustment based on the latest current status.

[0025] The technical solution provided in this application has significant advantages over existing technologies. The core of this application lies in its proposed digital twin model state maintenance mechanism, which proactively addresses data latency and out-of-order delivery. In complex scenarios with unstable communication, such as drone-based collaborative inspections, existing technologies often lead to virtual model distortion due to the direct adoption of delayed data, resulting in erroneous control decisions. This application, by introducing core logic of timestamp comparison, state backtracking, and time-series replay, can intelligently identify outdated data and backtrack the state of the digital twin model to the actual historical moment when the data was generated. Then, it re-simulates this late data and all subsequent data arranged in chronological order, ultimately obtaining a fully synchronized and up-to-date virtual model state with the physical world. This process ensures that regardless of when or in what order the data arrives, the digital twin model can ultimately be corrected to the correct state. Therefore, the accuracy and rationality of decisions made based on this highly consistent and reliable model, such as the reallocation of UAV tasks and adjustment of flight routes, are fundamentally guaranteed. This effectively avoids problems such as low inspection efficiency, task errors and omissions and safety risks caused by model inconsistency, and significantly improves the robustness and practicality of the UAV collaborative inspection system in complex environments. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a collaborative inspection and control method for unmanned aerial vehicles (UAVs) based on digital twins, provided as an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of a collaborative inspection and control system for unmanned aerial vehicles based on digital twins, provided as an embodiment of this application.

[0028] Labeling Explanation: 210, Data Acquisition and Storage Module; 220, Time Comparison and Backtracking Module; 230, Status Inference Module; 240, Inspection Control Module. Detailed Implementation

[0029] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] In large-scale infrastructure projects, such as water conservancy projects, the use of drone swarms for collaborative inspections has become a key technology for ensuring structural safety and improving operational efficiency. However, in actual operating environments, especially in areas with complex three-dimensional structures, numerous metal components, and undulating terrain, such as large concrete gravity dams, the wireless communication link between drones and ground control stations is often obstructed and interfered with, leading to unpredictable delays and out-of-order data transmission. When a digital twin system constructs a virtual world based on this data containing timing errors, the presented scene will be disconnected from physical reality. Subsequent collaborative control strategies based on this distorted model will not only reduce inspection efficiency but may even trigger air traffic conflicts, posing serious safety hazards.

[0032] To solve this problem, firstly, see [reference needed] Figure 1 This application provides a collaborative inspection and control method for unmanned aerial vehicles (UAVs) based on digital twins, the method comprising: S1. Obtain inspection data with first timestamps uploaded by multiple drones, and record the time when the inspection data is received as the second timestamp. Store the inspection data in the order of the second timestamps to form a status record sequence. S2. Compare the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is greater than the preset threshold, then the inspection data is determined to be delayed data, and the state of the digital twin virtual model is traced back to the historical moment corresponding to the first timestamp. S3. Starting from the historical moment, according to the order of the first timestamp of each inspection data in the state record sequence, re-determine the state changes of the digital twin virtual model from the historical moment to the current effective time, and take the result of the re-deduction as the latest current state of the digital twin virtual model; S4. Based on the latest current status, perform subsequent inspection task reassignment, drone flight route adjustment, and / or shooting strategy adjustment.

[0033] Among them, the digital twin virtual model is a virtual copy that is dynamically constructed in a computer system and highly synchronized with the inspection scene in the physical world (including drones, the dam being inspected, etc.). It is a comprehensive dynamic simulation environment that integrates geometric information, physical attributes, behavioral rules and real-time status data.

[0034] The first timestamp refers to the precise time at which inspection data is collected or generated on the drone. This time is typically provided by a high-precision clock module onboard the drone, synchronized with signals from a global navigation satellite system (such as BeiDou or GPS), recording the absolute time of the event in the physical world.

[0035] The second timestamp refers to the moment when the ground control station's server successfully receives a data packet from the drone. This time is recorded by the ground control station's system clock.

[0036] The difference between the first and second timestamps directly reflects the total delay that data experiences from generation to reception. This delay includes the time consumed by multiple stages such as data packaging, wireless channel transmission, and network routing.

[0037] The status log sequence is a raw data log arranged in the order in which data arrives at the ground control station (i.e., according to the second timestamp). Due to network latency and out-of-order delivery, the order of data in this sequence may not be consistent with its actual time sequence in the physical world (i.e., according to the first timestamp).

[0038] The current valid time is a key time marker within a digital twin virtual model, representing the latest point in time in the physical world that the virtual world is synchronized with. In other words, the state of all objects in the model is considered accurate and valid up to this point in time.

[0039] Delayed data refers to newly received data whose first timestamp is significantly earlier than the current valid time of the digital twin virtual model. This data is considered delayed. In other words, information belonging to the past is received only after time has already advanced in the virtual world.

[0040] In one specific embodiment, suppose there are two drones, Drone A and Drone B, conducting a collaborative inspection of a large hydroelectric power station's concrete dam. Drone A is responsible for inspecting the guardrails and gate hoisting equipment along the dam crest road, and its communication signal is good. Drone B is responsible for descending to the middle section of the dam to take close-up photos of the water discharge from a series of drainage holes. Due to its proximity to the massive concrete dam, its communication signal is blocked and becomes intermittent.

[0041] First, the data acquisition and storage module of the ground control station continuously receives data packets from UAV A and UAV B. For example, at 10:05:00 AM, the control station receives a photo of the dam top taken by UAV A at 10:04:59 AM. The control station records these two timestamps and stores the data packet in the status record sequence. Then, at 10:05:01 AM, it receives another photo taken by UAV A at 10:05:00 AM, which is also recorded and stored. At this point, the current valid time in the digital twin virtual model steadily advances with these timely arriving data packets, for example, updating to 10:05:00 AM.

[0042] Suddenly, at 10:05:30 AM, the ground control station received a data packet from UAV B. UAV B had just flown out of the signal dead zone in the middle of the dam, and the signal had been restored, so it uploaded the previously backlogged data all at once. The time comparison and backtracking module analyzed the data packet and found that the inspection data it contained, such as a close-up photo of a drainage hole, had a first timestamp of 10:04:15 AM.

[0043] At this point, the system performs a time comparison. The first timestamp of the new data (10:04:15) is much earlier than the current valid time of the digital twin virtual model (10:05:00). The system calculates the time difference between the two, which is 75 seconds. This time difference is much greater than a preset threshold, such as 5 seconds. This threshold is set to tolerate normal, minor network jitter and avoid triggering subsequent complex processes too frequently. Since 75 seconds is greater than 5 seconds, the system determines that the data packet is delayed data.

[0044] Once the data is determined to be delayed, the system will initiate a state rollback mechanism. This means that the digital twin virtual model cannot simply insert a past photo into the present scene, as this would cause a spatiotemporal distortion. The correct approach is for the system to command the virtual model to rewind time, rolling back its overall state to the point in time when the delayed data was generated, namely 10:04:15. This rollback can be achieved by the system periodically or at key event points during operation, saving snapshots of the virtual model's state in a historical database. At this point, the system will load the state snapshot closest to 10:04:15 and no later than that moment from the database, and revert all state changes that occurred after that snapshot. The model positions, attitudes, and completed task lists of drones A and B in the virtual world will all be restored to their state at 10:04:15.

[0045] Next, the system enters the state replay phase. Starting from the historical time of 10:04:15, it begins replaying the history. It retrieves all data in the state record sequence whose first timestamp is after 10:04:15, including the recently received delayed data from Drone B and the previously processed on-time data from Drone A. The system reorders this data strictly according to the order of its first timestamps and then applies it one by one to the virtual model, simulating the real-world state evolution from 10:04:15 to the original current effective time of 10:05:00. During this process, the photo taken by Drone B at 10:04:15 is correctly applied, and its position and task status in the virtual model are updated. Subsequently, photos taken by Drone A at later times are also processed sequentially.

[0046] Once all relevant data has been processed in the correct time sequence, the digital twin virtual model reaches a new, highly consistent state with the physical world. This state accurately reflects the actual positions and work results of the two drones as of 10:05:00.

[0047] Finally, the inspection control module uses this corrected and reliable latest current state to execute subsequent intelligent decisions. For example, the system might discover that because drone B's data has been processed correctly, its assigned area has actually been filmed, so the original plan to send drone A to that area for supplementary filming can be cancelled, and drone A can be assigned a new inspection point instead. Alternatively, the system might adjust drone B's subsequent flight path to avoid the newly identified signal blind spot.

[0048] Through this backtracking and replay mechanism, this application ensures that the digital twin model can maintain its spatiotemporal consistency even in harsh communication environments and with severe data delays and out-of-order data. This provides a solid and reliable data foundation for upper-level collaborative control and intelligent decision-making, avoiding scheduling chaos and security risks caused by model distortion.

[0049] Furthermore, the inspection data includes sensor-collected data, drone location information, drone flight trajectory, and first timestamp.

[0050] Sensor data acquisition is the core output of inspection tasks, and its form can be diverse. For example, in the dam inspection scenario mentioned above, it could be still photographs taken by a high-resolution visible light camera mounted on a drone to identify tiny cracks on the dam surface; it could be continuous ultra-high-definition video streams to record the dynamic process of gate opening and closing; or it could be thermal maps collected by an infrared thermal imaging camera to detect abnormal temperature areas caused by seepage inside the dam. Drone location information typically consists of longitude, latitude, and altitude, precisely indicating the drone's absolute position in space at the time of sensor data acquisition. The drone's flight trajectory is a time series composed of a series of continuous location information points, recording the drone's movement path over a period of time. The first timestamp, as metadata, is tightly bound to each piece of sensor data and location information, ensuring that all data has a precise temporal identity.

[0051] Furthermore, the digital twin virtual model includes a three-dimensional geometric model of the drone, a three-dimensional geometric model of the object being inspected, a status list of the drone, and a task schedule. The current valid time is the latest time point recorded in the digital twin virtual model that reflects the state of the physical world.

[0052] The 3D geometric model of a drone is a precise digital representation of its physical entity, including information such as its size, shape, rotor position, and camera mounting points. The 3D geometric model of the object being inspected, such as the overall model of a dam, is pre-constructed using technologies like LiDAR scanning or oblique photogrammetry, providing a unified geospatial reference for all inspection data. The drone's status list is a dynamically updated database that records key operational parameters for each drone in real time, such as remaining battery percentage, current flight mode (e.g., hovering, linear flight, manual control), flight speed, wind speed, and gimbal pitch angle. The task schedule acts like a project management tool, breaking down the entire inspection task into several sub-tasks (e.g., photographing drainage hole #3, inspecting the top of gate #2) and tracking the execution status of each sub-task (e.g., pending assignment, in progress, completed, failed). The current valid time, as a top-level attribute of the model, marks the synchronous time segment to which all this dynamic information in the model points.

[0053] Furthermore, prior to the step of acquiring inspection data with first timestamps uploaded by multiple drones, the following steps are also included: Data verification and transmission are performed when the drone is in an area with good signal. Multiple sets of first timestamps and corresponding second timestamps are collected, and a time mapping function between the first timestamp and the difference between the first timestamp and the second timestamp is constructed. The steps to obtain inspection data with first-timestamps uploaded by multiple drones include: The first timestamp of the inspection data is corrected using a time mapping function to obtain the corrected first timestamp.

[0054] This step introduces a pre-calibration mechanism designed to improve the accuracy of the first timestamp from the outset. Although drones synchronize their time via the Global Navigation Satellite System, their internal clocks may still experience slight drifts during long-term operation or in complex electromagnetic environments.

[0055] In practice, a calibration procedure can be performed before the drone takes off, or when flying over an open area with known excellent signal strength during the mission. The drone continuously sends a series of data packets with its own first timestamp, and the ground station records the second timestamp of each received packet. Since the communication link is of high quality and stable at this time, the transmission delay can be considered to be a relatively fixed minimum value, or its variation pattern is predictable. By collecting a large number of such (first timestamp, second timestamp) data pairs, the relationship between the two can be analyzed.

[0056] There are several ways to construct a time mapping function. A simple approach is to assume a linear deviation between the drone's clock and the ground station's clock, essentially constructing a quadratic equation. By collecting multiple data pairs, the two parameters of the quadratic equation can be fitted using the least squares method, resulting in a linear function. A more complex approach, if the clock deviation is non-linear, or if the network latency, while small, still fluctuates, can employ machine learning methods. For example, a small neural network or a lookup table model can be trained, taking the drone's timestamp as input and outputting a correction value.

[0057] During subsequent formal inspections, each time the ground station receives a data packet with the original first timestamp, it first performs a calculation using the pre-built time mapping function to obtain a corrected, more accurate first timestamp. For example, if the function calculates that the current drone's clock is 50 milliseconds ahead of the standard time, the system will subtract 50 milliseconds from the original first timestamp and use this corrected timestamp in all subsequent comparisons and deductions. The advantage of this approach is that it eliminates interference from the device's own clock errors, allowing subsequent latency assessments to focus entirely on network transmission issues, thereby improving the accuracy and reliability of the entire system.

[0058] Furthermore, the step of comparing the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model includes: If the first timestamp is not earlier than the current valid time, and the time difference between the first timestamp and the current valid time is less than or equal to a preset threshold, and the inspection data conforms to the expected behavior logic of the drone, then the state of the digital twin virtual model is updated using the inspection data, and the current valid time of the digital twin virtual model is updated to the first timestamp.

[0059] This step defines in detail the fast channel for processing normal, near real-time data, which, together with the aforementioned backtracking and replay mechanism for processing delayed data, constitutes a complete state update logic.

[0060] When a new data packet arrives, the system first checks its first timestamp. If this timestamp is equal to or later than the model's current valid time, and the time difference between the two is within a very small tolerance range (such as the aforementioned 5-second threshold), it indicates that the data is fresh.

[0061] However, mere time freshness is insufficient; the system also performs a logical compliance check. The so-called expected behavior logic refers to the system's prediction of the drone's behavior at the next moment based on its current flight plan and status. For example, if the system instructs drone A to fly east at a speed of 3 meters per second, the system expects that the location data received in the next second should be approximately 3 meters east of its original position. If a new data packet shows that drone A suddenly appears more than 1 kilometer away, even if its timestamp is normal, this data will be considered inconsistent with the expected behavior logic and may be marked as anomalous data for isolation and review, rather than being directly used to update the model. This check effectively prevents invalid data caused by sensor malfunctions such as sudden jumps in positioning signals from contaminating the digital twin model.

[0062] A data packet is considered valid and reliable only when it simultaneously meets both the conditions of time freshness and logical conformity. At this point, the system directly uses the information in the data packet to update the status of the corresponding drone in the virtual model, such as updating its 3D model position and refreshing the battery readings in its status list. After the update is complete, the system advances the current valid time of the digital twin virtual model to the first timestamp of the data packet it just processed. This process ensures that, under good communication conditions, the system can maintain synchronization with the physical world in the most efficient and lowest-overhead manner.

[0063] In the above implementation, when the system detects delayed data, it immediately triggers state backtracking and re-analysis, which is effective when dealing with single or sporadic delayed data. However, in scenarios where a drone briefly enters a signal blind spot and then resumes communication, a large number of continuous delayed data packets may flood in within a short period. Performing a complete backtracking and re-analysis process for each data packet would lead to a surge in system computational load, and the frequent state backtracking operations themselves would consume a lot of time, thus reducing the real-time performance of the system response.

[0064] To optimize this process, this application further proposes that, after the step of determining that the inspection data is delayed data, and before the step of backtracking the state of the digital twin virtual model to the historical moment corresponding to the first timestamp, the following steps are also included: Store the delayed data in the delayed processing buffer and set the aggregation time window based on the first timestamp of the delayed data; Within the aggregation time window, if subsequent delayed data with the first timestamp falling within the coverage area of ​​the aggregation time window is received, the subsequent delayed data will be stored together in the delayed processing buffer. When the aggregation time window is closed, all delayed data is extracted from the delayed pending buffer, and the target delayed data with the earliest first timestamp is identified.

[0065] The aggregation time window refers to a time range used to collect other latency data associated with the current latency data, ensuring that latency data arriving within a certain time period can be considered together. The aggregation time window can be set based on the first timestamp of the latency data. For example, it can start from the first timestamp and extend forward by a fixed duration (such as 5 seconds, 10 seconds, or longer) as the end point of the aggregation time window, or the window size can be dynamically adjusted according to system load and data arrival patterns.

[0066] This scheme introduces a buffering and aggregation mechanism for delayed data. When the system receives the first data packet determined to be delayed, it does not immediately trigger backtracking, but instead places it in a temporary delayed processing buffer. Simultaneously, the system starts a timer, i.e., an aggregation time window, for example, set to 2 seconds.

[0067] Within this 2-second window, the system continues to receive and inspect subsequent data packets. If a new data packet is also determined to be delayed data, and its first timestamp is temporally adjacent to existing data in the buffer (i.e., belongs to the same batch of delayed data), then this new data packet will also be stored in the buffer without triggering a new timer.

[0068] When the 2-second aggregation window expires and closes, the system considers that this wave of continuous delayed data has been largely received. Only then will it perform a one-time batch processing on all the delayed data collected in the buffer. The system will iterate through all data items in the buffer, find the one with the earliest timestamp, and mark it as the target delayed data.

[0069] In this way, the system integrates the previously scattered backtracking operations that might have required dozens or even hundreds of steps into a single, efficient batch process, greatly reducing system overhead and improving overall efficiency and stability when dealing with sudden and continuous network latency.

[0070] After aggregating batches of delayed data, the next key step is to efficiently integrate it into the digital twin model. To this end, this application further proposes that the steps for retrospectively tracing the state of the digital twin virtual model back to the historical moment corresponding to the first timestamp include: Using the first timestamp of the target delayed data as the backtracking anchor point, the digital twin virtual model is backtracked to the corresponding historical moment, and all delayed data in the delayed processing buffer is integrated into the digital twin virtual model in chronological order.

[0071] This scheme clarifies the specific backtracking and integration methods when batch processing delayed data. The system uses the first timestamp of the target delayed data identified in the previous step (i.e., the earliest occurrence time of all data in the buffer) as the unique backtracking anchor point.

[0072] Continuing with the dam inspection example, suppose that within a 2-second aggregation window, the buffer collects a total of 30 data packets generated by drone B between 10:04:15 and 10:04:45. The system identifies the earliest timestamp as 10:04:15. Therefore, the system performs only one state backtracking operation, precisely reverting the state of the entire digital twin virtual model to 10:04:15.

[0073] Subsequently, starting from this backtracking anchor point, the system rigorously sorts all 30 delayed data packets in the buffer, along with data from other drones (such as drone A) that arrived on time within the same time period (from 10:04:15 to the model's original current valid time), according to their respective first timestamps. Then, following this correct time sequence, the system continuously re-deduces the state changes one at a time until all data processing is complete.

[0074] This single-backtracking, batch integration method offers an order-of-magnitude improvement in computational efficiency compared to independently backtracking each delayed data point, ensuring that the state correction process is both accurate and efficient.

[0075] During the state re-simulation process, simply simulating based on data uploaded by the drone may still deviate from physical reality, as the inspection environment itself may dynamically change. To address this issue, this application further proposes that the steps for re-simulating the state changes of the digital twin virtual model from a historical moment to the current effective time include: Real-time capture of environmental change information in the inspection environment, and assign a base timestamp based on global standard time to each environmental change information to form a global event log; Using the first timestamp as an index, search the global event log for overlapping time anchors; If overlapping time anchors are detected, and there is a logical conflict between the inspection data and the environmental change information corresponding to the overlapping time anchors, the inspection data will be labeled with attributes according to the priority of the environmental change information, and a consistency warning will be issued. The physical constraints generated by environmental change information are used as boundary conditions to correct the flight trajectory of the UAV, and the state changes of the digital twin virtual model are re-inferred based on the corrected inspection data.

[0076] This approach introduces dynamic constraints from the external environment into the state replay process. The system maintains a global event log independent of the UAV data. This log records all environmental changes (i.e., environmental change information) that may affect the inspection operation. For example, if on-site management personnel manually designate a temporary no-fly zone through the console interface, the system will record: Event: New No-Fly Zone Z; Time: 10:04:30; Range: Coordinates (x1, y1) to (x2, y2). Alternatively, if a weather sensor detects that the wind speed in a certain area suddenly exceeds the safety threshold, the system will also record this.

[0077] When the system is replaying a state, for each unprocessed drone data packet, it considers not only the data itself but also its first timestamp to query the global event log. Suppose the system is processing the location data of drone B at 10:04:35, which shows that drone B is inside no-fly zone Z. The system queries the log and finds that this area was designated a no-fly zone at 10:04:30. This creates a logical conflict: the drone data indicates it entered a restricted area, while environmental rules prohibit its entry.

[0078] At this point, the system will assign special attributes to this drone data based on preset priorities, such as higher priority for environmental information from official instructions or physical sensors. For example, it might mark it as conflicting with environmental constraints and issue a consistency warning pop-up or audio alert to the operator interface. More importantly, the system will not blindly accept this conflicting data. Instead, it will initiate a correction process, using the physical constraint of not entering no-fly zone Z as a boundary condition to recalculate the drone's trajectory, and then use the corrected data for further simulation. This ensures that the re-simulated model state is not only correct in time but also physically and rule-wise reasonable and reliable.

[0079] When a logical conflict is detected between drone data and environmental events, effective trajectory correction is crucial to ensuring the model's realism. To address this, this application further proposes a method for correcting the drone's flight trajectory using physical constraints generated by environmental change information as boundary conditions. This method includes: Access distant views from nearby drones or monitoring images from ground cameras in the inspection environment at the time corresponding to the first timestamp. Based on distant views or monitoring images from ground cameras, the actual flight trajectory in inspection data with logical conflicts is recalculated so that the corrected actual flight trajectory conforms to the physical constraints generated by environmental change information.

[0080] This scheme provides a specific method for resolving data conflicts and reconstructing real trajectories using multi-source visual information.

[0081] Taking the aforementioned data showing that drone B entered no-fly zone Z at 10:04:35 as an example, at the same time as issuing the warning, the correction program will automatically start and query the system to see if any other cameras captured images of the vicinity of no-fly zone Z around 10:04:35.

[0082] The system might detect that the camera field of view of drone A, which was performing a wide-area surveillance mission at high altitude, happened to cover the area. The system would immediately retrieve the video clip recorded by drone A at that moment. Using image recognition algorithms, the system could locate drone B and an obstacle representing a no-fly zone (such as a construction vehicle in operation) in the distant view of drone A. The algorithm would analyze the trajectory of drone B relative to the construction vehicle and find that drone B was actually approaching the construction vehicle while circling around its edge, and did not enter the restricted area. The GPS positioning data uploaded by drone B itself might have drifted due to signal reflection or multipath effects, leading to the incorrect location.

[0083] Based on the actual motion path observed from a third-party perspective of UAV A, the system can recalculate a more accurate flight trajectory that conforms to physical constraints. Then, the system uses this cross-validated and corrected trajectory data to replace the original conflict data and continue to complete the state deduction.

[0084] Similarly, if fixed ground surveillance cameras are deployed on-site, the system can also access their footage for similar analysis and correction. This method of using multi-source data for corroboration and correction greatly enhances the self-correction capability and ultimate fidelity of the digital twin model in the face of data errors and environmental changes.

[0085] Secondly, see Figure 2 This application also provides a digital twin-based UAV collaborative inspection and control system for executing any of the aforementioned digital twin-based UAV collaborative inspection and control methods. The system includes: The data acquisition and storage module 210 is used to acquire inspection data with a first timestamp uploaded by multiple drones, record the time of receiving the inspection data as a second timestamp, and store the inspection data in the order of the second timestamps to form a status record sequence. The time comparison and backtracking module 220 is used to compare the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is not within the preset range, the inspection data is determined to be delayed data, and the state of the digital twin virtual model is backtracked to the historical moment corresponding to the first timestamp. The state deduction module 230 is used to re-deduce the state changes of the digital twin virtual model from the historical moment to the current effective time according to the order of the first timestamp of each inspection data in the state record sequence, starting from the historical moment, and use the result of the re-deduction as the latest current state of the digital twin virtual model. The inspection control module 240 is used to perform subsequent inspection task reallocation, drone flight route adjustment and / or shooting strategy adjustment based on the latest current status; The time function construction module is used before the step of acquiring inspection data with first timestamps uploaded by multiple drones. Data verification and transmission are performed when the drone is in an area with good signal. Multiple sets of first timestamps and corresponding second timestamps are collected, and a time mapping function between the first timestamp and the difference between the first timestamp and the second timestamp is constructed. The data aggregation module is used after the step of determining that the inspection data is delayed, and before the step of retrospectively tracing the state of the digital twin virtual model back to the historical moment corresponding to the first timestamp. Store the delayed data in the delayed processing buffer and set the aggregation time window based on the first timestamp of the delayed data; Within the aggregation time window, if subsequent delayed data with the first timestamp falling within the coverage area of ​​the aggregation time window is received, the subsequent delayed data will be stored together in the delayed processing buffer. When the aggregation time window is closed, all delayed data is extracted from the delayed pending buffer, and the target delayed data with the earliest first timestamp is identified.

[0086] This technical solution provides a physical system architecture that can implement the above methods, deconstructing the complex method process into functionally defined modules, providing a clear hardware or software functional division for the engineering implementation and deployment of the methods, and has strong practical value.

[0087] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A collaborative inspection and control method for unmanned aerial vehicles (UAVs) based on digital twins, characterized in that, The method includes: The system acquires inspection data with a first timestamp uploaded by multiple drones, records the time when the inspection data is received as a second timestamp, and stores the inspection data in the order of the second timestamps to form a status record sequence. The first timestamp of the newly received inspection data is compared with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is greater than a preset threshold, the inspection data is determined to be delayed data, and the state of the digital twin virtual model is traced back to the historical moment corresponding to the first timestamp. Starting from the historical moment, according to the order of the first timestamps of each of the inspection data in the state record sequence, the state changes of the digital twin virtual model from the historical moment to the current effective time are re-deduced, and the result of the re-deduction is taken as the latest current state of the digital twin virtual model; Based on the latest current status, subsequent inspection tasks will be reassigned, drone flight routes will be adjusted, and / or shooting strategies will be adjusted.

2. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, The inspection data includes sensor-collected data, UAV location information, UAV flight trajectory, and the first timestamp.

3. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, The digital twin virtual model includes a three-dimensional geometric model of the UAV, a three-dimensional geometric model of the object being inspected, a status list of the UAV, and a task progress table. The current valid time is the latest time point recorded in the digital twin virtual model that reflects the state of the physical world.

4. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, Before the step of acquiring inspection data with first timestamps uploaded by multiple drones, the following is also included: When the drone is in a good signal area, data verification and transmission are performed, and multiple sets of the first timestamp and the corresponding second timestamp are collected to construct a time mapping function between the first timestamp and the difference between the first timestamp and the second timestamp. The steps for obtaining inspection data with first timestamps uploaded by multiple drones include: The first timestamp of the inspection data is corrected using the time mapping function to obtain the corrected first timestamp.

5. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, The step of comparing the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model includes: If the first timestamp is not earlier than the current valid time, and the time difference between the first timestamp and the current valid time is less than or equal to the preset threshold, and the inspection data conforms to the expected behavior logic of the drone, then the state of the digital twin virtual model is updated using the inspection data, and the current valid time of the digital twin virtual model is updated to the first timestamp.

6. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, After the step of determining that the inspection data is delayed data, and before the step of retrospectively estimating the state of the digital twin virtual model to the historical moment corresponding to the first timestamp, the method further includes: The delayed data is stored in a delayed processing buffer, and an aggregation time window is set according to the first timestamp of the delayed data; Within the aggregation time window, if subsequent delayed data with the first timestamp falling within the coverage area of ​​the aggregation time window is received, the subsequent delayed data is stored together in the delay pending buffer. When the aggregation time window is closed, all the delay data is extracted from the delay pending buffer, and the target delay data with the earliest first timestamp is identified.

7. The UAV collaborative inspection control method based on digital twin according to claim 6, characterized in that, The step of retrospectively tracing the state of the digital twin virtual model back to the historical moment corresponding to the first timestamp includes: Using the first timestamp of the target delay data as the backtracking anchor point, the digital twin virtual model is backtracked to the corresponding historical moment, and all delay data in the delay buffer to be processed is integrated into the digital twin virtual model in chronological order.

8. The UAV collaborative inspection control method based on digital twin according to claim 1, characterized in that, The step of re-deducing the state changes of the digital twin virtual model from the historical moment to the current effective time includes: Real-time capture of environmental change information in the inspection environment, and assign a reference timestamp based on global standard time to each environmental change information to form a global event log; Using the first timestamp as an index, retrieve whether there are overlapping time anchors in the global event log; If overlapping time anchors are detected, and the inspection data and the environmental change information corresponding to the overlapping time anchors have logical conflicts, then the inspection data will be attribute-labeled according to the priority of the environmental change information, and a consistency warning will be issued. The physical constraints generated by the environmental change information are used as boundary conditions to correct the flight trajectory of the UAV, and the state changes of the digital twin virtual model are re-inferred based on the corrected inspection data.

9. A collaborative inspection and control method for unmanned aerial vehicles based on digital twins according to claim 8, characterized in that, The step of using the physical constraints generated by the environmental change information as boundary conditions to correct the flight trajectory of the UAV includes: Call up distant views of nearby drones in the inspection environment at the time corresponding to the first timestamp or monitoring images from ground cameras. Based on the distant view or the monitoring images from the ground camera, the actual flight trajectory in the inspection data that contains the logical conflict is recalculated so that the corrected actual flight trajectory conforms to the physical constraints generated by the environmental change information.

10. A digital twin-based UAV collaborative inspection control system, used to execute the digital twin-based UAV collaborative inspection control method as described in any one of claims 1 to 9, characterized in that, The system includes: The data acquisition and storage module is used to acquire inspection data with a first timestamp uploaded by multiple drones, record the time of receiving the inspection data as a second timestamp, and store the inspection data in the order of the second timestamps to form a status record sequence. The time comparison and backtracking module is used to compare the first timestamp of the newly received inspection data with the current valid time of the digital twin virtual model. If the first timestamp is earlier than the current valid time and the time difference between the first timestamp and the current valid time is not within a preset range, the inspection data is determined to be delayed data, and the state of the digital twin virtual model is backtracked to the historical moment corresponding to the first timestamp. The state deduction module is used to re-determine the state changes of the digital twin virtual model from the historical moment to the current effective time, according to the order of the first timestamps of each of the inspection data in the state record sequence, starting from the historical moment, and to use the result of the re-deduction as the latest current state of the digital twin virtual model; The inspection control module is used to perform subsequent inspection task reallocation, UAV flight route adjustment, and / or shooting strategy adjustment based on the latest current status.

Images