An unmanned aerial vehicle virtual-real combination verification device and method

Abstract

This invention provides a virtual-real hybrid verification device and method for unmanned aerial vehicles (UAVs), solving the problem that the number of nodes in virtual-real hybrid application scenarios is too small to execute complex tasks. It includes: a central scheduling and management module for parsing scenario configuration files, extracting information from elements, and setting target parameters; configuring the interaction relationships between elements and distributing all elements to multiple node computers; a UDP distributed data transmission module for sending and receiving data packets; a running thread management module for managing multiple computing threads enabled on a single computer and the UAV nodes allocated to each thread; a UAV six-degree-of-freedom model calculation module for performing centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the next corresponding task for each UAV; and a formation maintenance and change implementation module for reorganizing the UAV formations based on their current positions and corresponding tasks.

Description

Technical Field

[0001] This invention relates to the field of aviation technology, and specifically to a virtual-real combined verification device and method for unmanned aerial vehicles (UAVs). Background Technology

[0002] Drones are now widely used in real-world scenarios for real-time target observation, tracking, and deployment against various air defense forces. As mission complexity increases, single or small groups of drones are insufficient, necessitating the use of a high number of nodes to ensure desired results. However, relying solely on physical drones is not only costly but also presents challenges due to factors such as collisions, failure rates, test site limitations, and weather conditions. Therefore, a hybrid virtual-real approach offers higher fault tolerance and greater realism. Currently, hybrid virtual-real applications typically involve fewer than a hundred nodes, largely limited by the processing power of a single computer. Furthermore, both physical and virtual nodes are often of a single type and model, using only drones of the same type and model for verification. This limited number and restricted drone types and models result in relatively simple cluster operations, far removed from truly comprehensive missions. Therefore, a hybrid virtual-real device capable of supporting thousands of concurrent nodes performing complex tasks is urgently needed. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a device and method for verifying virtual and real integration of unmanned aerial vehicles (UAVs), which solves the problem that the number of nodes in virtual and real integration application scenarios is too small to execute tasks with relatively complex content.

[0004] In a first aspect, embodiments of the present invention provide a virtual-real hybrid verification device for unmanned aerial vehicles (UAVs), comprising: a central scheduling and management module, used to parse scene configuration files, extract information from each element in the scene, and set target parameters for each element; configure the interaction relationships between each element, and allocate all elements to multiple node computers; a UDP distributed data transmission module, used to send and receive data packets, and read relevant information based on the content of the data packets; a running thread management module, used to manage multiple computing threads enabled on a single computer, the UAV nodes allocated to each thread, and process the results after completing a single-step calculation; a UAV six-degree-of-freedom model calculation module, used to perform centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the corresponding tasks for each UAV in the next step; and a formation maintenance and change implementation module, used to reorganize each UAV formation based on the current position of each UAV formation and the corresponding task.

[0005] In one implementation, the drone includes: a digital drone, a semi-physical drone, and a physical drone.

[0006] In one embodiment, the UAV virtual-real combination verification device further includes: a real-machine access module, used to verify whether the access scenario and algorithm are applicable to the real machine, and simultaneously acquire the status of the digital UAV and the instructions from the ground end, transmit the current status and image of the physical UAV and send them to the computer so that the model can determine whether to adjust the task allocation and formation reorganization according to the current status and image of the physical UAV.

[0007] In one implementation, the UDP distributed data transmission module is further configured to: maintain a list of publishers and subscribers; establish and maintain a mapping relationship between publishers and subscribers; and transmit the corresponding tasks of each UAV for the next step to the list of publishers and subscribers, based on the mapping relationship between publishers and subscribers, to transmit the corresponding tasks of each UAV for the next step. In one implementation, multiple node computers share computing data through a shared memory network.

[0008] In one implementation, the central scheduling management module is also used to switch drone nodes representing physical drones to adjust the formation and position of physical drones in the scene; and / or, drone nodes representing physical drones can coordinate with drone nodes representing virtual drones to perform tasks through coordinate transformation.

[0009] Secondly, a method for combining virtual and real-world verification of unmanned aerial vehicles (UAVs) includes: parsing a scene configuration file, extracting information from each element in the scene, and setting target parameters for each element; configuring the interaction relationships between each element and distributing all elements to multiple node computers; receiving and reading data packets, and managing multiple computing threads enabled on a single computer and the UAV nodes allocated to each thread based on the content of the read data packets to complete the result processing after single-step calculation; performing centralized calculations on the dynamic models of all simulated UAV nodes on each node computer based on the processed single-step calculation results to obtain the corresponding tasks for each UAV in the next step; transmitting the corresponding tasks for each UAV in the next step, and reorganizing each UAV formation based on the current position of each UAV formation and the corresponding tasks.

[0010] In one embodiment, the UAV virtual-real combination verification method further includes: verifying whether the access scenario and algorithm are applicable to the actual UAV, while acquiring the status of the digital UAV and the instructions from the ground end, transmitting the status and image of the current physical UAV and sending them to the computer, so that the model can determine whether to adjust the task allocation and formation reorganization according to the status and image of the current physical UAV.

[0011] Thirdly, a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0012] Fourthly, a computer program product includes a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0013] This invention provides a virtual-real integration verification device and method for unmanned aerial vehicles (UAVs). The UAV virtual-real integration verification device includes a central scheduling and management module for parsing scene configuration files, extracting information from each element in the scene, and setting target parameters for each element; configuring the interaction relationships between elements and distributing all elements to multiple node computers; a UDP distributed data transmission module for sending and receiving data packets and reading relevant information based on the content of the data packets; a running thread management module for managing multiple computing threads enabled on a single computer, the UAV nodes allocated to each thread, and processing the results after completing a single-step calculation; a UAV six-degree-of-freedom model calculation module for performing centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the corresponding tasks for each UAV in the next step; a formation maintenance and change implementation module for reorganizing each UAV formation based on the current position of each UAV formation and the corresponding task; and a real-machine access module for verifying whether the access scene and algorithm are applicable to the real machine, while acquiring the status of the digital UAV and the instructions from the ground end, transmitting the current status and image of the physical UAV and sending it to the computer so that the model can determine whether to adjust the task allocation and formation reorganization based on the current status and image of the physical UAV.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] (1) The central scheduling management module and the real aircraft access module enable the real aircraft to be in any position in any formation in the scene. Therefore, it can not only verify the possibility and rationality of real flight at each position, but also save costs greatly. It is not necessary to use real aircraft for all flights, but only to switch the digital nodes represented by the real aircraft.

[0016] (2) The central scheduling management module allows virtual nodes to be set in any position, such as in areas that the real machine cannot reach. Real nodes can coordinate with virtual nodes to maintain and change their formations through coordinate transformation, thus enabling the module to perform tasks collaboratively.

[0017] (3) The central scheduling and management module and the UDP distributed data transmission module distribute all nodes to multiple computers, utilize shared memory for rapid information exchange, drive the thread management module to allocate to each thread, and then run the UAV six-degree-of-freedom model calculation module for efficient computation. This allows the invention to support a high number of nodes, exceeding thousands, thus enabling the simulation of highly complex tasks. The formation maintenance and change implementation module realizes collaboration within and across formations, as well as the continuous execution of multiple tasks by a single machine, switching formation positions and modes, etc.

[0018] (4) The physical access module is relatively independent of other modules in the device. It supports access to physical nodes including rotor and fixed-wing, covering the two major categories of UAVs. It also supports mixed programming of different models. Digital nodes can also be replaced with the model to be verified. It has high flexibility and strong design framework inclusiveness. It only needs to be configured and the model imported in the central scheduling and management module. Then the UAV six degrees of freedom model calculation module performs the calculation.

[0019] (5) The actual aircraft access module is equipped with a high-performance airborne board and a MESH distributed networking data link, which can support the embedded access verification of various intelligent algorithms that consume a variety of computing resources, realize decentralized distributed communication of UAVs, and flexibly switch the aerial topology network and formation. It is also equipped with a variety of visible light and infrared pods, which can support intelligent algorithms to identify targets in real time in day and night environments. The high-bandwidth data link can also support the real-time transmission of high-resolution video images back to the ground for analysis and processing. Attached Figure Description

[0020] Figure 1 The diagram shown is a structural schematic of a drone-based virtual-real combined verification device according to an embodiment of the present invention.

[0021] Figure 2 The diagram shown is a flowchart of a method for combining virtual and real verification of unmanned aerial vehicles (UAVs) according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This embodiment provides a drone virtual-real combination verification device 100, such as... Figure 1 As shown, the UAV virtual-real integration verification device 100 consists of six main modules: a central scheduling and management module 10, a UDP distributed data transmission module 20, a running thread management module 30, a formation maintenance and change implementation module 50, a UAV six-degree-of-freedom model calculation module 40, and a real-machine access module 60. The functions of each module are described below.

[0024] The central scheduling and management module 10 is used to parse the scene configuration file, extract information from each element in the scene, set the target parameters of each element, configure the interaction relationship between each element, and distribute all elements to multiple node computers.

[0025] The UDP distributed data transmission module 20 is used to send and receive data packets and read relevant information based on the content of the data packets;

[0026] The thread management module 30 is used to manage the multiple computing threads enabled in a single computer, the drone nodes allocated to each thread, and the result processing after completing a single-step calculation.

[0027] The UAV six-degree-of-freedom model calculation module 40 is used to perform centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the corresponding tasks of each UAV in the next step.

[0028] The formation maintenance and change implementation module 50 is used to reorganize each UAV formation based on its current position and the corresponding task.

[0029] The real-machine access module 60 is used to verify whether the access scenario and algorithm are applicable to the real machine. It also acquires the status of the digital UAV and commands from the ground, and transmits the current status and images of the physical UAV to the computer. This allows the model to determine whether to adjust task allocation and formation reorganization based on the current status and images of the physical UAV. There are no restrictions on the different power modes and models used by the real UAV; simply import the corresponding UAV model and specify it in the configuration input of the central scheduling module 10. Then, it will allocate corresponding tasks according to the UAV's capabilities, and the imported model will be called by the UAV six-degree-of-freedom model calculation module 40 to complete the calculation.

[0030] Meanwhile, the virtual-real combination verification device 100 for the UAV is equipped with a high-performance airborne board, a high-bandwidth MESH distributed networking data link, and visible light and infrared pods on each actual UAV, which supports intelligent algorithm import and high-resolution image collection and transmission in terms of hardware.

[0031] In one implementation, the central scheduling management module 10 serves as the overall control management module, primarily performing two functions: (1) parsing the scenario configuration file and constructing corresponding areas, weather conditions, terrain features, dynamic and static elements, etc., in the simulation environment for the set tasks. (2) configuring the interaction relationships between elements, such as the initial state where each UAV node needs information from other UAVs in the same formation to maintain the formation and spacing, the lead UAVs need to exchange information to ensure the relative positions of each formation and whether the corresponding type of UAV can be selected when switching formations in the central scheduling.

[0032] In one implementation, the UDP distributed data transmission module 20 is responsible for sending and receiving data packets transmitted under multicast and unicast relationships, and reading relevant information based on the packet header content. Simultaneously, when drones within the same formation are distributed across different computers, their information is written to a shared memory card to achieve faster and more efficient cross-machine computing.

[0033] In one implementation, the thread management module 30 manages the multiple computing threads enabled in a single computer, the drone nodes allocated to each thread, the merging of results after single-step calculation, and operations such as binding, unbinding, and swapping CPUs.

[0034] In one implementation, the UAV six-degree-of-freedom model calculation module 40 calculates the position and attitude control results to be reached by the next node based on the dynamic model of all simulated UAV nodes used, such as the speed and angle of the rotor, servo motor, and control surface.

[0035] In one implementation, the formation maintenance and change implementation module 50 is responsible for maintaining the spacing within the formation after issuing an instruction, avoiding and recovering from obstacles, maintaining the formation, and splitting, reorganizing, creating a new formation, and implementing the relative positions of each node after receiving a change instruction.

[0036] In one implementation, the real-machine access module 60 is responsible for real-time calculation of the real-machine model, image acquisition from visible light / infrared sensors, and information transmission with the ground terminal (virtual drone node).

[0037] The UAV virtual-real integration verification device 100 provided in this embodiment allows for flexible switching between virtual and real nodes, as the difference between virtual and real nodes lies only in their assigned ID numbers and information acquisition methods. The central scheduling and management module 10 and the real node access module 60 enable the real node to be positioned at any location within any formation in the scene. This not only verifies the feasibility and rationality of actual flight at various locations but also significantly reduces costs, eliminating the need for all flights to be conducted with real nodes; only the digital nodes represented by the real nodes need to be switched. Furthermore, virtual nodes can be set in arbitrary locations, such as areas inaccessible to real nodes. Real nodes can coordinate with virtual nodes through coordinate transformation, relying on formation maintenance and changes to achieve module-wide task execution. Additionally, the UAV virtual-real integration verification device 100 distributes all nodes across multiple computers using the central scheduling and management module 10 and the UDP distributed data transmission module 20, utilizing shared memory for rapid information exchange. The driving thread management module 30 allocates resources to each thread, and the UAV six-degree-of-freedom model calculation module 40 performs efficient computation. This allows the invention to support a high number of nodes, exceeding a thousand, thus enabling the simulation of highly complex tasks. The formation maintenance and change implementation module 50 enables coordination within and across formations, as well as the continuous execution of multiple tasks by a single drone, and the switching of formation positions and modes. In one embodiment of the invention, the physical drone can be a flexible heterogeneous combination of rotorcraft and fixed-wing drones, supporting virtual, semi-physical, and fully physical methods for pre-flight verification combining virtual and physical methods. Since the physical drone access module 60 is relatively independent of other modules in the device, it supports access to physical nodes including rotorcraft and fixed-wing drones, covering the two major categories of drones, and supports mixed formation of different models. Digital nodes can also be replaced with the models to be verified, offering high flexibility and a strong design framework. Only the corresponding configuration and model import are required in the central scheduling and management module 10, after which the drone six-degree-of-freedom model calculation module 40 performs the calculations. The physical drones are equipped with computing boards to calculate their own information, and signals and visible light / infrared sensor image information are transmitted between physical drones and the ground end through a distributed MESH network data link. The actual aircraft access module 60 is equipped with a high-performance airborne board and MESH distributed networking, which can support the embedded access verification of various intelligent algorithms that consume a variety of computing resources, realize decentralized distributed communication for UAVs, and flexibly switch aerial topology networks and formations. It is also equipped with a variety of visible light and infrared pods, which can support intelligent algorithms to identify targets in real time in day and night environments. The high-bandwidth data link can also support the real-time transmission of high-resolution video images back to the ground for analysis and processing.

[0038] In one embodiment of the present invention, multiple node computers share computing data through a shared memory network.

[0039] In one embodiment of the present invention, the central scheduling management module 10 first parses the scene configuration file, extracts information from various static and dynamic elements in the scene, and sets parameters such as the position, boundary, and capabilities of each element. Then, the central scheduling management module 10 configures the interaction relationships between each element, such as whether element A and element B need to have a publish-subscribe relationship. Subsequently, all elements are allocated to node computers, and the final distributed interaction relationships form an overall network mapping list. Further, the simulated UAV nodes transmit data such as the current position, speed, heading, attitude, assigned tasks and roles, and the status of static and dynamic elements in other scenes (including the position of actual flying nodes, considering inter-flying collisions, collisions with obstacles, etc.) to the running thread management module 30 through the UDP distributed data transmission module 20. The running thread management module 30 manages the multiple computing threads enabled in a single computer and the UAV nodes allocated to each thread to complete the result processing after single-step calculation. Then, the UAV six-degree-of-freedom model calculation module 40 performs centralized and efficient calculations on the dynamic models of all simulated UAV nodes on each node computer to calculate the position and state of each UAV in the next step. Finally, the calculated position and status information of each UAV is transmitted to the formation maintenance and change implementation module 50 via the UDP distributed data transmission module 20. The formation maintenance and change implementation module 50 reorganizes the formations according to their arrival positions, corresponding tasks, and current situation. For example, it may send out UAVs with visible light / infrared capabilities that are scattered in multiple formations to form a new formation and then separate them to carry out reconnaissance missions after arriving at a specific area. The remaining UAVs in the original formation also maintain their formation, change their relative positions, or enter a new formation according to the instructions they receive. The calculation of the actual flight nodes is completed by the actual aircraft access module 60, which transmits the acquired images and current position status to the ground terminal and sends back the required information on other virtual node UAVs and scene environment.

[0040] In one embodiment of the present invention, the central scheduling management module 10 distributes all elements to multiple node computers primarily based on the number of initial formations, the number of UAV nodes within each formation, and their types. For example, a reconnaissance UAV carrying a pod needs to process a large amount of image information, thus consuming significant resources. The central scheduling management module 10 distributes resources evenly to each node computer based on the degree of computational resource consumption, such as initial formations 1, 4, 9, ... to node 1 computer, and initial formations 2, 3, 5, ... to node 2 computer. This method maximizes the utilization of computational resources and ensures real-time information synchronization. Furthermore, since the operations required for scene initialization are far more numerous than those at each time interval after the simulation begins, having nodes in the same initial formation located on the same computer maximizes the speed of scene initialization. In addition, the central scheduling management module 10 is also used to switch UAV nodes representing physical UAVs to adjust the formation and position of physical UAVs in the scene; and / or, UAV nodes representing physical UAVs coordinate with UAV nodes representing virtual UAVs to perform tasks through coordinate transformation. In the input scenario configuration file, you can select the type of drone, its initial position, and whether it is a real drone for each node. Alternatively, you can directly select to generate a specific type of drone and specify whether it is a real drone in the interactive interface, and then drag its position interactively to place it where the task requires. Therefore, for nodes that are specified in the scenario as requiring real drone verification, simply configure them as real drones. For real drones, if you need to simulate flight at a certain location but the real drone cannot reach that location due to limitations, since the physical coordinates of the takeoff airport are already determined, it will reach the previously set position through coordinate transformation at the start of the task, thus achieving collaboration with the virtual drone.

[0041] In one embodiment of the present invention, the UDP distributed data transmission module 20 uses the UDP protocol for communication. UDP (User Datagram Protocol) is a protocol within the Internet communication framework that allows computers to send messages to other systems on an IP network without prior communication to set up special transmission channels or data paths. UDP is used for time-sensitive transmissions, such as video playback or games. It is a simpler connectionless protocol with little or no error checking and tracking mechanisms to achieve faster data transmission rates.

[0042] The UDP operating mode is as follows:

[0043] Packet creation. This process begins with the program that wants to send data. This data is divided into smaller chunks called datagrams. Each UDP datagram contains a payload and a header, which includes necessary addressing information, such as the source and destination port numbers, the length of the datagram, and a checksum for verifying data integrity.

[0044] No connection established. Unlike TCP, UDP does not establish a connection before sending data. There is no interaction between the sender and receiver, which eliminates the latency caused by the setup process.

[0045] Data transmission. Once the datagram is ready, it is handed over to the IP network layer, which encapsulates the UDP datagram within an IP packet and sends it to its destination. UDP itself does not track datagrams after transmission; it simply sends the data.

[0046] Minimal error checking. At the receiving end, the UDP protocol layer processes the received datagrams. This involves checking the checksum to ensure the data is not corrupted. However, if the checksum fails, the packet is silently discarded. UDP does not attempt to retransmit the data.

[0047] No order guarantee or reliability. UDP does not guarantee that datagrams arrive in the order they were sent, or even that they all arrive. Programs using UDP are responsible for handling these issues if necessary.

[0048] Processing incoming data. Programs using UDP are typically designed to handle datagram loss or reordering. For example, a video streaming application might simply skip lost packets, while a more interactive program might implement its own methods to request retransmission.

[0049] In one embodiment of the present invention, the UDP distributed data transmission module 20 implements the following functions:

[0050] 1. Maintain a list of publishers and subscribers for easy creation and review of subscription relationships. The list includes information such as object name and VMIC number.

[0051] 2. Establish and maintain the mapping relationship between publishers and subscribers. Publishers announce the type of data they are publishing, and subscribers also announce the types of data they are interested in. Publishers and subscribers of the same data type will establish a mapping relationship. When a publisher broadcasts the latest data, subscribers interested in that type will receive the data. Each publisher and subscriber will be assigned a unique ID. Each publisher will create an interest list, and each subscriber will search the interest list, establishing topic links using the publisher ID. When a publisher publishes a new topic, they will send a topic publication event (flag) to the system. Subscribers will then re-search their interest list and determine whether they should subscribe to the topic; if so, they will establish a new link.

[0052] 3. Provide objects, send and receive information. Data types for transmission are divided into object data and message data, and the methods for sending and receiving these two types differ.

[0053] a. Object Data: During the establishment of the publish-subscribe mapping connection, the publisher and subscriber obtain the addresses of objects in reflected memory. When the publisher transmits object data, it looks up the object queue using the object address; if the queue is not full, the free queue requests data storage space to store the object data and suspends it at the end of the object queue. When the queue is full, a circular overwrite occurs. When the subscriber server receives object data, it also uses the object address to look up the object queue. If the queue is not empty, it retrieves the object data from the header position and returns it to the subscriber.

[0054] b. Message Data: The message structure in reflected memory consists only of a name and a data storage area. After the publisher determines the address of the message data, it directly writes the data to overwrite the existing data. Once the address is found, the subscriber only needs to retrieve the message data and return it.

[0055] In one embodiment of the present invention, the UDP distributed data transmission module 20 transmits the corresponding tasks of each UAV to the publisher and subscriber list for the next step, based on the mapping relationship between the publisher and subscriber. After the central scheduling module 10 distributes all elements to multiple node computers, when UAVs in the same formation are distributed on different computers, the UDP distributed data transmission module 20 writes their information to a shared memory card, which improves efficiency compared to traditional cross-machine read and write information, especially for application scenarios such as UAV collaboration that require high real-time information synchronization. Thanks to the computing power of the multi-node computers utilized by the UDP distributed data transmission module 20, the number of nodes supported for verification by the device is greatly increased, thus supporting larger-scale task scenarios, covering a larger area of ​​maps, terrain and other elements, increasing the intended complexity and making it closer to real-world application scenarios.

[0056] In one embodiment of the present invention, when the running thread management module 30 receives the allocated UAV node, it distributes the node to each thread in real time based on the resource usage of all threads on the local machine. Each thread calculates the information at the current moment by running the UAV six-degree-of-freedom model calculation module 40.

[0057] In one embodiment of the present invention, after receiving the latest UAV node information, the formation maintenance and change implementation module 50 determines whether adjustments to each node are needed for the next moment based on formation maintenance and spacing within the formation. When different formations are flying, it is also necessary to ensure that no node within a different formation collides, or that the distance between formations needs to be shortened / increased; the formation maintenance and change implementation module 50 is also responsible for coordinating and resolving such situations. Simultaneously, if a task is received requiring the formation to be dismantled to deploy certain specific nodes, or the existing formation to be dismantled and reorganized to execute new changes, the formation maintenance and change implementation module 50 will also respond. The formation maintenance and change implementation module 50 can implement the above functions by importing the formation control algorithm to be verified through the central scheduling management module 10 before the UAV virtual-real integration verification device 100 starts operating. This device also provides a built-in example control algorithm.

[0058] In one embodiment of the invention, a time synchronization server connects to each computer to maintain time synchronization, while the clock of the physical drone is acquired by GPS. Real-time clock management sends a clock synchronization command before the simulation begins. Each node records its current physical clock upon receiving the command; this current time is used as the start time. To ensure high-precision clock synchronization, a compensation algorithm for communication delay is considered. When the physical model is used in conjunction with the simulation system, the physical clock serves as the system's universal clock. Based on the interconnection of numerous subsystems with different time-progression granularities, a centralized time management mode is selected.

[0059] In one embodiment of the present invention, the central scheduling management module 10 is further used to switch drone nodes representing physical drones to adjust the formation and position of physical drones in the scene; and / or, drone nodes representing physical drones can coordinate with drone nodes representing virtual drones to perform tasks through coordinate transformation. Physical drones can be located at any position within any formation in the scene, thus not only verifying the feasibility and rationality of actual flight at each position, but also significantly saving costs, as it is not necessary to use physical drones for all flights; only the drone nodes representing physical drones need to be switched. Furthermore, drone nodes representing virtual drones can be set at any position, such as in areas inaccessible to physical drones, and can coordinate with drone nodes representing virtual drones to perform tasks through coordinate transformation.

[0060] This embodiment provides a method for combining virtual and real-world verification of unmanned aerial vehicles (UAVs), such as... Figure 2 As shown, the virtual-real combined verification method for this UAV includes:

[0061] Step 01: Parse the scene configuration file, extract information from each element in the scene, and set the target parameters for each element;

[0062] Step 02: Configure the interaction relationships between the elements and distribute all elements to multiple node computers;

[0063] Step 03: Receive and read data packets, and manage the multiple computing threads enabled in a single computer and the drone nodes allocated to each thread based on the content of the read data packets, in order to complete the result processing after the single-step calculation;

[0064] Step 04: Based on the processed single-step calculation results, perform centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the corresponding tasks of each UAV in the next step.

[0065] Step 05: Transmit the corresponding tasks for each UAV in the next step, and reorganize each UAV formation based on the current position of each UAV formation and the corresponding tasks.

[0066] In one embodiment of the present invention, the virtual-real combination verification method for drones further includes: verifying whether the access scenario and algorithm are applicable to the actual drone, while acquiring the status of the digital drone and the instructions from the ground end, transmitting the status and image of the current physical drone and sending them to the computer, so that the model can determine whether to adjust the task allocation and formation reorganization based on the status and image of the current physical drone.

[0067] In one embodiment of the present invention, the UAV virtual-real combined verification method includes:

[0068] 1. Configure UDP broadcast, multicast, and unicast addresses and ports;

[0069] 2. Set the number of processors allocated for model computation, the number of times a thread runs in one cycle, the model running step size, the API latency for starting the model task, and the list of array IDs of the scene to be parsed on the local machine.

[0070] 3. Initialize UDP broadcast, multicast, and unicast, and bind addresses and ports.

[0071] 4. Set up the communication domain, establish a publish-subscribe relationship, send the registration topic, and enable distributed parallel computing. Parallel computing refers to the process of executing multiple processors, programs, or computations simultaneously. It is usually a computing architecture in which large problems are broken down into independent, smaller, and usually similar parts that can be processed at once. It is accomplished by multiple CPUs communicating through shared memory, which merges the results after completion. It helps to perform large computations because it distributes large problems among multiple processors. Parallel computing also helps to speed up application processing and task solving by increasing the available computing power of the system. Most supercomputers operate on the principle of parallel computing. Operational scenarios that require a large amount of processing power or computing power usually use parallel processing. Parallel computer architectures are classified according to the level of parallelism supported by the hardware. This invention adopts two classifications: (1) Multi-core computing, a computer processor integrated circuit containing two or more different processing cores is called a multi-core processor, which has the ability to execute program instructions simultaneously. The cores can implement architectures such as Very Long Instruction Word (VLIW), superscalar, multi-threaded, or vector, and are integrated on a single integrated circuit chip or multiple chips in a single chip package. Multi-core architectures are classified as heterogeneous architectures composed of different cores, or as homogeneous architectures composed of only identical cores. (2) Distributed computing: The components of a distributed system reside on different networked computers. These networked computers coordinate their operations by communicating via HTTP, message queues similar to Remote Procedure Calls (RPC), and connectors. Component concurrency and independent component failure are characteristics of distributed systems.

[0072] Due to the reflective memory card's transfer mode, reading shared memory takes a significant amount of time, so it's crucial to minimize the number of reads and writes to the shared memory pointer. Local shared memory is a circular queue with head and tail pointers. To monitor where they write data, each publisher tracks its own head and tail pointers. Similarly, each subscriber tracks its own head and tail pointers to monitor where it reads data. A monitoring process is established when publishers and subscribers are created to track the state of each publisher and subscriber, as well as their head and tail pointers. When writing data, the publisher first determines if there is sufficient space by checking the size of the data recorded in the header packet. If so, it writes the data directly, reads the head pointer position, and modifies the head pointer position in the circular queue, without changing the tail pointer in this instance. After reading the packet length, the tail pointer position is only changed if there isn't enough actual space to store the written data. When changing the tail pointer position, the monitoring process finds the last tail pointer, compares it with the tail pointer of each process, and then moves the tail pointer of the circular queue to the position of that process. At this point, the publisher can write data because the circular queue has created new storage space. This reduces the amount of data required to read the pointer and speeds up the transfer efficiency of the reflective memory card.

[0073] Publishers are formed when data must be sent. The publish data cache contains the data transmitted by the publisher. When data is sent via broadcast or peer-to-peer, the publisher submits the data to a circular queue. When subscribers read data, they first place it in the receive data cache, and then categorize it according to its type. For example, broadcast data is placed on a broadcast data list, and peer-to-peer data is placed on a peer-to-peer data list.

[0074] 5. When the initialization command is received via UDP: (taking a master-slave structure as an example)

[0075] (1) Set the longitude, latitude, and altitude information of the overall waypoints. According to the formation, set the ID number of each virtual node (the ID number is used to distinguish the node category, where 1-10000 are digital aircraft, 10001-20000 are semi-physical aircraft, and 20001-30000 are physical aircraft), aircraft type (rotor or fixed-wing), affiliated faction (red or blue), takeoff point longitude, latitude, altitude, field height relative to the ground, azimuth, speed, formation ID number, role in the formation (lead or follower), formation mode (master-slave structure, virtual structure, etc.), and longitude, latitude, and altitude information of the waypoints to be flown by this formation.

[0076] (2) Set the relative position of the lead machine in the formation, add the lead machine ID to the sender list in the formation (for filtering information when the slave machines receive data), and add the slave machine ID to the listener list (for the lead machine to confirm the sending object). Calculate the x, y, and z coordinates of each slave machine relative to the lead machine in the current formation, the formation mode, and the formation.

[0077] (3) Create a scene. Create a thread for each aircraft in the formation, assign it to the corresponding CPU ID and verify its availability. Set the aircraft model running interval and the number of times a single thread runs in a heartbeat. Check whether all threads have been initialized. Send the calculated information to the corresponding topic cache, including the local cache and reflection memory.

[0078] (4) Transfer the route information, virtual node ID number, longitude, latitude, altitude, field height relative to the ground, azimuth, speed, formation ID number, and formation mode to the aircraft model, and set the virtual node flight mode (automatic, guidance, etc.).

[0079] 6. After initialization, when the start heartbeat command is received via UDP: (taking a master-slave structure as an example)

[0080] (1) Upon receiving the heartbeat, run the aircraft model according to the simulation step size of the heartbeat (i.e. how long the model needs to calculate the aircraft changes this time), and output the virtual node ID number, longitude, latitude, altitude, x, y, z coordinates with the takeoff point as the origin, airspeed, velocity components in the x, y, and z directions, roll, pitch, yaw angle and radian, and offset distance from the waypoint line.

[0081] (2) Update the true latitude, longitude, altitude, current coordinate position, and speed information of the lead aircraft in the formation, output formation control commands (longitude, latitude, altitude, lead aircraft speed, and control surface deflection angle), and send formation communication information (list of slave aircraft IDs in the formation, the expected longitude, latitude, altitude, airspeed, speed components in the x, y, and z directions, and control surface deflection angle for each ID).

[0082] (3) The aircraft first obtains the current real latitude, longitude, altitude, current coordinate position, and speed information, and then flies according to the expected values ​​of the formation communication information sent by the lead aircraft.

[0083] (4) Upon receiving a formation change command, the system will assign the latest primary drone ID, the list of slave drone IDs, and the mission area based on the new mission requirements, the current position of each formation, and the type of each drone. The system will reset the relative positions of the primary drones within the formation, add the primary drone ID to the sender list (for filtering information when slaves receive data), and add the slave drone ID to the listener list (for the primary drone to confirm the recipient). The system will calculate the x, y, and z coordinates of each slave drone relative to the primary drone in the current formation, the formation mode, and the formation configuration. When drones that were previously in different formations form a new formation, nodes closer to the mission area will fly at a slower speed, while nodes farther away will use near-maximum speed to pursue.

[0084] (5) When a change formation command is received, the relative positions of each aircraft will be calculated according to the set corresponding formation function. The lead aircraft's position remains unchanged, while the slave aircraft change their speed and attitude according to their relative positions, and calculate the desired longitude, latitude, altitude, airspeed, speed components in the x, y, and z directions, and control surface deflection angle.

[0085] (6) When a change waypoint instruction is received, the original waypoint information is cleared, the longitude, latitude and altitude information of the overall waypoint is updated, more detailed waypoints are generated that can correspond to each step of long-thread calculation, the x, y and z coordinates of each slave aircraft relative to the lead aircraft in the current formation are calculated, and the expected longitude, latitude, altitude, airspeed, speed components in the x, y and z directions and control surface deflection angle are calculated.

[0086] (7) The completed computation information is then sent to the publish / subscribe list queue, where interested subscribers to this topic will receive it. Nodes distributed across different computers will read the content written to the reflected memory.

[0087] 7. The actual nodes interact with the MESH self-organizing network data link via transceiver modules mounted on the aircraft. The ground end of the data link is connected to the device via Ethernet. The computer controlling the ground station also needs to be connected to Ethernet and configured with IP addresses for two frequency bands: one the same as the data link frequency band (Class B address), and the other the same as the frequency band of other computers within the device (Class A address). Image information from the visible light / infrared pods is transmitted to the ground station computer via the data link. The ground station computer can also control the pods' angle, vertical and horizontal movement, magnification, etc., and send commands uplink to the data link. Image information is also forwarded to other computers within the device. Other computers subscribed to this information can process and analyze it to obtain the target's location, type, and confidence level, which the model uses to determine whether to adjust task allocation and formation reorganization based on the latest identification.

[0088] In one embodiment of the present invention, different drone models can be quickly connected to this device in a variety of ways.

[0089] a) Through a proxy computer:

[0090] The agent computer, acting as a node in the platform, is responsible for transmitting data from proprietary systems (such as embedded models) to the platform and vice versa, thus completing the interconnection between the proprietary systems and the platform. The agent node subscribes to data published by the models within the platform, decomposes it into a structured form, writes it to a direct connection (such as Ethernet), and then sends an interrupt notification to the proprietary system. Upon receiving the interrupt information, the proprietary system reads the data from the direct connection and processes it.

[0091] b) Through a proxy program:

[0092] After subscribing to data from the simulation platform, the agent program gains access to shared memory, parses the data into a structure, and writes it to shared memory after waiting for the object to be read and written to complete the operation and unlock. Subsequently, the agent program alerts the user that the model is reading data from shared memory by triggering a model read event. After gaining access to shared memory, the model waits for the object being read and written to complete the operation and unlock before proceeding with the data reading.

[0093] The way data is sent to the platform is that the model must wait for the objects being read from and written to complete and unlock before it can access shared memory and write the data to the shared memory as a structure. The model instructs the agent to read data from shared memory via events. After gaining access to shared memory and waiting for the objects to complete the read / write process and unlock, the agent reads the data, converts it to the platform's standard data format, and then publishes it to the platform.

[0094] c) Dynamic link library or source code method:

[0095] There are two types of data: message data and object data. Message data is defined as data that cannot persist in the simulation, while object data is persistent. For iterative simulation, the decorator retrieves the message and object data needed by the model from the reflective memory network and sends it to the model via the model manager. The decorator also broadcasts the object and message data to the reflective memory network so that other models can use them for iterative simulation.

[0096] Reflective memory networks are a branch of distributed shared memory, thus combining the advantages of both shared and distributed memory. They are defined as a distributed shared memory system based on automatically updated copies of remote shared memory. When shared data needs to be reused, all processors' local memory should retain an exact copy. Therefore, shared reads can always be satisfied from local memory. Reflective memory networks are sometimes called mirrored systems or replicated memory systems, but their use is more common. A key characteristic is that each computer physically has its own local memory, resulting in the appearance that all computers are connected to a large, shared memory. Reflective memory consists of physically distributed, logically mapped dual-port memory to a global shared address space. Reflective memory updates can occur on different types of interconnect networks: buses, bus hierarchies, ring networks, mesh networks, or crossover networks. Shared memory takes various forms: pages, words, segments, or blocks. They can also be dynamically or statically mapped to shared memory regions. Reflective memory offers several advantages, such as typically constant and deterministic memory access times, support for multi-reader / multi-writer algorithms, and good fault tolerance.

[0097] In one embodiment of the present invention, initialization, data reception, calculation, and data output are four steps that most simulation models must go through, and the calculation part needs to be encapsulated using an interface.

[0098] Given its versatility, a unified component-based model interface must be provided. The interface is divided into:

[0099] (1) Initialize the model interface, set the initial state of the model according to the input parameters, and allow users to modify the input parameters according to simulation requirements.

[0100] (2) In order to transmit the message of interest to the model to the model for simulation iteration, the message is sent to the data input model interface.

[0101] (3) In order to enable other models that need to use these messages to perform simulation iterations, the message output model interface function is responsible for publishing the messages that the model must publish.

[0102] (4) To retain object input, calculation and object output when calling simulation model, a calculation iteration interface is required.

[0103] In one embodiment of the present invention, the model launcher starts before the simulation begins and does not stop until the simulation is complete. It forwards some necessary data to the model on the node and receives the configuration file and model library provided to the node from the server. After receiving an interrupt event from the server, the model launcher reads the configuration file and model library from a specified location in shared memory and sends an interrupt event upon completion.

[0104] The decorator process is initiated by the model initiator using the model library and configuration files. Upon startup, each decorator loads the model library and requests shared memory space from the server via an interrupt event. The model initiator receives the callback address from the server, and then the decorator receives the address from the model initiator's callback function. Initiated models wait for a start command after sending an interrupt event to the main controller. When the server receives interrupt events from all models and issues a start command, each model enters simulation.

[0105] In one embodiment of the present invention, the model initiator receives interrupt events from the main controller throughout the simulation process and controls the model simulation iteration or sends notifications such as messages and objects to the model based on the type of interrupt event. When a message event occurs in the model, the model sends an interrupt event to the server. Then, the server forwards the corresponding interrupt event to the model initiator, which in turn notifies all models on the node that a message event has occurred, and the model performs the corresponding processing procedure.

[0106] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0107] In some embodiments of this example, a computer program product is provided, including a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0108] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component, for executing the methods described in the above embodiments.

[0109] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0110] Computer-readable storage media may also store at least one computer-executable program, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0111] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0112] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0113] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0114] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] Although the embodiments disclosed in this disclosure are as described above, the above content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope disclosed herein. However, the patent protection scope of this disclosure shall still be determined by the scope defined in the appended claims. The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A device for verifying the combination of virtual and real of a UAV, characterized in that, include: The central scheduling and management module is used to parse the scene configuration file, extract information from each element in the scene, and set the target parameters for each element. Configure the interaction relationships between the elements and distribute all elements to multiple node computers; The UDP distributed data transmission module is used to send and receive data packets and read relevant information based on the content of the data packets. Run the thread management module to manage the multiple computing threads enabled on a single computer, the drone nodes allocated to each thread, and the result processing after completing a single-step calculation; The UAV six-degree-of-freedom model calculation module is used to perform centralized calculations on the dynamic models of all simulated UAV nodes on each node computer to obtain the corresponding tasks of each UAV in the next step. The formation maintenance and change implementation module is used to reorganize each UAV formation based on its current position and the corresponding task. 2.The UAV virtual-real combined verification device according to claim 1, characterized in that, The drones include: digital drones, semi-physical drones, and physical drones. 3.The UAV virtual-real combined verification device according to claim 2, characterized in that, Also includes: The real-machine access module is used to verify whether the access scenario and algorithm are applicable to the real machine. At the same time, it acquires the status of the digital drone and the instructions from the ground end, transmits the current status and images of the physical drone and sends them to the computer so that the model can determine whether to adjust the task allocation and formation reorganization based on the current status and images of the physical drone. 4.The UAV virtual-real combined verification device according to claim 1, characterized in that, The UDP distributed data transmission module is also used for: Maintain the list of publishers and subscribers; Establish and maintain the mapping relationship between publishers and subscribers; The next step is to transmit the corresponding tasks of each drone to the publisher and subscriber list, based on the mapping relationship between the publisher and subscriber. 5.The UAV virtual-real combined verification device according to claim 1, characterized in that, Multiple node computers share computing data through a shared memory network. 6.The UAV virtual-real combined verification device according to claim 2, characterized in that, The central scheduling and management module is also used to switch drone nodes representing physical drones to adjust the formation and position of physical drones in the scene; and / or, drone nodes representing physical drones can coordinate with drone nodes representing virtual drones to perform tasks through coordinate transformation.

7. A method for verifying the combination of virtual and real of a UAV, characterized in that, include: The scene configuration file is parsed, information of each element in the scene is extracted, and target parameters of each element are set. Configure the interaction relationships between the elements and distribute all elements to multiple node computers; It receives and reads data packets, and manages the multiple computing threads enabled in a single computer and the drone nodes allocated to each thread based on the content of the read data packets, in order to complete the result processing after single-step calculation; Based on the processed single-step calculation results, the dynamic models of all simulated UAV nodes on each node computer are centrally calculated to obtain the corresponding tasks of each UAV in the next step. The next task for each drone is transmitted, and the drone formations are reorganized based on their current positions and the corresponding tasks. 8.The UAV virtual-real combined verification method of claim 7, wherein, Also includes: Verify whether the access scenario and algorithm are applicable to the actual machine, and at the same time obtain the status of the digital drone and the instructions from the ground, transmit the current status and image of the physical drone and send them to the computer so that the model can determine whether to adjust the task allocation and formation reorganization based on the current status and image of the physical drone.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 7 to 8.

10. A computer program product comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 7 to 8.

Images