Flying body tracking device, tracking system, and tracking method

Abstract

In a state in which sensor data acquisition of a flying body is difficult, this tracking device: estimates a flying body state without using sensor data by means of a sensor fusion filter which is a state estimation algorithm based on sensor data; estimates a flying body state by using a machine learning model; fuses the flying body state estimated by means of the sensor fusion filter and the flying body state estimated by using the machine learning model; and sets the flying body state after the fusion to be the resulting estimated flying body state.

Description

Aircraft tracking device, tracking system, and tracking method

[0001] The present invention relates to a tracking device, a tracking system, and a tracking method for tracking an aircraft.

[0002] Tracking of an aircraft is important in terms of operation, management, etc. Regarding tracking of an aircraft, for example, a technology disclosed in Patent Document 1 is known. This publication discloses a technology for estimating the local position of a target position from multiple modules that each process sensor data for navigation control of the aircraft.

[0003] US11,429,111

[0004] Tracking of an air vehicle generally depends on communication between the air vehicle and a ground sensor station. However, the air vehicle may be blocked from the ground sensor station due to reasons such as the presence of terrain, buildings, trees, or other obstacles between the air vehicle and the ground sensor station, resulting in a non-line-of-sight (NLOS) state, i.e., a state in which the air vehicle is out of sight of the ground sensor station. As a result, tracking of the air vehicle may be difficult.

[0005] Patent Document 1 does not take into consideration the difficulty in tracking an aircraft in an NLOS state caused by an obstacle, etc. The NLOS state is an example of a state in which it is difficult to acquire sensor data from an aircraft (for example, a state in which sensor data from an aircraft cannot be acquired).

[0006] The tracking device includes a communication interface device and a controller. The communication interface device communicates with the air vehicle and one or more sensors that detect the air vehicle. The controller tracks the air vehicle. The air vehicle tracking includes estimating the air vehicle state for each time. The air vehicle state includes at least one of the position, velocity, and attitude of the air vehicle. At a first target time, which is a time when it is difficult for the controller to acquire sensor data including measurements from one or more sensors via the communication interface device, or when it is difficult for the controller to acquire an air vehicle ID associated with the sensor data from the air vehicle via the communication interface device, the controller performs a first air vehicle state estimation according to a first tracking mode, which is a tracking mode when it is difficult to acquire sensor data from the air vehicle. The first air vehicle state estimation includes a first filter estimation process, a first model estimation process, and a first fusion process. The first filter estimation process includes estimating the air vehicle state using a sensor fusion filter, which is a predetermined algorithm for estimating the air vehicle state using state data based on sensor data. The first model estimation process includes estimating an aircraft state by inputting data representing an estimated aircraft state after fusion obtained at a time prior to the first target time into a first machine learning model that outputs data representing an estimated aircraft state and inputs data representing an aircraft state after fusion by the first fusion process, and outputting data representing the estimated flight state. The first fusion process includes fusing the flight states represented by the data representing the aircraft state estimated by the sensor fusion filter and the data representing the aircraft state estimated using the first machine learning model into a single aircraft state, and outputting data representing the fused aircraft state. The controller sets the aircraft state after fusion by the first fusion process as the aircraft state at the first target time.

[0007] The tracking accuracy of the flying object can be improved in a state where it is difficult to acquire sensor data from the flying object. Problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention.

[0008] 1 shows an example of UAV swarm flight; 2 shows an example of data input and output to the tracking module; 3 shows an example of the configuration of the processing performed by the tracking module; 4 shows an example of the configuration of the mode processing module; 5 shows an example of the ground truth flight path and predicted flight path of a UAV; 6 shows an example of a mesh-based communication network; 7 shows an example of UAV tracking processing; 8 shows an example of the configuration of a ground sensor station.

[0009] In the following description, a "communication interface apparatus" may refer to one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0010] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0011] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and more specifically, may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0012] Also, in the following description, "storage device" may be at least one of memory and persistent storage device.

[0013] In the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. At least one processor device may be a processor device in a broad sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0014] Furthermore, in the following description, functions may be described using expressions such as a control module and a tracking module. However, the functions may be implemented by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is implemented by a program executed by a processor, the specified processing is performed using a storage device and / or a communication interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may also be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a non-transitory storage medium). The description of each function is merely an example; multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0015] In the following description, when elements of the same type are described without distinction, common reference symbols are used, and when elements of the same type are described with distinction, reference symbols are used.

[0016] An embodiment will be described below. In the following embodiment, a UAV (Unmanned Aerial Vehicle) is an example of an air vehicle.

[0017] FIG. 1 shows an example of a UAV swarm flight.

[0018] Ground sensor station 103a is provided at point A, and ground sensor station 103b is provided at point B. UAVs 100a, 100b, and 100c that make up UAV swarm 50 fly from point A to point B along a predetermined trajectory line 101. Ground sensor station 103 may be located on or near the ground route from point A to point B (e.g., the path of trajectory line 101 in a planar view) instead of or in addition to at least one of point A and point B.

[0019] Each UAV 100 has one or more sensors, such as a camera or a Global Navigation Satellite System (GNSS) sensor. Each UAV 100 communicates with a ground sensor station 103, either via one or more other UAVs 100 in the UAV swarm. The ground sensor station 103 receives data from the UAVs 100.

[0020] 1 , the UAV swarm 50 performs a cooperative task at low altitude. At least one of the environmental elements, such as buildings 102 a and 102 b and forest 99, may obstruct UAV tracking. This may result in a non-linear out-of-sight (NLOS) state, resulting in reduced UAV tracking accuracy. However, in this embodiment, UAV tracking accuracy can be maintained even in the non-linear out-of-sight state.

[0021] Furthermore, when a strong wind blows, wireless communication between the UAV 100 and the ground sensor station 103 or between the UAVs 100 in the UAV swarm 50 may be temporarily interrupted. In other words, even if the state is a line-of-site (LOS) state, the accuracy of tracking the UAV may decrease. However, in this embodiment, the accuracy of tracking the UAV in the LOS state can be maintained.

[0022] In this manner, this embodiment achieves tracking that is robust to both relatively long-term occlusions by obstacles and relatively short-term disruptions by wind gusts.

[0023] As illustrated in FIG. 8 , the ground sensor station 103 includes a communication interface device 801 for communicating with the UAV 100, a sensor 802 (e.g., at least one of LiDAR, RADAR, and camera), and a controller 803. The communication interface device 801 may include a communication interface device for the sensor 802. The controller 803 is communicatively connected to the communication interface device 801 and may transmit and receive data through the communication interface device 801. The controller 803 may include, for example, a storage device and a processor. For example, functions such as a control module 810 and a tracking module 105 are realized by the processor executing a program stored in the storage device. The control module 810 controls the navigation and coordination of the UAV 100. The tracking module 105 tracks the UAV 100.

[0024] FIG. 2 shows an example of data input and output to the tracking module 105 .

[0025] The data input to the tracking module 105 includes planning data 108 , measurement data 107 , and wind data 115 .

[0026] The planning data 108 is data representing a flight plan or mission objectives that are predefined before flight in the ground sensor station 103, which is an example of a tracking system for the UAV 100. The planning data 108 may include, for example, a time series of control data for the UAV 100 for each UAV 100 (e.g., for each ID of the UAV 100). The control data may include data representing the position (e.g., latitude, longitude, and altitude), speed, attitude, and time.

[0027] The measurement data 107 includes data from multiple sources. The multiple sources may include at least one sensor onboard the UAV 100 and / or at least one sensor onboard the ground sensor station 103. The sensor onboard the UAV 100 and / or the ground sensor station 103 may be at least one of LiDAR, RADAR, and camera. The measurement data 107 includes data representing physical quantities such as the position and velocity of the UAV 100. For example, the measurement data 107 includes radio data 109 and detection data 110.

[0028] The wireless data 109 is data acquired from sensors mounted on the UAV 100 and / or sensors mounted on the ground sensor station 103. Cooperation between the UAVs 100 may be performed based on the wireless data 109.

[0029] The detection data 110 is data representing elements detected by the UAV 100. The elements may be objects that the UAV 100 is trying to identify or track, or may be obstacles that the UAV 100 is trying to avoid.

[0030] The wind data 115 is data representing detected or observed values ​​(physical quantities related to wind) such as wind speed and wind direction. In this embodiment, the source of the wind data 115 is an external source such as a weather station (e.g., a weather server) and / or a sensor mounted on the ground sensor station 103, but instead of or in addition to that, the source may be a sensor mounted on the UAV 100. In other words, the wind data 115 may be included in the measurement data 107.

[0031] Data output from tracking module 105 includes UAV status data 106. For each UAV 100, UAV status data 106 is data representing the status of the UAV 100 and may include, for example, data representing the position (e.g., latitude, longitude, and altitude), speed, and attitude of the UAV 100.

[0032] FIG. 3 shows an example of the configuration of the tracking module 105.

[0033] An example of a mission of the UAV swarm 50 is when multiple UAVs 100 cooperate to achieve a common objective, such as surveillance, reconnaissance, mapping, or search and rescue tasks. To facilitate coordination, communication, tracking, etc. between the UAVs 100, each UAV 100 is assigned a UAV 100 ID.

[0034] The tracking module 105 includes a switching module 116 and a mode processing module 300 .

[0035] The switching module 116 acquires sensor data (data including measurements from various sensors mounted on the UAV 100) from the measurement data 107 (S301). The sensor data may include a signature of the UAV 100 that is the source of the sensor data. The switching module 116 also receives a UAV_ID of each UAV 100 (S302). The signature of the UAV 100 may be associated with the UAV_ID of the UAV 100.

[0036] The switching module 116 detects the UAV 100 using the UAV signature of the sensor data acquired in S301 and the UAV ID received in S302 (S303). Note that, in addition to the sensor data and the UAV ID, the plan data 108 shown in FIG. 2 may also be used to detect the UAV in S303.

[0037] The switching module 116 calculates a Mahalanobis distance for each detected UAV 100 (S304). "Mahalanobis distance" is a statistical measure used to quantify the similarity or dissimilarity between two data sets (in this case, the signature of the detected UAV 100 and a reference distribution). By calculating the Mahalanobis distance, the switching module 116 evaluates deviations from expected or typical patterns in the sensor data for each UAV 100 and distinguishes between real UAVs 100 and potential false detections. After calculating the Mahalanobis distance, the switching module 116 applies weights to each detection output (e.g., output resulting from the detection in S303) (S305). These weights may be confidence levels assigned to each UAV 100 based on the Mahalanobis distance and other factors. UAVs 100 with shorter Mahalanobis distances or closer to the reference distribution may be assigned higher weights, i.e., higher confidence in their validity as actual UAVs within the UAV swarm 50.

[0038] After S305, the switching module 116 fuses the UAV data (sensor data of the detected UAVs) with the UAV_ID (S306). This fusion process creates a fusion of sensor data and UAV_ID for each detected UAV 100, or in other words, a comprehensive representation of the UAVs 50 in the UAV swarm 50.

[0039] Based on the fusion data (fusion of sensor data and UAV_ID) for each detected UAV 100, the switching module 116 detects the absence of a UAV_ID if there is one (S307). A "missing UAV_ID" refers to the case where sensor data was acquired in S301 but there is no UAV_ID to fuse with that sensor data, or the case where a UAV_ID was received in S302 but there is no sensor data to fuse with that UAV_ID. The switching module 116 selects either the LOS tracking mode or the NLOS tracking mode depending on whether there is a missing UAV_ID (S308). Specifically, for UAVs with no missing UAV_ID, the LOS tracking mode is selected because there is sensor data (sensor data fused with the UAV_ID) in the measurement data 107, and the mode processing module 300 tracks the UAV according to the LOS tracking mode. For UAVs with missing UAV_ID, there is no sensor data in the measurement data 107, so the NLOS tracking mode is selected and the mode processing module 300 tracks the UAV according to the NLOS tracking mode.

[0040] FIG. 4 shows an example of the configuration of the mode processing module 300.

[0041] The mode processing module 300 tracks the UAV according to the LOS tracking mode and the NLOS tracking mode. In this embodiment, the mode processing module 300 has components common to the LOS tracking mode and the NLOS tracking mode, components dedicated to the LOS tracking mode, and components dedicated to the NLOS tracking mode. Alternatively, the mode processing module 300 may have a physically or logically separated LOS tracking module and an NLOS tracking module. The LOS tracking module may track the UAV according to the LOS tracking mode, and the NLOS tracking module may track the UAV according to the NLOS tracking mode.

[0042] The mode processing module 300 includes a sensor fusion filter 112, deep learning modules 114a and 114b, and a sensor fusion architecture (SFA) 400. The sensor fusion filter 112 and the SFA 400 are components common to the LOS tracking mode and the NLOS tracking mode. The deep learning module 114a is a component dedicated to the LOS tracking mode. The deep learning module 114b is a component dedicated to the NLOS tracking mode.

[0043] The sensor fusion filter 112 combines data from various sensors, such as GNSS sensors, an IMU (inertial measurement unit), LiDAR, RADAR, and cameras, to estimate the state of the UAV (e.g., position, velocity, and other parameters (physical quantities)). The sensor fusion filter 112 can use techniques such as Kalman filtering, Bayesian inference, or particle filtering to fuse sensor data while taking into account the uncertainty, bias, or noise inherent in each sensor. Specifically, the sensor fusion filter 112 includes a state prediction module 401 and a state update module 402. For each UAV, the state prediction module 401 predicts (estimates) the UAV state at each time t and outputs state data representing the predicted UAV state. The state update module 402 updates the UAV state at each time t based on the state data representing the predicted UAV state at time t and outputs state data representing the updated UAV state. The sensor fusion filter 112 may be a Kalman filter (e.g., an Extended Kalman Filter (EKF) or an Unscented Kalman Filter (UKF)). The Kalman gain may be based on the reliability of each sensor as a data source.

[0044] The deep learning module 114 uses a trained deep learning model to predict (estimate) the UAV state. The deep learning model may be, for example, a neural network model (e.g., a Long Short Term Memory (LSTM) model). The deep learning module 114 can function as a time memory when data regarding the UAV's motion patterns or behavior cannot be identified. The deep learning model used by the deep learning module 114 to predict the likely future state of the UAV can be trained from historical data. The deep learning model is trained using a training dataset of historical weather measurements (e.g., wind speed, wind direction, etc.) collected from various sources, such as weather stations, weather sensors, or numerical weather prediction models. The training dataset may be prepared based on various scenarios. Because the deep learning module 114 has been pre-trained using such a training dataset, it can effectively predict and mitigate the effects of wind gusts and other disturbances on UAV tracking.

[0045] Specifically, for the deep learning model (hereinafter referred to as Model 1) used by the deep learning module 114a, the input data are the measurement data 107, the wind data 115, and the state data after fusion by the SFA 400, and the output data is the predicted state data. Model 1 may be trained using a training dataset including multiple pairs of these input data and output data (typically ground truth data). Model 1 is preferably trained using a training dataset that follows a scenario in which the LOS tracking mode is applied.

[0046] Unlike the input data of Model 1, the deep learning model (hereinafter referred to as Model 2) used by the deep learning module 114b does not have measurement data 107 as input data. This is because the deep learning module 114b is dedicated to the NLOS tracking mode. That is, for Model 2, the input data is wind data 115 and state data after fusion by SFA 400, and the output data is predicted state data. Model 2 may be trained using a training dataset including multiple pairs of these input data and output data (typically ground truth data). Model 2 is preferably trained using a training dataset that conforms to a scenario in which the NLOS tracking mode is applied.

[0047] Although the deep learning module 114a is not necessary in the LOS tracking mode, it is preferable to have the deep learning module 114a because, although the sensor fusion filter 112, which is a Kalman filter, cannot accurately predict the UAV state in a windy environment, the training data set used in learning Model 1 (and Model 2) includes wind data such as strong winds, and therefore can contribute to improving the prediction accuracy of the UAV state in a windy environment (especially an environment with strong winds).

[0048] The SFA 400 fuses the state data output from the sensor fusion filter 112 with the state data output from the deep learning module 114a or 114b, and outputs the fused state data. The fused state data is the UAV state data 106 output from the tracking module 105.

[0049] The processing in each of the LOS tracking mode and NLOS tracking mode will be described below.

[0050] Examples of LOS tracking modes are as follows:

[0051] The state update module 402 updates (estimates) the UAV state at time t based on the input measurement data 107 (e.g., data such as the UAV's speed and acceleration) and the state data for time t from the state prediction module 401, and outputs state data representing the updated UAV state. In the initial update (estimation), the state data for time t from the state prediction module 401 may be predetermined data. The state data from the state update module 402 is fed back to the state prediction module 401. The state prediction module 401 uses the fed-back state data as state data for time (t-1), predicts the UAV state for the next time t based on the fed-back state data for time (t-1), and outputs state data representing the predicted UAV state. The measurement data 107 may be input to the state prediction module 401, and the measurement data 107 may be used for prediction by the state prediction module 401.

[0052] The deep learning module 114a inputs the state data for time t fed back from the SFA 400, the acquired wind data 115, and the measurement data 107 into the model 1, and outputs state data representing the predicted UAV state for time t. In the initial prediction, the state data for time t fed back from the SFA 400 may be predetermined data.

[0053] The SFA 400 fuses the state data for time t output from the state update module 402 with the state data for time t output from the deep learning module 114a, thereby outputting fused state data for time t. This fused state data is fed back to the deep learning module 114a. This fused state data may be used as state data for time (t-1) to predict the next time t.

[0054] The NLOS tracking modes are, for example:

[0055] The NLOS tracking mode is selected when communication between the UAV 100 and the ground sensor station 103 is blocked by a gust of wind, an obstacle, or the like, and the ground sensor station 103 is unable to acquire the measurement data 107. In the NLOS tracking mode, for example, the sensor fusion filter 112 predicts the state of the UAV 100 based on a constant velocity pattern. That is, the state data from the sensor fusion filter 112 at the start time t of the NLOS tracking mode may be predetermined data based on the assumption that the motion of the UAV 100 follows a consistent velocity over time. The deep learning module 114b inputs the wind data 115 and data (e.g., predetermined data) from the SFA 400 into Model 2, thereby outputting state data representing the predicted UAV state at time t. The SFA 400 fuses the state data output from the sensor fusion filter 112 and the state data output from the deep learning module 114b to output fused state data for time t.

[0056] State data for time t is fed back from the deep learning module 114b to the state prediction module 401 of the sensor fusion filter 112 as state data for time (t-1). This integrates the state data (estimated value) of the deep learning module 114b into the prediction by the state prediction module 401, allowing the sensor fusion filter 112 to consider complex, nonlinear dynamics in the UAV's motion and environmental factors, resulting in more accurate and robust state estimation. That is, the state prediction module 401 predicts the UAV state at the next time t using the state data for time (t-1) from the deep learning module 114b and the state data for time (t-1) from the state update module 402, and outputs state data for time t. The state update module 402 outputs the state data for time t from the state prediction module 401 as is (or after some processing). This output state data is fed back to the state prediction module 401 and input to the SFA 400. The SFA 400 combines and outputs state data from the state update module 402 and the deep learning module 114b for time t.

[0057] The SFA 400 may be designed to maximize the overall accuracy and reliability of the UAV state prediction (estimation). Specifically, the output data from two sources, the deep learning module 114a or 114b and the sensor fusion filter 112, may be fused, with weights of the output data set based on the reliability or uncertainty of each of the two sources. The fusion may be dynamically adjusted depending on whether the NLOS tracking mode or the LOS tracking mode is being implemented. A Bayesian framework may be used to probabilistically combine the output data from the two sources. More specifically, for example, in the LOS tracking mode, the W Ｄ (weight of the output data of the deep learning module 114a) and W Ｓ (the weights of the output data of the sensor fusion filter 112) may be equal (e.g., W Ｄ :W Ｓ On the other hand, in NLOS tracking mode, W Ｄ Is W Ｓ may be higher than (for example, W Ｄ :W Ｓ =80:20).

[0058] In NLOS tracking mode, feedback of state data from the deep learning module 114b to the state prediction module 401 can contribute to modifying the state transition matrix of the state prediction of the sensor fusion filter 112 and updating physical quantities such as velocity to perform mathematical calculations of the state prediction.

[0059] Specifically, for example, the state prediction module 401 ｔ =F ｔ－１ *x ｔ－１ +w ｔ Using the state equation, the UAV state x at time t ｔ You can predict x ｔ－１ may be the UAV state at time t−1 predicted by the sensor fusion filter 112 (state prediction module 401). ｔ may be a noise or disturbance at time t. ｔ－１may be a state transition matrix at time t-1. An example of the state transition matrix may be as follows. In the matrix below, v may be velocity and a may be acceleration. F ｔ－１ =np. array([ [1,v,a,0,0,0,0,0,0], [0,1,v,0,0,0,0,0,0], [0,o,0,0,0,0,0,0,0], [0,0,0,1,v,a,0,0,0], [0,0,0,0,1,v,0,0,0], [0,0,0,0,o,0,0,0,0], [0,0,0,0,0,0,1,v,a], [0,0,0,0,0,0,0,1,v], [0,0,0,0,0,0,0,o,0] ]) v=delta_t a=p*(delta_t) ２ (p is an arbitrary value) o = -q / (delta_t) (q is an arbitrary value)

[0060] The state update module 402 ｔ = H ｔ *x ｔ +b ｔ Using the observation equation, the UAV state Z at time t is ｔ You can find H ｔ may be the observation matrix at time t. ｔ may be the predicted UAV state at time t. ｔ may be a noise vector.

[0061] In NLOS tracking mode, there is no measurement data 107, so F ｔ－１ In this case, it is not possible to obtain values ​​to be set as the parameters v and a in the state equation, and the state prediction module 401 is configured to calculate the state equation by setting predetermined values ​​as the parameters v and a. Therefore, the values ​​of the parameters v and a differ from the values ​​when the measurement data 107 is available. The state data output from the deep learning module 114b may include the values ​​of v and a, and the state data including the values ​​of v and a is fed back to the state prediction module 401. The values ​​of v and a in the state data are applied to the state transition matrix, which is expected to improve the accuracy of prediction by the state prediction module 401.

[0062] 2 may be used in the processing by the mode processing module 300. For example, the teacher data set used for learning the models 1 and 2 may further include the planning data as input data, and the planning data 108 may be input in the processing by the deep learning module 114a or 114b (inference using the model 1 or the model 2).

[0063] FIG. 5 shows an example of the ground truth flight path and predicted flight path of the UAV 100a.

[0064] The UAV 100a needs to track a ground truth path 124 from point A to point B. However, at point C, an obstacle blocks communication between the UAV 100a and the ground sensor station 103, and the ground sensor station 103 cannot use the measurement data 107, including the sensor data of the UAV 100a, for tracking. As a result, the state data predicted by the sensor fusion filter 112 from point C onward becomes unreliable, and therefore the predicted flight path 125 according to the time series of the state data deviates from the ground truth path 124.

[0065] To address the limitations of the sensor fusion filter 112 in occluded regions (regions encompassing the range of the path from point A to point B), which are regions where there are obstacles between the UAV 100a and the ground sensor station 103, the tracking module 105 tracks the UAV according to the NLOS tracking mode. That is, the tracking module 105 integrates inference processing by the deep learning module 114b into the tracking processing. By integrating the output data (inference results) of the deep learning module 114b with the output data (predicted values) of the sensor fusion filter 112, the tracking module 105 can predict that the predicted flight path 126 of the UAV 100a will be closer to the ground truth path 124 than the flight path 125, even in occluded regions where measurement data 107 including sensor data of the UAV 100a is unavailable.

[0066] FIG. 6 shows an example of a mesh-based communication network.

[0067] Each of the UAVs 100a, 100b, and 100c includes a mesh grid 128. The mesh grid 128 functions as a network infrastructure. While there is an obstacle between the UAV 100 and the ground sensor station 103 (while the state is NLOS), a communication link is established between the UAVs 100 (between the mesh grids 128), and messages can be exchanged between the UAVs 100.

[0068] Mesh networking within the UAV swarm 50 (communication over a mesh-based communication network) can reduce the likelihood of a UAV 100 going missing (missing UAV ID) in harsh environments where line-of-sight communication is limited or unreliable. Mesh networking provides redundancy, adaptability, or coordination that enhances the robustness and effectiveness of the UAV swarm 50, preventing communication failures or obstacles from causing a missing or unresponsive UAV. For example, if UAV 100a in the UAV swarm 50 experiences communication problems due to wind interference, a nearby UAV 100b can relay messages or signals from UAV 100a. The UAV swarm 50 can dynamically reconfigure its network topology to maintain communication continuity.

[0069] FIG. 7 shows an example of a UAV tracking process.

[0070] Measurement data 107 is received for each UAV from ground sensor station 103a or 103b and may include position, velocity, and other physical quantities. Tracking module 105 is implemented in ground sensor station 103, but may also be implemented in a computing system external to ground sensor station 103.

[0071] When the tracking module 105 receives the measurement data 107, the tracking module 105 is initialized to start the tracking process (S701). This initialization sets the tracking module 105 up to efficiently process the received data. The tracking module 105 may be initialized in advance, for example, when the power is turned on.

[0072] The tracking module 105 obtains the initial location of each UAV along with its UAV ID (S702), which serves as an individual label for each UAV, facilitating individual tracking and monitoring.

[0073] The tracking module 105 determines (S703) whether the number of UAV_IDs currently being tracked is less than the total number of UAVs 100 present in the UAV swarm 50. The total number of UAVs 100 may be determined from the planning data 108. The planning data 108 may include data representing the total number of UAVs 100 present in the UAV swarm 50.

[0074] If the number of tracked UAV_IDs is less than the total number of UAVs 100 (S703: YES), the tracking module 105 identifies missing UAV_IDs (S706). The tracking module 105 activates NLOS tracking mode for the missing UAV_IDs (S707). The tracking module 105 monitors the duration of the NLOS tracking mode. If the duration of the NLOS tracking mode is longer than a time threshold T (S708: YES), the tracking module 105 performs data offloading and alerts (e.g., communication alerts and / or alerts indicating that the missing UAV_IDs cannot be found) to all ground sensor stations 103 (S709). For example, in S709, a mesh network may be established as a robust and resilient network for data exchange and coordination in response to a UAV being tracked by a ground sensor station 103 receiving a predetermined signal from the ground sensor station 103.

[0075] On the other hand, if the number of tracked UAV_IDs exceeds the total number of UAVs 100 (S703: NO and S704: YES), the tracking module 105 implements redundancy measures to maintain tracking accuracy and reliability (S705). Redundancy measures are introduced to mitigate potential tracking errors or inconsistencies and ensure robust performance in the face of challenges such as sensor malfunctions or communication outages.

[0076] If the number of tracked UAV_IDs matches the total number of UAVs 100 (S703: NO and S704: NO), the tracking module 105 activates the LOS tracking mode (S710).

[0077] If the duration of the NLOS tracking mode ends before the time threshold T (S708: NO), or if the duration of the LOS tracking mode ends, the tracking module 105 evaluates the tracking quality (S711). In S711, the quality (typically accuracy) of the predicted UAV state (such as the UAV's position, velocity, or attitude) is evaluated.

[0078] Although one embodiment has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms.

[0079] The above description can be summarized as follows: The following summary may include supplementary explanations and explanations of variations of the above description.

[0080] The tracking device includes an interface device and a controller that tracks the air vehicle. An example of the tracking device is the ground sensor station 103, but instead, the tracking device may be a computer system (e.g., a server system) that can communicate with all of the ground sensor stations 103a and 103b. The tracking device may be mounted on the air vehicle. An example of the communication interface device may be the interface device 801, and an example of the controller may be the controller 803. An example of the air vehicle is the UAV 100, but the air vehicle may be an air vehicle other than the UAV 100, such as an eVTOL (Electric Vertical Take-off and Landing). Furthermore, a tracking system may be constructed that includes a tracking device that estimates the state of the air vehicle and one or more sensors that detect the air vehicle.

[0081] "Aircraft tracking" is a process that includes estimating the state of an aircraft at each time. The "aircraft state" includes at least one of the position, velocity, and attitude of the aircraft.

[0082] For a first time of interest, the controller performs a first vehicle state estimation according to a first tracking mode.

[0083] Here, the "first target time" is a time identified by the controller, specifically a time at which it is difficult (e.g., not possible) for the controller to obtain sensor data including measurement values ​​from one or more sensors that detect the flying object via a communication interface device, or a time at which it is difficult (e.g., not possible) for the controller to obtain an flying object ID associated with the sensor data from the flying object via a communication interface device.

[0084] The "first append mode" is a tracking mode when it is difficult (for example, impossible) to acquire sensor data from the flying object, and an example is a NLOS tracking mode.

[0085] Furthermore, the "one or more sensors detecting the air vehicle" may include at least one of LiDAR, RADAR, and a camera. All or part of the "one or more sensors" may be located on the air vehicle, or all or part of the "one or more sensors" may be provided in the tracking device. Therefore, the sensor serving as the source of the measurement values ​​included in the "sensor data" may be a sensor mounted on the air vehicle or a sensor mounted on the tracking device. An example of sensor data may be the measurement data 107 or part of the data included in the measurement data 107. Furthermore, an example of "difficulty in obtaining an air vehicle ID from the air vehicle" may be, for example, the inability to actually obtain an air vehicle ID (e.g., a list of air vehicle IDs of all air vehicles constituting an air vehicle swarm) identified from ground truth data (e.g., planning data 108) from the air vehicle.

[0086] The first air vehicle state estimation is a process included in air vehicle tracking, and includes a first filter estimation process, a first model estimation process, and a first fusion process.

[0087] The first filter estimation process includes estimating the vehicle state (specifically, estimating without sensor data) using a sensor fusion filter, which is a predetermined algorithm for estimating the vehicle state using state data based on sensor data. An example of a sensor fusion filter is sensor fusion filter 112.

[0088] The first model estimation process includes estimating the aircraft state by inputting data representing the fused aircraft state obtained at a time earlier than the first target time (e.g., one time before the first target time) into a first machine learning model that outputs data representing the estimated aircraft state and inputs data representing the aircraft state after fusion by the first fusion process, and outputting data representing the estimated aircraft state. An example of the first machine learning model is a deep learning model (model 2). A model other than a deep learning model may be adopted as the first machine learning model. The first machine learning model may be a model that has been trained using a dataset including the above-mentioned input and output pairs.

[0089] The first fusion process may include fusing the aircraft states represented by the first and second data into a single aircraft state based on first data representing the aircraft state estimated by the sensor fusion filter and second data representing the aircraft state estimated using the first machine learning model, and outputting data representing the fused aircraft state. The first fusion process may be performed, for example, by the SFA 400. The controller sets the fused aircraft state represented by the data obtained in the first fusion process as the aircraft state at the first target time.

[0090] Since the sensor fusion filter, which is configured to estimate the state of the aircraft based on sensor data, estimates the state of the aircraft without sensor data, the accuracy of the estimation by the sensor fusion filter decreases. However, this decrease in estimation accuracy is compensated for by the estimation using the first machine learning model. This makes it possible to improve the tracking accuracy of the aircraft in situations where it is difficult to obtain sensor data from the aircraft. As a result, it is possible to achieve robust aircraft tracking.

[0091] In the first fusion process, the controller may fuse the aircraft states represented by the first and second data into a single aircraft state based on the weight of the first data and the weight of the second data. In the first fusion process, the weight of the second data may be higher than the weight of the first data. In other words, the aircraft state after fusion may reflect more of the aircraft state estimated using the first machine learning model. This is expected to improve the accuracy of aircraft state estimation in situations where it is difficult to obtain sensor data from the aircraft.

[0092] Some of the state data based on sensor data may be data to which data obtained from the sensor data is applied. In other words, the accuracy of the some data is low due to the absence of sensor data. State data representing the aircraft state estimated in the first model estimation process may be fed back to the sensor fusion filter. In the first filter estimation process, the controller may determine the some data based on the state data fed back in the first model estimation process for a time earlier than the first target time (e.g., the time immediately preceding), and estimate the aircraft state using the state data including the determined some data. This improves the estimation accuracy of the sensor fusion filter, thereby further improving the accuracy of the aircraft state after fusion. Specifically, for example, the sensor fusion filter may be a Kalman filter, and the state data may be a state transition matrix including a value of the aircraft's velocity (e.g., the above-mentioned v). The aircraft state may include the aircraft's velocity. The some data may include a velocity value in the state transition matrix. As described above, the state transition matrix may include the aircraft's acceleration (e.g., the above-mentioned a) in addition to the aircraft's velocity. The acceleration of the aircraft may be included in the aircraft state or may be estimated from a time series of the aircraft state including the position and velocity of the aircraft.

[0093] The controller may issue an alert indicating that the air vehicle of the air vehicle ID is undetectable if the continuous time during which the first tracking mode is applied to the air vehicle ID exceeds a certain period of time. This allows measures to be taken to mitigate risks (e.g., the risk of the undetectable air vehicle colliding with an environmental element such as a building or with other air vehicles in the air vehicle swarm that includes the air vehicle) based on the alert. For example, the alert may be notified to all ground sensor stations that communicate with the air vehicle. This further increases the likelihood of promptly taking measures to mitigate the risk. The controller may time the duration of the first tracking mode on an air vehicle swarm basis (e.g., if the first tracking mode for one air vehicle ends but the first tracking mode for another air vehicle continues, the first tracking mode may be considered to continue), or may time the duration for each air vehicle.

[0094] When the continuous time during which the first tracking mode is applied to the air vehicle ID exceeds a certain period of time, the controller may transmit a request to construct a mesh network (e.g., the network described with reference to FIG. 6 ) in the air vehicle swarm consisting of multiple air vehicles, including the air vehicle with the air vehicle ID, to the air vehicle corresponding to the acquired air vehicle ID (i.e., the air vehicle that maintains communication with the tracking system). The air vehicle that receives this request may cooperate with other air vehicles to construct a mesh network. This allows the tracking system to communicate with the air vehicle through cooperation between the air vehicles, and is expected to result in an early termination of the first tracking mode thereafter. Note that the mesh network may be constructed continuously or periodically, regardless of whether the continuous time during which the first tracking mode is applied to the air vehicle ID exceeds a certain period of time.

[0095] The input of the first machine learning model may include wind data including data representing wind speed in addition to the state of the flying object after fusion. An example of the wind data may be wind data 115. The wind data may be forecast data such as weather data, or observation data observed by a sensor or the like. In the first model estimation process, the controller may estimate the state of the flying object by inputting wind data at the first target time in addition to the state of the flying object after fusion obtained at a time prior to the first target time into the first machine learning model. This is expected to improve tracking accuracy. Specifically, for example, by training the first machine learning model using wind data representing a gust of wind strong enough to apply the first tracking mode, it is expected to improve tracking accuracy in the first tracking mode.

[0096] For a second time of interest, the controller may perform a second vehicle state estimation according to a second tracking mode.

[0097] Here, the "second target time" is a time identified by the controller, specifically, the time at which the controller is able to acquire sensor data via the communication interface device and is able to acquire the aircraft ID associated with the sensor data from the aircraft via the communication interface device.

[0098] The "second tracking mode" is a tracking mode in which sensor data of the aircraft can be acquired, and an example thereof may be a line-of-sight (LOS) tracking mode.

[0099] The second air vehicle state estimation may be a process included in air vehicle tracking. The second air vehicle state estimation may include a second filter estimation process, a second model estimation process, and a second fusion process. The second filter estimation process may include estimating the air vehicle state based on sensor data using a sensor fusion filter. The second model estimation process may include estimating the air vehicle state by inputting data representing the fused air vehicle state obtained at a time prior to the second target time (e.g., the immediately preceding time) and the sensor data at that time into a second machine learning model that outputs data representing the estimated air vehicle state and inputs data representing the air vehicle state after the fusion process and the sensor data, thereby estimating the air vehicle state and outputting data representing the estimated air vehicle state. The second fusion process may include fusing the air vehicle states represented by the third and fourth data representing the air vehicle state estimated by the sensor fusion filter and the fourth data representing the air vehicle state estimated using the second machine learning model into a single air vehicle state, and outputting data representing the fused air vehicle state. The controller may set the fused air vehicle state represented by the data obtained in the second fusion process as the air vehicle state at the second target time. This improves the accuracy of air vehicle state estimation in the second tracking mode. Note that in the second fusion process, the controller may fuse the air vehicle states represented by the third and fourth data into a single air vehicle state based on the weight of the third data and the weight of the fourth data, and in the second fusion process, the weight of the air vehicle state estimated by the sensor fusion filter may be the same as the weight of the air vehicle state estimated using the second machine learning model.

[0100] The input of the second machine learning model may include wind data including data representing wind speed, in addition to the fused aircraft state and sensor data. In the second model estimation process, the controller may estimate the aircraft state by inputting the wind data and sensor data at the second target time, in addition to the fused aircraft state obtained at a time prior to the second target time, to the second machine learning model. Although wind can affect the tracking accuracy of the aircraft, even if the data on which the sensor fusion filter estimates does not include wind data, improved accuracy of the aircraft state estimation in the second tracking mode is expected.

[0101] Since the air vehicle states estimated using the machine learning model are fused in the second tracking mode as well as the first tracking mode, the tracking device can be expressed, for example, as follows. That is, the tracking device includes a communication interface device and a controller that tracks the air vehicle. In tracking the air vehicle, the controller performs air vehicle state estimation for each time. The air vehicle state estimation includes a filter estimation process, a model estimation process, and a fusion process. The filter estimation process includes estimating the air vehicle state using a sensor fusion filter, which is a predetermined algorithm for estimating the air vehicle state. The model estimation process includes estimating the air vehicle state by inputting data representing the fused air vehicle state obtained at a time prior to the current time into a machine learning model that outputs data representing the estimated air vehicle state and inputs data representing the air vehicle state after fusion by the fusion process, thereby estimating the air vehicle state and outputting data representing the estimated air vehicle state. The fusion process includes fusing the air vehicle states represented by data A and B into a single air vehicle state based on data A representing the air vehicle state estimated by the sensor fusion filter and data B representing the air vehicle state estimated using the machine learning model, and outputting data representing the fused air vehicle state. The controller sets the aircraft state after fusion represented by the data obtained in the fusion process as the aircraft state at that time.

[0102] 100: UAV 103: Ground sensor station

Claims

1. A system comprising: a communication interface device; and a controller that tracks an air vehicle, wherein the communication interface device communicates with the air vehicle and one or more sensors that detect the air vehicle, wherein the air vehicle tracking includes estimating the air vehicle state for each time, wherein the air vehicle state includes at least one of the position, velocity, and attitude of the air vehicle, and at a first target time that is a time when it is difficult for the controller to acquire sensor data including measurements of the one or more sensors via the communication interface device, or when it is difficult for the controller to acquire an air vehicle ID associated with the sensor data from the air vehicle via the communication interface device, the controller performs a first air vehicle state estimation in accordance with a first tracking mode that is a tracking mode when it is difficult to acquire sensor data of the air vehicle, and the first air vehicle state estimation includes a first filter estimation process, a first model estimation process, and a first fusion process, and the first filter estimation process includes estimating the air vehicle state using a sensor fusion filter that is a predetermined algorithm for estimating the air vehicle state using state data based on the sensor data, A tracking device in which the first model estimation process includes estimating the aircraft state by inputting data representing the fused aircraft state obtained at a time prior to the first target time into a first machine learning model that outputs data representing an estimated aircraft state and inputs data representing the aircraft state after fusion by the first fusion process, and outputting data representing the estimated aircraft state; the first fusion process includes fusing the aircraft states represented by the first and second data into one aircraft state based on first data representing the aircraft state estimated by the sensor fusion filter and second data representing the aircraft state estimated using the first machine learning model, and outputting data representing the fused aircraft state; and the controller sets the aircraft state after fusion by the first fusion process as the aircraft state at the first target time.

2. The tracking device described in claim 1, wherein in the first fusion process, the controller fuses the aircraft states represented by the first and second data into one aircraft state based on the weight of the first data and the weight of the second data, and in the first fusion process, the weight of the second data is higher than the weight of the first data.

3. A tracking device as described in claim 1, wherein a portion of the data in the status data is data to which data obtained from the sensor data is applied, data representing the aircraft status estimated in the first model estimation process is fed back to the sensor fusion filter, and in the first filter estimation process, the controller determines the portion of data based on the fed-back data of the first model estimation process for a time prior to the first target time, and estimates the aircraft status using the sensor fusion filter using status data including the determined portion of data.

4. The tracking device of claim 3, wherein the sensor fusion filter is a Kalman filter, the state data is a state transition matrix including values ​​of the velocity of the aircraft, the aircraft state includes the velocity of the aircraft, and the portion of the data includes values ​​of the velocity in the state transition matrix.

5. The tracking device of claim 1, wherein the controller issues an alert indicating that the flying object of the flying object ID cannot be detected if the continuous time during which the first tracking mode is applied to the flying object ID exceeds a certain period of time.

6. The tracking device of claim 5, wherein the alert is communicated to all ground sensor stations in communication with the air vehicle.

7. The tracking device described in claim 1, wherein, when the continuous time during which the first tracking mode is applied to the aircraft ID exceeds a certain period of time, the controller sends a request to construct a mesh network in an aircraft swarm consisting of multiple aircraft including the aircraft with the aircraft ID to the aircraft in the aircraft swarm corresponding to the acquired aircraft ID.

8. The tracking device described in claim 1, wherein the input of the first machine learning model includes wind data including data representing wind speed in addition to the aircraft state after fusion, and in the first model estimation process, the controller estimates the aircraft state by inputting wind data at the first target time in addition to data representing the aircraft state after fusion obtained at a time earlier than the first target time into the first machine learning model.

9. With respect to a second target time, which is a time when the controller can acquire the sensor data via the communication interface device and can acquire an air vehicle ID associated with the sensor data from the air vehicle via the communication interface device, the controller performs a second air vehicle state estimation according to a second tracking mode, which is a tracking mode in which sensor data of the air vehicle can be acquired, and the second air vehicle state estimation includes a second filter estimation process, a second model estimation process, and a second fusion process, and the second filter estimation process includes estimating the air vehicle state based on the sensor data by the sensor fusion filter, and the second model estimation process includes estimating the air vehicle state by inputting data representing the air vehicle state after fusion obtained at a time earlier than the second target time and the sensor data at the second target time into a second machine learning model, which outputs data representing the estimated air vehicle state and inputs data representing the air vehicle state after fusion by the second fusion process and sensor data, and outputting data representing the estimated air vehicle state, The tracking device of claim 1, wherein the second fusion process includes fusing the aircraft states represented by the third and fourth data into one aircraft state based on third data representing the aircraft state estimated by the sensor fusion filter and fourth data representing the aircraft state estimated using the second machine learning model, and outputting data representing the aircraft state after the fusion, and the controller sets the aircraft state after the fusion by the second fusion process as the aircraft state at the second target time.

10. The tracking device described in claim 9, wherein in the second fusion process, the aircraft states represented by the third and fourth data are fused into one aircraft state based on the weight of the third data and the weight of the fourth data, and in the second fusion process, the weight of the third data and the weight of the fourth data are the same.

11. The tracking device described in claim 9, wherein the input of the second machine learning model includes wind data including data representing wind speed in addition to the fused aircraft state and sensor data, and in the second model estimation process, the controller estimates the aircraft state by inputting the wind data and sensor data at the second target time in addition to the fused aircraft state obtained at a time prior to the second target time into the second machine learning model.

12. The tracking device of claim 1, which is a ground sensor station having at least one of said one or more sensors and communicating with said air vehicle, or a computer system in communication with all of said ground sensor stations.

13. A tracking system comprising the tracking device according to claim 1 and one or more sensors for detecting the flying object.

14. A first air vehicle state estimation is performed by a computer in accordance with a first tracking mode, which is a tracking mode when it is difficult to acquire sensor data including measurements from one or more sensors that detect the air vehicle, or when it is difficult to acquire an air vehicle ID associated with the sensor data from the air vehicle, wherein the first air vehicle state estimation is included in air vehicle tracking including estimation of the air vehicle state for each time, and the air vehicle state includes at least one of the position, velocity, and attitude of the air vehicle, and the first air vehicle state estimation includes a first filter estimation process, a first model estimation process, and a first fusion process, and the first filter estimation process includes estimating the air vehicle state using a sensor fusion filter, which is a predetermined algorithm for estimating the air vehicle state using state data based on the sensor data, A tracking method in which the first model estimation process includes estimating the aircraft state by inputting data representing the fused aircraft state obtained at a time prior to the first target time into a first machine learning model that outputs data representing an estimated aircraft state and inputs data representing the aircraft state after fusion by the first fusion process, and outputting data representing the estimated aircraft state; the first fusion process includes fusing the aircraft states represented by each piece of data based on the data representing the aircraft state estimated by the sensor fusion filter and the data representing the aircraft state estimated using the first machine learning model into one aircraft state, and outputting data representing the fused aircraft state; and the aircraft state after fusion by the first fusion process is the aircraft state at the first target time.

Images