Flying object recognition system, sensing data processing device, and flying object recognition method

Abstract

To provide a flying object recognition system which improves recognition accuracy of a flying object.SOLUTION: In a flying object recognition system comprising a plurality of ground sensing devices installed on the ground and a sensing data processing device for processing data transmitted from the plurality of ground sensing devices, the ground sensing device includes a sensor which observes a flying object. The sensing data processing device includes: a data reception section for receiving sensor data transmitted by the plurality of ground sensing devices; a storage section for storing the sensor data received by the data reception section; a flight track calculation section for calculating a flight track of the flying object within a predetermined time using the sensor data stored by the storage section; and a flying object identification section for determining whether the flying objects observed by the plurality of ground sensing devices are the same flying object using the flight track of the flying object calculated by the flight track calculation section.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an aircraft recognition system for recognizing an aircraft, a sensing data processing device, and an aircraft recognition method.

Background Art

[0002] When implementing an aircraft such as an airplane as a transportation and logistics infrastructure in society, it is necessary to control a large number of aircraft safely and efficiently. In particular, since the area around the takeoff and landing ports is a place where aircraft are concentrated, it is important to ensure the safe flight of the aircraft by installing sensors on the ground or using the self-position information transmitted by the aircraft.

[0003] However, since a single sensor cannot cover the area around the port, it is necessary to install a plurality of sensors. At this time, when the same object is detected by a plurality of ground sensors, if they cannot be determined to be the same, the detection results overlap, and there is a problem that the true situation cannot be grasped.

[0004] For example, Patent Document 1 discloses an object tracking device that tracks a plurality of objects in a plurality of videos captured by a plurality of imaging devices. For each imaging device, from a frame image included in the video captured by the imaging device, a detection unit that detects a tracklet, which is a series of information indicating the presence of an object in the frame image, a feature amount extraction unit that extracts a predetermined feature amount from the region included in the detected tracklet, and a tracking unit that updates or creates tree-structured data having identification information of the object and using the detected tracklet and the extracted feature amount as nodes.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technology described in Patent Document 1 enables identification of the same person from a group of person images captured at each time. However, in Patent Document 1, depending on the posture and performance of the camera used as a sensor, different feature amounts may be extracted for the same person. That is, even when a plurality of cameras detect the same person, the same feature amount is not necessarily obtained. Therefore, there is a possibility that it cannot be determined that they are the same person.

[0007] An object of the present invention is to provide an aircraft recognition system, a sensing data processing device, and an aircraft recognition method that improve the recognition accuracy of an aircraft.

Means for Solving the Problems

[0008] The present invention is configured as follows.

[0009] In an aircraft recognition system including a plurality of ground sensing devices installed on the ground and a sensing data processing device that processes data transmitted from the plurality of ground sensing devices, the ground sensing device includes a sensor for observing an aircraft, and the sensing data processing device includes a data receiving unit that receives sensor data transmitted by the plurality of ground sensing devices, a storage unit that stores the sensor data received by the data receiving unit, a flight trajectory calculation unit that calculates a flight trajectory of the aircraft within a predetermined time using the sensor data stored in the storage unit, and a flight trajectory of the aircraft calculated by the flight trajectory calculation unit And an aircraft identification unit that determines whether the aircraft observed by the plurality of ground sensing devices is the same aircraft.

[0010] Also, in a sensing data processing device that processes data transmitted from a plurality of ground sensing devices installed on the ground, a data receiving unit that receives sensor data transmitted by the plurality of ground sensing devices, a storage unit that stores the sensor data received by the data receiving unit, a flight trajectory calculation unit that calculates the flight trajectory of a flying object within a predetermined time using the sensor data stored by the storage unit, and a flying object identification unit that determines whether the flying objects observed by the plurality of ground sensing devices are the same flying object using the flight trajectory of the flying object calculated by the flight trajectory calculation unit.

[0011] Also, in a flying object recognition method for recognizing a flying object by a plurality of ground sensing devices installed on the ground and a sensing data processing device that processes data transmitted from the plurality of ground sensing devices, the method includes receiving sensor data transmitted by the plurality of ground sensing devices, storing the received sensor data, calculating the flight trajectory of the flying object within a predetermined time using the stored sensor data, and determining whether the flying objects observed by the plurality of ground sensing devices are the same flying object using the calculated flight trajectory of the flying object.

Advantages of the Invention

[0012] Provided are a flying object recognition system, a sensing data processing device, and a flying object recognition method that can improve the recognition accuracy of a flying object by capturing the temporal change of the observation result of the flying object and determining the identity of the detected flying object by a plurality of ground sensors.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various components of the present invention do not necessarily have to exist independently of each other. It is allowed that one component is composed of a plurality of members, a plurality of components are composed of one member, a certain component is a part of another component, or a part of a certain component overlaps with a part of another component. Also, in each figure, the same reference numerals are given to equivalent elements, and duplicate descriptions are omitted as appropriate.

Examples

[0015] (Example 1) FIG. 1 is a conceptual diagram of a situation to which the aircraft recognition system according to Example 1 of the present invention is applied. In FIG. 1, the aircraft 104 is assumed to take off and land at the takeoff / landing port 101 and fly along the planned flight route created by the air traffic control center 111.

[0016] When creating the flight path of the aircraft 104, the control center 111 creates a flight path from the starting point to the ending point while avoiding the no-fly zone 109 over the private house 108, and utilizes the following information obtained from various sensors and the like at that time. These are the information on the airspace above the takeoff and landing port 101 grasped by the camera 102 and radar 103 installed around the takeoff and landing port 101, the weather information from the weather company 107, or the information on the wind conditions 106 grasped by the wind condition sensor 105 installed on the rooftop of the building 110, and the aircraft information grasped by the aircraft 104, etc.

[0017] The camera 102 and radar 103 detect the environment around the aircraft 104 and the takeoff and landing port 101, and transmit it to the control center 111. The wind condition sensor 105 installed in the building 110 detects the wind condition 106 in the vicinity of the takeoff and landing port 101, and transmits it to the control center 111. As the wind condition sensor 105, for example, a Doppler lidar can be applied.

[0018] As described above, in the operation management of FIG. 1, the ground facilities and the aircraft 104 cooperate with each other for flight operation. However, the present invention is not limited to such a configuration. For example, as another configuration example, a part of the functions on the control system side may be on the aircraft system side.

[0019] FIG. 2 is a hardware configuration diagram of the control center 111. In FIG. 2, the control center 111 is composed of a ground sensing device 207 and a sensing data processing device 208.

[0020] The sensing data processing device 208 includes a control unit (CPU) 201, a memory 202, a communication device 203, a display device 204, an input device 205, and a storage device 206. The control unit 201 is connected to the memory 202, the communication device 203, the display device 204, and the input device 205. The storage device 206 may be a storage disk. Also, the memory 202 may be, for example, a RAM.

[0021] Further, the control unit 201 is connected to the camera 102, the radar 103, and the wind condition sensor 105. The control unit 201 receives information from the flying object 104 and information from the meteorological company 107 via the communication device 203.

[0022] The ground sensing device 207 includes a radar 103, a camera 102, a communication device 209, and a control unit (CPU) 210, and is installed at a plurality of locations on the ground, for example, around the takeoff and landing port 101. The radar 103 and the camera 102 are paired sensors, and the ground sensing device 207 recognizes the flying object 104 by fusing the data of these. The control unit (CPU) 210 processes the data obtained by the camera 102 and the radar 103 to calculate the positions of the flying object 104 and the objects existing in the detection range. The communication device 209 transmits the result to the sensing data processing device 208.

[0023] FIG. 3 shows a monitoring screen 2400 of the flying object detection result by the ground sensing device 207 in the control center 111. On the monitoring screen 2400, the flying object detection result is superimposed on the video of each camera 102, so that the personnel in the control center 111 can visually confirm the situation of the flying object. The monitoring screen 2400 is displayed on the display device 204 and can be operated by the personnel using the input device 205. The monitoring screen 2400 of the flying object detection result is a display unit that displays the flying objects observed by the ground sensing device 207.

[0024] Next, the operation will be described. The personnel designates the camera to be displayed from the camera selection box 2406. In the example shown in FIG. 3, cameras 1, 4, 7, 11, 14, and 15 are selected.

[0025] At this time, when the full-screen button 2401 is selected, the video of the camera selected in the camera selection box 2406 is displayed in full screen. Since six cameras are selected here, the video of one of the selected cameras is displayed. At this time, by selecting the previous button 2404 or the next button 2405, it is possible to cyclically change the displayed camera video.

[0026] When the four-way split button 2402 is selected, the video of the selected camera can be displayed in a four-way split screen, and FIG. 3 shows an example thereof. At this time, when the previous button 2404 or the next button 2405 is selected, the displayed camera videos shift one by one. In the example of FIG. 3, when the next button 2405 is selected, cameras 4, 7, 11, and 14 are displayed, and when the previous button 2404 is selected, cameras 15, 1, 4, and 7 are displayed. Similar to the full-screen display, the camera videos are cyclically changed.

[0027] When the nine-way split button 2403 is selected, the video of the selected camera can be displayed in a nine-way split screen. The operation results of each button are as described above.

[0028] When the close button 2407 is selected, the monitoring screen 2400 is closed.

[0029] Next, the display of each camera video will be described. Objects detected by each ground sensing device 207 are highlighted with parentheses [ ]. For example, in the camera 1 screen, flying objects 104-11, 104-12, and 104-13 are detected. The same is true for other camera screens, and flying objects 104-14, 104-15, and 104-16 are detected. Regarding the flying object 104-17, although it is reflected in the camera video, it means that it was not detected.

[0030] FIG. 4 is a hardware configuration diagram of the flying object 104. In FIG. 4, the flying object 104 includes a control unit (CPU) 305, a memory 306, a communication device 307, and a storage device 304.

[0031] The control unit 305 is connected to the memory 306, the communication device 307, and the storage device 304. The control unit 305 is also connected to a GNSS (Global Navigation Satellite System) 303. The GNSS 303 is a device that measures its position using satellite radio waves such as GPS (Global Positioning System). The control unit 305 performs information communication with the sensing data processing device 208 via the communication device 307. The control unit 305 also receives information from the meteorological company 107 via the communication device 307.

[0032] The communication device 307 has a function of notifying information such as an incident occurring on the airframe, passengers, cargo, etc. of the aircraft 104. Further, regardless of the occurrence of an incident, it has a function of communicating the self-position measured by the GNSS 303 at regular intervals.

[0033] FIG. 5 is a configuration diagram of the aircraft recognition system 100A in the first embodiment. In FIG. 5, the aircraft recognition system is composed of a ground sensing device 207 and a sensing data processing device 208, and the ground sensing device 207 is installed at a plurality of locations on the ground. The data reception unit 402, flight trajectory calculation unit 404, aircraft identification unit 406, and aircraft identification unit 408 shown in FIG. 5 correspond to the CPU 201 shown in FIG. 2. Further, the detection result DB (detection result database) 403, flight plan DB 405, and tracking result DB 407 shown in FIG. 5 correspond to the storage device 206 shown in FIG. 2.

[0034] The ground sensing device 207 includes a camera 102 and a radar 103, and further includes a flying object detection unit (flying object position data creation unit) 401 that processes these data to detect a flying object. The flying object detection unit 401 identifies the position of the detected flying object 104, and transmits the result, together with the ID assigned to the ground sensing device 207, to the sensing data processing device 208 through the communication device 307. Also, when the model of the flying object 104 can be identified, the result is also transmitted.

[0035] The flying object detection unit 401 calculates the position of the flying object observed by the sensor, and creates sensor data associating the position of the flying object with the observation time.

[0036] On the other hand, the sensing data processing device 208 receives the data transmitted from the ground sensing device 207 by the data receiving unit 402, and stores it in the detection result DB (detection result database) 403. The flight trajectory calculation unit 404 uses the detection results stored in the detection result DB 403 to estimate the position of the flying object 104 according to the flow shown in FIG. 10, and stores the estimation results in the tracking result DB 407. The flight trajectory calculation unit 404 calculates the flight trajectory of the flying object 104 within a predetermined time for each of the flying object position data of the sensing devices 102, 103, and 401 stored in the detection result DB 403. The data creation unit 401 provided in the ground sensing device 207 calculates the position of the flying object observed by the sensor, and creates sensor data associating the position of the flying object with the observation time.

[0037] As shown in FIG. 6, when the function of the flying object detection unit 401 is not in the ground sensing device 207, the sensing data processing device 208 may be provided with a flying object position data creation unit 408 having the function of the flying object detection unit that calculates the position of the flying object observed by the sensor and creates sensor data associating the position of the flying object with the observation time.

[0038] The aircraft identification unit 406 identifies the ID (identifier) of the aircraft 104 according to the flow shown in FIG. 10 using the data in the flight plan DB 405 and the tracking result DB 407. The flow of FIG. 10 will be described later. The aircraft identification unit 408 determines the identity of the aircraft 104 according to the flows shown in FIGS. 15 and 19. The flows of FIGS. 15 and 19 will be described later.

[0039] FIG. 7 is a configuration diagram of the aircraft detection unit 401. In FIG. 7, the aircraft detection unit 401 includes a sensor parameter 501, a camera processing unit 502, a radar processing unit 503, an object positioning unit 504, and a communication unit 505. The processing of the aircraft detection unit 401 is preferably executed at regular intervals.

[0040] The sensor parameter 501 stores the external parameters (position and attitude) and internal parameters (focal length and origin coordinates in the camera 102) of the camera 102 and the radar 103. Regarding the position, by equipping with GNSS, it is possible to always store accurate values.

[0041] The camera processing unit 502 recognizes the aircraft 104 based on the image. Specifically, it recognizes the position and type of the aircraft 104. As the camera 102, a stereo camera is preferably used to calculate the position of the detected object. For calculating the object position by the stereo camera, an algorithm for obtaining the corresponding points of both eyes using SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) is generally used. Also, if the size of the aircraft 104 is known, a monocular camera may be used.

[0042] The radar processing unit 503 recognizes the position and type of the flying object 104 based on the detected object information obtained by the radar 103. The radar 103 can obtain a feature amount called RCS (RADAR Cross Section) and the position information of the detected object. The radar processing unit 503 extracts desired information from the detected object information based on these data. That is, data corresponding to noise other than the data of the flying object 104 is excluded, and only the data of the flying object 104 is extracted. For example, it is preferable to measure in advance the range of the RCS of the flying object 104 and extract only the objects within that range as the flying object 104.

[0043] The object positioning unit 504 finally determines the position of the flying object 104 using the object position calculation results from the camera processing unit 502 and the radar processing unit 503. The algorithm is preferably, for example, taking the average value of the results from the camera processing unit 502 and the results from the radar processing unit 503, or adopting the result with the shorter distance from the sensor among the results of both. The latter method is generally a method considering that the result with the shorter distance from the sensor has a smaller distance error. Also, when the processing of the camera processing unit 502 cannot be performed due to bad conditions or the like, the result of the radar processing unit 503 may be adopted, and vice versa.

[0044] The communication unit 505 transmits the result of the object positioning unit 504 to the sensing data processing device 208.

[0045] Figure 8 is a configuration diagram of the transmission data from the ground sensing device 207 to the sensing data processing device 208. In Figure 8, the transmission data is composed of the ground sensing device ID 601, the number of detected objects 602, the detected object information 603, and the detection time 604. The detected object information 604 is stored in the number corresponding to the number of detected objects 602.

[0046] The detected object information 604 is composed of position information 605, velocity information 606, and model 607. The position information 605 is composed of longitude 608, latitude 609, and altitude 610. The velocity information 606 is composed of velocity (longitude direction) 611, velocity (latitude direction) 612, and velocity (altitude direction) 613.

[0047] The detection time 603 is shown in the format of YYYY-MM-DD hh:mm:ss, and it is the time measured by a clock (not shown) held by the ground sensing device 207. In the first embodiment, it is assumed that the ground sensing device 207 detects every second, but this is just an example, and it may be transmitted up to a time less than one second. Also, it is preferable to use values in meters in the UTM (Universal Transverse Mercator) coordinate system for the position information 605. This is convenient for calculating the positional relationship between the aircraft 104 in the sensing data processing device 208. However, it is also preferable to transmit in degrees, minutes, and seconds and convert to meters in the sensing data processing device 208.

[0048] FIG. 9 is a configuration diagram of the detection result DB403. In FIG. 9, the detection result DB403 is composed of columns of time 701, sensor ID 702, aircraft latitude 703, aircraft longitude 704, aircraft altitude 705, velocity (longitude) 706, velocity (latitude) 707, and velocity (altitude) 708. That is, it stores the data shown in FIG. 8 as it is. Note that one detection result is stored as one record.

[0049] The time 701 is the time when the result indicated by the record is obtained by each ground sensing device 207, and this corresponds to the detection time 603. In the first embodiment, it is assumed that an accurate time is obtained in each ground sensing device 207.

[0050] The sensor ID 702 is the ID of the ground sensing device 207 that obtained the record result. This corresponds to the ground sensing device ID 601.

[0051] The aircraft latitude 703, the aircraft longitude 704, and the aircraft altitude 705 respectively correspond to the latitude 609, the longitude 608, and the altitude 610. Also, the velocity (longitude) 706, the velocity (latitude) 707, and the velocity (altitude) 708 respectively correspond to the velocity (longitude direction) 611, the velocity (latitude direction) 612, and the velocity (altitude direction) 613.

[0052] Figure 10 shows the processing flow of the flight trajectory calculation unit 404. In Figure 10, this processing flow is composed of two types of processing: the learning of the aircraft model and the prediction of the position and velocity of the flying object 104. The learning of the aircraft model is in steps S802 to S804, and the prediction of the position and velocity is in steps S805 to S806.

[0053] The learning of the aircraft model is executed when the position and velocity of the flying object 104 are observed. Therefore, in step S801, the flight trajectory calculation unit 404 checks whether the position and velocity of the flying object 104 are observed in a certain time period. If they are observed, the flight trajectory calculation unit 404 proceeds to step S802 to acquire the position and velocity data, learns the aircraft model in step S803, and updates the aircraft model in step S804. When the position and velocity of the flying object 104 cannot be obtained in the said time period, the learning process of steps S802 to S804 is not executed.

[0054] On the other hand, in step S801, even when the observation results of the position and velocity of the flying object 104 are not obtained in a certain time period, the observation results in the said time period are complemented using the prior estimation values in the time period before the said time period to calculate the flight trajectory, so that the current position and velocity can be estimated. At this time, the flight trajectory calculation unit 404 proceeds from step S801 to step S805 to acquire the prior estimation results of the position and velocity. For example, in the case of a Kalman filter, the prior estimation value should have been calculated at the previous estimation, so this is regarded as the current position and velocity.

[0055] Next, the flight trajectory calculation unit 404 acquires the airframe models in step S806 and uses them to predict the position and velocity of the flying object 104 in step S807. It is preferable to use a Kalman filter for this prediction as described above. The flight trajectory calculation unit 404 stores this result in the tracking result DB407.

[0056] FIG. 11 is a processing flow of the flying object identification unit 406. In FIG. 11, this processing flow identifies the flying object 104 by comparing the position of the flying object 104 with the position in the flight plan DB405.

[0057] In the first step S901, the flight plan at that time is obtained. The flight plan is composed of the ID of the flying object 104 scheduled to fly and the planned passing positions in each time zone. In the loop of the next step S902, the estimated positions of all the flying objects 104 detected within the monitoring range at that time are obtained, and the Euclidean distances between the estimated positions of the flying object 104 and the planned passing positions of all the flying objects are calculated for all combinations. In the next step S903, the coincidence probability is updated by Bayes' theorem to obtain the probability that the flying object 104 matches each flight planed flying object. In the next step S904, for all combinations, the optimal value is selected from the calculated coincidence probabilities. The Hungarian method is suitable for this, and it is possible to output, without duplication, the combination of the optimal flying object ID and the coincidence probability for each detected flying object 104.

[0058] Next, enter the loop of the number of detected flying objects. In step S908, the coincidence probability is compared with the threshold value. If it is smaller than the threshold value, it is determined in step S912 that they do not match, and the tracking result DB407 remains unchanged. If it is larger, it is determined in step 909 whether a dummy ID has been assigned. If a dummy ID has been assigned, then in step S911, -1 is stored in the airframe ID column of the tracking result DB407. If an ID of any of the flying objects 104 stored in the flight plan DB405 instead of the dummy ID has been assigned, the assigned ID is stored in the airframe ID column of the tracking result DB407.

[0059] Here, the above dummy ID will be described with reference to FIGS. 12A and 12B. When using the Hungarian method, the Euclidean distance between the planned position and the estimated position needs to be stored in a square matrix. That is, the number of aircraft for which flight is planned and the number of aircraft for which the position is estimated need to match. However, when they do not match, dummy elements are inserted to generate a square matrix.

[0060] In the first embodiment, it may happen that there are more estimated positions than planned positions. This example is shown in FIG. 12A. E1 to E5 are estimated positions, and P1 to P3 are planned positions. At this time, in order to generate a square matrix, P4 and P5 are created as dummy planned positions, and the square matrix shown in FIG. 12B is generated. At this time, two of E1 to E5 are assigned to P4 or P5 which are dummy. Since an aircraft ID cannot be assigned when a dummy is assigned, -1 is stored as a mark in the aircraft ID of the tracking result DB407. Incidentally, in FIG. 12B, the value of the dummy element is described as "∞" for convenience. The value of this element may be a large value that is impossible in terms of system operation.

[0061] Here, the formula for Bayesian update in step 903 is shown in formula (1).

[0062]

Equation

[0063] In formula (1), if the likelihood function Pj(Dj|Hi) is set to be inversely proportional to the Euclidean distance between the planned position and the estimated position, the shorter the distance, the higher the probability, and the longer the distance, the lower the probability. By applying such a likelihood function and setting the above large value to the dummy element, it is possible to avoid inappropriate assignment of dummy IDs by the Hungarian method.

[0064] FIG. 13 is a configuration diagram of the tracking result DB 407. In FIG. 13, the tracking result DB 407 is composed of columns of time 1101, sensor ID 1102, aircraft latitude 1103, aircraft longitude 1104, aircraft altitude 1105, speed (longitude) 1106, speed (latitude) 1107, speed (altitude) 1108, and aircraft ID 1109. As a configuration, the aircraft ID 1109 is added to the detection result DB 403, and it is different from the detection result DB 403 in that the estimated value by the flight trajectory calculation unit 404 is entered instead of the detected value. Also, similar to the detection result DB 403, one estimation result is stored as one record.

[0065] Also, as shown in FIG. 10, the flight trajectory calculation unit 404 can estimate the position and speed of the flying object 104 even when the position and speed of the flying object 104 are not observed. The tracking result DB 407 includes records at times when no observation results exist.

[0066] FIGS. 14A and 14B are conceptual diagrams of the processing content of the flying object identification unit 408. FIG. 14A shows a case where it can be determined that they are the same flying object, and FIG. 14B shows a case where it can be determined that they are different flying objects. In FIG. 14A, the flying objects 104-1 and 104-2, which are observation results by different ground sensing devices 207, are tracked, and since there is no change in their positional relationship, it is determined that they are the same flying object. In FIG. 14B, the flying objects 104-3 and 104-4, which are also observation results by different ground sensing devices 207, are tracked, and since their positional relationship has changed, it is determined that they are different flying objects. This process will be further described with reference to FIG. 15.

[0067] FIG. 15 is a processing flow of the flying object identification unit 408. In FIG. 15, the flying object identification unit 408 integrates the information obtained from each ground sensing device 207, searches for combinations of flying objects 104 that are close to each other, tracks the combined flying objects 104 for a certain period of time, and determines that they are the same flying object if there is no change in their mutual positional relationship.

[0068] First, in step S1301, the aircraft position at the current time is retrieved from the tracking result DB403, and in step S1302, combinations of the flying objects 104 that are close to each other are searched for.

[0069] Next, the combined flying objects 104 are tracked, and their relative positions are calculated. In FIG. 15, it is up to 5 seconds ahead, but this time can be arbitrarily set as long as there is no problem in system operation. In steps S1303 and S1304, the tracking results of each flying object 104 are retrieved from the tracking result DB403 for the number of combinations up to 5 seconds ahead, the differences (dx, dy, dz) in the coordinates between the flying objects 104 are calculated pairwise, and buffered in the memory 202.

[0070] In FIG. 15, a loop is used to obtain 5 seconds' worth of data for the process in step S1303, but it is also possible to use SQL in the retrieval from the tracking result DB403. To obtain the tracking result of a flying object with a specific ID, a command like "select * from tracking result where aircraft ID = 25 and time > '2023-06-22 13:35:20' and time < '2023-06-22 13:35:25' order by time" is issued to the tracking result DB. With this SQL, it becomes possible to obtain the tracking results for 5 seconds in chronological order for the aircraft with aircraft ID 25, which is equivalent to obtaining the trajectory of the aircraft. In this way, if the tracking result DB403 is used as a relational DB and SQL is issued, the 5-second loop becomes unnecessary.

[0071] Next, enter the loop of the number of combinations again, and select one flying object 104 from the combinations. In step S1305, for all combinations of (dx, dy, dz), it is determined whether there is a change in the relative position. For the determination, the average and standard deviation of the differences are calculated for each of (dx, dy, dz). If there is even one component whose average is greater than or equal to the threshold, it is considered that there is a change. If the average is less than the threshold but the standard deviation is greater than or equal to the threshold, it is also considered that there is a change.

[0072] If there is a change, it ends as it is. If there is no change, at step S1306, one flying object 104 is selected according to the determination criteria. Regarding the determination criteria, the one with the shortest distance from the ground sensing device 207 is given priority, and the others are deleted from the tracking result DB407. At this time, if the ID of the selected flying object 104 is not -1, it remains as it is, and if it is -1, it inherits the ID of the deleted flying object 104.

[0073] Furthermore, applying this method, the flying object identification unit 408 can, for example, perform DP matching on the trajectories of both, calculate the similarity of the flight trajectories by each sensing device 102, 103, 401, and determine whether the flying objects 104 observed by each sensing device 102, 103, 401 are the same flying object 104, thereby enabling identity determination. As a result, even when the results of the ground sensing device 207 are not obtained in the same time period, it is possible to obtain the similarity. That is, it is not necessary to strictly obtain the positional relationship for each time period.

[0074] By the processing described above, it becomes possible to determine the identity of the flying object 104.

[0075] According to the above-described Example 1, it is possible to provide a flying object recognition system and a flying object recognition method that can improve the recognition accuracy of flying objects and ensure the safety around the takeoff and landing ports by determining the identity of the detected flying objects by a plurality of ground sensors without depending on the feature amounts of camera images.

[0076] Note that the data reception unit may receive the flying object position data and integrate the data.

[0077] (Example 2) In Example 1, the identity of the flying object 104 was determined using the relative relationship of the results detected by the ground sensing device 207. In contrast, in Example 2, a method using the position information transmitted from the flying object 104 to the sensing data processing device 208 will be described.

[0078] FIG. 16 is a configuration diagram of the aircraft recognition system 100B in this embodiment. FIG. 16 shows a configuration when using the data of GNSS 303 transmitted from the aircraft 104, and the same reference numerals are added to the same components as those in the drawings described in the first embodiment.

[0079] The aircraft 104 transmits the self-position observed by the GNSS 303 to the sensing data processing device 208 at regular intervals through the communication device 307. The sensing data processing device 208 stores the data of GNSS 303 in the flight position DB 1401 via the data receiving unit 402.

[0080] The data structure of the flight position DB 1401 is as shown in FIG. 17. That is, it is composed of the time 1501, the aircraft ID 1502 of the aircraft 104, the aircraft latitude 1503, the aircraft longitude 1504, the aircraft altitude 1505, the speed (longitude) 1506, the speed (latitude) 1507, and the speed (altitude) 1508. The time 1501 is the time in the GNSS 303 mounted on the aircraft 104.

[0081] FIG. 18 is a conceptual diagram of the processing content of the aircraft identification unit 408. In FIG. 18, the aircraft 104-11 and 104-12 are the observation results by different ground sensing devices 207, and the aircraft 104-10 is the observation result by the GNSS 303. At this time, the aircraft 104-10 and 104-12 are tracked, and the changes in the positional relationship with the aircraft 104-10 are respectively determined. As a result, there is no change in the positional relationship in the combination of the aircraft 104-10 and 104-11, and since these can be determined to be the same, it can be known that the aircraft 104-11 is the aircraft 104-10. On the other hand, in the combination of the aircraft 104-10 and 104-12, since the positional relationship has changed, it can be known that the aircraft 104-12 is a separate entity from the aircraft 104-10.

[0082] FIG. 19 shows the processing flow of the aircraft identification unit 408. In FIG. 19, the aircraft identification unit 408 integrates the information obtained from each ground sensing device 207, and further searches for a combination of aircraft 104 that is close to the data of GNSS 303 obtained from the aircraft 104, tracks the combined aircraft 104 for a certain period of time, and determines that they are the same aircraft if there is no change in the relationship with the observed position by GNSS 303.

[0083] First, in step S1701, the aircraft position at the current time is retrieved from the tracking result DB 403, and in step S1702, the aircraft position at the current time is retrieved from the flight position DB 1401. Next, in step S1703, a combination of tracking results with a short distance to the flight position DB 1401 is extracted.

[0084] Next, the combined aircraft 104 is tracked and the positional relationship between them is calculated. In the flow shown in FIG. 19, it is up to 5 seconds ahead, but this time can be set arbitrarily as long as there is no problem in system operation. In step S1704, for each aircraft 104 in the combination, the tracking result is retrieved from the tracking result DB 407 up to 5 seconds ahead, and the flight position is retrieved from the flight position DB 1401. Next, in step S1705, the coordinate differences (dx, dy, dz) with the flight position DB 1401 are calculated for all aircraft and buffered in the memory 202.

[0085] In the flow shown in FIG. 19, a loop is used to obtain 5 seconds of data for the process in step S1704. Similar to step S1303 shown in FIG. 15, it is also possible to use SQL for the search from the tracking result DB 403. The details are as described above, and it is possible to obtain the tracking results for 5 seconds in time order for the aircraft with a specific aircraft ID, which is equivalent to obtaining the trajectory of the aircraft. In this way, if the tracking result DB is a relational DB and SQL is issued, the 5 - second loop is unnecessary.

[0086] Next, entering the loop of the combination number again, step S1706 is a process of setting the initial value of the parameters in this loop, setting the shortest distance between the tracking result and the flight position, and the index of the record with the shortest distance in the tracking result DB407. Further, entering the loop of the number of flying objects 104 included in the combination, at step S1707, for all combinations of (dx, dy, dz), it is determined whether there is a change in the positional relationship. For the determination, the average and standard deviation of the differences are calculated for each of (dx, dy, dz). If there is at least one component whose average is greater than or equal to the threshold, it is considered that there is a change. If the average is less than the threshold but the standard deviation is greater than or equal to the threshold, it is also considered that there is a change.

[0087] If there is a change, proceed to step S1712 and delete the tracking result of the flying object 104 from the tracking result DB403. In step S1707, if there is no change, at step S1708, calculate the distance between the tracking result of the flying object 104 and the flight position. If the distance is greater than the shortest distance, proceed to step S1712. Otherwise, proceed to step S1709 and rewrite the aircraft ID of the tracking result of the flying object 104 to the ID in the flight position DB1401. Next, at step S1710, delete the record of the shortest distance index from the tracking result DB403. However, if the value of the index is -1, it is not necessary to execute. Next, at step S1711, update the parameters in this loop.

[0088] Alternatively, as described above, it is also possible to determine identity by performing DP matching on the trajectories and calculating the similarity. Thereby, even when the results of the ground sensing device 207 are not obtained in the same time zone, it is possible to obtain the similarity.

[0089] By the processing described above, it is possible to determine the identity of the flying object 104.

[0090] In the second embodiment as well, the same effects as those in the first embodiment can be obtained.

[0091] (Embodiment 3) An embodiment for determining the identity of the flying object 104 described above is applied, and an example of calibration of the camera 102 and the radar 103 used in the ground sensing device 207 will be described as Example 3.

[0092] FIG. 20 is a configuration diagram of the flying object recognition system 100C in Example 3, and has a configuration including a calibration function. This is a configuration in which a parameter setting unit 1801 and a GNSS 1802 are added to the configuration shown in FIG. 5. The parameter setting unit 1801 adjusts the attitude parameters (roll, pitch, yaw) of the camera 102 and the radar 103 of the ground sensing device 207. That is, when the flying object identification unit 408 determines that it is the same flying object 104, the parameter setting unit 180 adjusts the parameters of the sensors 102 and 103 of the ground sensing device 207. If the position of the ground sensing device 207 is accurately measured by the GNSS 1802, no correction is required.

[0093] FIG. 21 is a conceptual diagram of the calibration process of the ground sensing device 207 in this Example 3. In FIG. 21, it is assumed that the ground sensing device 207-1 detects the flying object 104-1 and the ground sensing device 207-2 detects the flying object 104-2. As a result, a scene is assumed in which the flying object identification unit 408 determines that the flying objects 104-1 and 104-2 are the same flying object. At this time, the estimated positions using the detection results of the ground sensing devices 207-1 and 207-2, and the shortest distances 1901-1 and 1901-2 from the ground sensing devices 207-1 and 207-2 are obtained. Then, the ground sensing device 207-1 or 20-2 with the smaller of the shortest distances 1901-1 and 1901-2 is trusted (in the example of FIG. 21, the ground sensing device 207-1), and the external parameters of the other ground sensing device 207 are adjusted. The parameter setting unit 1801 executes this process.

[0094] FIG. 22 shows the processing flow of the parameter setting unit 1801. In FIG. 22, first, it enters the loop of the number of installed ground sensing devices 207. In step S2001, from the position information stored in a buffer (not shown) by the flight trajectory calculation unit 404, the minimum value of the detection distance by the ground sensing device 207 is obtained. In the example shown in FIG. 21, it corresponds to the shortest distances 1901-1 and 1901-2.

[0095] It exits the loop. Next, in step S2002, the ID of the ground sensing device 207 with the minimum minimum value of the shortest distance is obtained. In the example shown in FIG. 21, it corresponds to comparing the shortest distances 1901-1 and 1901-2 and finding the shorter one, and specifying the ID of the ground sensing device 207 at that time. Here, it is the ground sensing device 207-1.

[0096] Next, in step S2003, the external parameters of the ground sensing device 207 excluding the said ID are obtained. In the example shown in FIG. 21, taking the aircraft position 104-11 that generated the shortest distance 1901-1 as the true position of the flying object 104, the parameters of the ground sensing device 207-2 are updated. At this time, the estimated position 104-21 of the flying object 104-2 at the time closest to the estimated time of the aircraft position 104-11 is obtained, and the coordinate differences (dx, dy, dz) between the aircraft positions 104-11 and 104-21 are obtained. By using this value, it is possible to calculate the correction values for roll, pitch, and yaw respectively. Finally, in step S2004, the correction values obtained for the ground sensing device 207 excluding the said ID are transmitted. The ground sensing device 207 uses the correction values to update the sensor parameter 501 in step S2005.

[0097] In Example 3 as well, the same effects as in Example 1 can be obtained.

[0098] (Example 4) An example of calibrating the camera 102 and the radar 103 used in the ground sensing device 207 by applying the example of determining the identity of the flying object 104 described above is shown for Example 4.

[0099] FIG. 23 is a configuration diagram of the aircraft recognition system 100D in Embodiment 4, and has a configuration including a calibration function. This is a configuration in which a parameter setting unit 1801 is added to the configuration of Embodiment 2 shown in FIG. 16, and the parameter setting unit 1801 adjusts the attitude parameters (roll, pitch, yaw) of the camera 102 and the radar 103 of the ground sensing device 207. If the position of the ground sensing device 207 is accurately measured by the GNSS 1802, correction is not required.

[0100] FIG. 24 is a conceptual diagram of the calibration process of the ground sensing device 207 in the present Embodiment 4. In FIG. 24, the ground sensing device 207-1 detects the aircraft 104-1, and the ground sensing device 207-2 detects the aircraft 104-2, assuming a scene in which the aircraft identification unit 408 determines that the aircraft 104-1 and 104-2 are the same aircraft. FIG. 24 shows the estimated positions of the aircraft 104-1 and 104-2 at time T, and further shows the position estimation result 104-0 by the flight trajectory calculation unit 404 at time T. The position 104-0 is the observation result (self-position transmitted by the aircraft 104) of the aircraft 104 by the GNSS 303 at time T. At this time, regarding the position 104-0 as the true position of the aircraft 104, the external parameters of the sensors 102 and 103 provided in the ground sensing devices 207-1 and 207-2 are adjusted. The parameter setting unit 1801 executes this process.

[0101] FIG. 25 is a processing flow of the parameter setting unit 1801. This process is executed for all the ground sensing devices 207. First, in step S2301, the ID of the aircraft 104 detected by the ground sensing device 207 is obtained. As a result, it is determined in step S2302 whether there is a detected aircraft 104. If not, the process for the ground sensing device 207 ends. If there is a detected aircraft 104, the estimated position of the aircraft 104 at the current time is obtained from the tracking result DB 407 in step S2303, and the observed position of the aircraft 104 by the GNSS 303 at the current time is obtained from the flight position DB 1401 in step S2304.

[0102] Here, the processes of steps S2303 and S2304 may be in reverse order.

[0103] Next, in step S2305, the differences (dx, dy, dz) in the positions of these two are obtained, and by using this value, correction values for roll, pitch, and yaw of the ground sensing device 207 are calculated respectively. Finally, the external parameter correction value obtained in step S2306 is transmitted to the aircraft detection unit 401. The ground sensing device 207 updates the sensor parameter 501 in step S2307 by using the correction value.

[0104] Note that when there are a plurality of pieces of position information of the aircraft 104 determined to be the same by the aircraft identification unit 408, the parameter setting unit 1801 may be configured to adjust the parameters of the sensors 102 and 103 included in the ground sensing device 207 by using any one of them.

[0105] According to the fourth embodiment, in addition to obtaining the same effects as those of the third embodiment, there is an effect that more accurate calibration can be performed.

[0106] Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Also, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible. For example, the present invention is applicable not only to the vicinity of the takeoff and landing port but also to the entire flight area over a wide area. Also, in this embodiment, it was implemented when detecting an aircraft in the vicinity of the takeoff and landing port, but it is not limited to the vicinity of the takeoff and landing port and can be used for an aircraft in flight.

[0107] In addition, each of the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware by designing a part or all of them, for example, by using an integrated circuit. Further, each of the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, file, etc. for realizing each function can be stored in a memory, a recording device such as a hard disk or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0108] Also, control lines and information lines show those considered necessary for explanation, and not necessarily all control lines and information lines are shown on the product. In fact, it may be considered that almost all configurations are interconnected.

Description of Reference Numerals

[0109] 100A, 100B, 100C, 100D ··· Aircraft recognition system, 101 ··· Landing port, 102 ··· Camera, 103 ··· Radar, 104 ··· Aircraft, 105 ··· Wind condition sensor, 106 ··· Wind condition, 107 ··· Weather company, 108 ··· Private house, 109 ··· No-fly zone, 110 ··· Building, 111 ··· Control center, 201 ··· Control unit (CPU), 202 ··· Memory, 203, 209 ··· Communication device, 204 ··· Display device, 205 ··· Input device, 206 ··· Storage device, 207 ··· Ground sensing device, 208 ··· Sensing data processing device, 210 ··· Control unit (CPU), 303 ··· GNSS (Global Navigation Satellite System), 304 ··· Storage device, 305 ··· Control unit (CPU), 306 ··· Memory, 307 ··· Communication device, 401 ··· Aircraft detection unit (Aircraft position data creation unit), 402 ··· Data reception unit, 403 ··· Detection result DB, 404 ··· Flight trajectory calculation unit, 405 ··· Flight plan DB, 406 ··· Aircraft identification unit, 407 ··· Tracking result DB, 408 ··· Aircraft identification unit, 409 ··· Aircraft position data creation unit, 501 ··· Sensor parameter, 502 ··· Camera processing unit, 503 ··· Radar processing unit, 504 ··· Object position determination unit, 505 ··· Communication unit, 1401 ··· Flight position DB, 1801 ··· Parameter setting unit, 1802 ··· GNSS, 2400 ··· Monitoring screen, 2406 ··· Camera selection box

Claims

1. In a flying object recognition system including a plurality of ground sensing devices installed on the ground and a sensing data processing device that processes data transmitted from the plurality of ground sensing devices, each of the plurality of ground sensing devices includes a sensor for observing a flying object, the sensing data processing device includes: a data receiving unit that receives sensor data transmitted by the plurality of ground sensing devices; a storage unit that stores the sensor data received by the data receiving unit; a flight trajectory calculation unit that calculates a flight trajectory of the flying object within a predetermined time using the sensor data stored in the storage unit; a flying object identification unit that determines whether the flying objects observed by the plurality of ground sensing devices are the same flying object using the flight trajectory of the flying object calculated by the flight trajectory calculation unit; A flying object recognition system characterized by comprising the above.

2. In the flying object recognition system according to Claim 1, the ground sensing device includes a flight position data creation unit that calculates the position of the flying object observed by the sensor and creates sensor data associating the position of the flying object with the observation time. A flying object recognition system characterized by this.

3. In the flying object recognition system according to Claim 1, the sensing data processing device includes a flight position data creation unit that calculates the position of the flying object observed by the sensor and creates sensor data associating the position of the flying object with the observation time. A flying object recognition system characterized by this.

4. In the flying object recognition system according to Claim 1, the flying object identification unit compares the flight trajectories of the flying objects and determines whether the flying objects are the same flying object based on the comparison result. A flying object recognition system characterized by this.

5. In the flying object recognition system according to Claim 1, a flying object identification unit that identifies the flying objects observed by the plurality of ground sensing devices using the flight trajectory of the flying object calculated by the flight trajectory calculation unit and the flight plan indicating the planned passing positions of the flying object at each time zone of the planned flight; The flying object identification unit determines whether the flying objects observed by the plurality of ground sensing devices are the same flying object using the identifier of the flying object identified by the flying object identification unit. A flying object recognition system characterized by this.

6. In the flying object recognition system according to Claim 1, The flight object identification unit calculates the similarity of the flight trajectories based on the plurality of ground sensing devices, and determines whether the flight objects observed by the ground sensing devices are the same flight object. A flight object recognition system characterized by this.

7. In the flight object recognition system according to claim 1, When the observation result of the flight object cannot be obtained in a certain time period, the flight trajectory calculation unit uses the pre-estimated value in the time period before the certain time period to complement the observation result in the certain time period and calculates the flight trajectory. A flight object recognition system characterized by this.

8. In the flight object recognition system according to claim 1, When the flight object identification unit determines that the flight objects are the same, the system is provided with a parameter setting unit that adjusts the parameters of the sensors of the ground sensing device. A flight object recognition system characterized by this.

9. In the flight object recognition system according to claim 8, The parameter setting unit adjusts the parameters of the sensors of the ground sensing device using the self-position recognized by the ground sensing device and transmitted by the flight object. A flight object recognition system characterized by this.

10. In the flight object recognition system according to claim 8, When there are multiple pieces of position information of the flight object determined to be the same by the flight object identification unit, the parameter setting unit uses any one of them to adjust the parameters of the sensors of the ground sensing device. A flight object recognition system characterized by this.

11. In a sensing data processing device that processes data transmitted from a plurality of ground sensing devices installed on the ground, A data receiving unit that receives sensor data transmitted by the plurality of ground sensing devices; A storage unit that stores the sensor data received by the data receiving unit; A flight trajectory calculation unit that calculates the flight trajectory of a flight object within a predetermined time using the sensor data stored in the storage unit; A flight object identification unit that determines whether the flight objects observed by the plurality of ground sensing devices are the same flight object using the flight trajectory of the flight object calculated by the flight trajectory calculation unit; A sensing data processing device characterized by comprising the above.

12. In a flight object recognition method for recognizing a flight object by a plurality of ground sensing devices installed on the ground and a sensing data processing device that processes data transmitted from the plurality of ground sensing devices, Receiving sensor data transmitted by the plurality of ground sensing devices, Storing the received sensor data, Calculating a flight trajectory of the flying object within a predetermined time using the stored sensor data, Determining whether the flying object observed by the plurality of ground sensing devices is the same flying object using the calculated flight trajectory of the flying object, A flying object recognition method characterized by the above.

Images