Flying object detection system, flying object detection method, and flying object control system

Abstract

To safely control traffic of a flying object within an airspace.SOLUTION: A flying object detection system includes an acquisition unit 308 that receives airspace environment data items of an airspace, which have been acquired multiple times within a predetermined period, in relation to a flying object detected in the airspace, a unit area discrimination unit that discriminates boundary area attributes concerning boundaries of respective unit areas of the airspace contained in an image 409, which is a parallax image of the airspace based on the airspace environment data items, and a specific unit area, which relates to the flying object, out of the unit areas, and a posture calculation unit 310 that inputs the specific boundary area attribute of the flying object associated with the specific unit area out of the unit areas of the airspace, and a range and azimuth angle of the specific unit area to a model 416, and uses the model 416 to estimate a three-dimensional posture of the flying object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention is suitable for application to an air vehicle detection device, an air vehicle detection method, and an air vehicle control system that detects air vehicles within an airspace. [Background technology]

[0002] In recent years, air traffic and transportation (hereinafter simply referred to as "traffic") using small flying objects such as UAVs (Urban Air Vehicles) (hereinafter simply referred to as "flying objects") have been increasingly considered. In an era in which flying objects are widely used, detecting flying objects in airspace is an important element for ensuring safety and security. As a method for detecting flying objects, for example, Patent Document 1 discloses a technology for estimating object attitude based on attributes of an object area detected three-dimensionally using a stereo camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Australian Patent Publication No. 2023203521 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, it is difficult to calibrate the stereo camera according to various conditions to improve the detection accuracy of flying objects, and as a result, it is not possible to ensure the reliability of the detection accuracy of flying objects, and it is not possible to safely control the traffic of flying objects within the airspace.

[0005] In view of the above, an object of the present invention is to provide an aircraft detection device, an aircraft detection method, and an aircraft control system that can safely control the traffic of aircraft within an airspace. [Means for solving the problem]

[0006] In order to solve this problem, the present invention comprises an acquisition unit to which airspace environment data within an airspace is input, the airspace environment data being acquired multiple times within a predetermined period for an airborne object detected in the airspace; a unit area identification unit that derives boundary area attributes relating to the boundaries of each unit area of ​​the airspace contained in a parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area from among the unit areas that relates to the airborne object; and an attitude calculation unit that inputs the specific boundary area attributes of the airborne object corresponding to the specific unit area, and the range and azimuth angle of the specific unit area, into a learning model, and estimates the multidimensional attitude of the airborne object using the learning model.

[0007] In addition, the present invention includes a data acquisition step in which an acquisition unit inputs airspace environment data within the airspace acquired multiple times within a predetermined period for an aircraft detected in the airspace; a unit area identification step in which a unit area identification unit derives each boundary area attribute related to the boundary of each unit area of ​​the airspace included in a parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area related to the aircraft from among the unit areas; and an attitude calculation step in which an attitude calculation unit inputs the specific boundary area attribute of the aircraft corresponding to the specific unit area, and the range and azimuth angle of the specific unit area, into a learning model, and estimates the three-dimensional attitude of the aircraft using the learning model.

[0008] In addition, the present invention includes an air vehicle flying in an airspace and an air vehicle detection device that detects the air vehicle, wherein the air vehicle detection device includes an acquisition unit to which airspace environment data within the airspace, acquired multiple times within a predetermined period, is input for the air vehicle detected in the airspace; a unit area identification unit that derives boundary area attributes related to the boundaries of each unit area of ​​the airspace included in a parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area related to the air vehicle from among the unit areas; an attitude calculation unit that inputs the specific boundary area attributes of the air vehicle corresponding to the specific unit area, and the range and azimuth angle of the specific unit area into a learning model and estimates the multidimensional attitude of the air vehicle using the learning model; a risk estimation unit that determines whether or not a potential risk exists based on the estimated multidimensional attitude of the air vehicle; a warning notification generation unit that generates a warning notification based on the determination result of the risk estimation unit; and an output unit that outputs the warning notification generated by the warning notification generation unit and can output another warning notification depending on the behavior of the air vehicle in response to the warning notification. [Effects of the Invention]

[0009] According to the present invention, traffic of flying objects within an airspace can be safely controlled. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of an aircraft control system including an aircraft detection device according to this embodiment. [Figure 2] 2 is a block diagram showing an example of a hardware configuration of the flying object detection device shown in FIG. 1. FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the software configuration of the flying object detection device according to the present embodiment. [Figure 4] FIG. 1 is a conceptual diagram mainly showing an example of data processing by an aircraft detection device. [Figure 5] FIG. 10 is a diagram showing a modified example of the flying object detection device according to the present embodiment. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a model learning method. [Figure 7] FIG. 1 is a diagram showing an example of how a package is delivered using an air vehicle. [Figure 8] 10 is a flowchart illustrating an example of a procedure for a flying object detection process. [Figure 9] 10 is a flowchart showing another example of the procedure of the flying object detection process. [Figure 10] 10 is a flowchart illustrating an example of a procedure for a risk estimation process. [Figure 11] FIG. 10 is a diagram illustrating an example of a screen on which a warning event occurs. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a conceptual diagram showing an example configuration of an air vehicle control system including an air vehicle detection device according to this embodiment. The air vehicle detection device according to this embodiment includes a sensing platform. The sensing platform has a function of acquiring data (hereinafter referred to as "airspace environment data") related to the airspace environment including, for example, airborne flight assets (hereinafter collectively referred to as "air vehicles"). The air vehicle control system according to this embodiment includes an air vehicle detection device and an air vehicle. This embodiment will be described using, as an example, a verdiport area 101 in an urban area where landing and takeoff operations of an air vehicle 104 are performed. Note that this embodiment can be applied not only to the urban scenario described above, but also to rural scenarios, and can be extended to scenarios in other environments.

[0012] There are multiple types of air vehicles 104, each of which can fly in the airspace above the birdie port area 101. The air vehicles 104 are artificial air vehicles that can fly in the air by rotating multiple propellers facing upward. The air vehicles 104 are equipped with a sensor, for example, an IMU (Internal Measurement Unit) described below.

[0013] Birdieport area 101 is an area where flying vehicles 104 that may fly within the airspace can land and / or take off. Birdieport area 101 is marked with the letter "H" indicating the location where flying vehicles 104 should be located when taking off or landing.

[0014] Around birdieport area 101, facilities such as central control center 111, buildings 107, 108, and 110, and central control center 111 are located. Cameras 102 and radar 103 are dispersedly located around this birdieport area 101, and the area is covered by a GPS (Global Positioning System) and a GNSS (Global Navigation Satellite System) 110. Note that central control center 111 may be located at a location away from birdieport area 101. In the flying object detection device according to this embodiment, the locations and altitudes of these facilities are registered in advance as map information.

[0015] In this embodiment, for example, an air vehicle detection device is installed in the central control center 111. The air vehicle detection device includes a computer. The central control center 111 controls the flight of the air vehicle 104. Note that the air vehicle detection device may be installed in a location other than the central control center 111.

[0016] A wind detection sensor 105 is provided on the roof of the building 110. The airspace above the building 108 is set as a no-fly zone 109 where the flight of the flying object 104 is restricted, for example. This fact is registered in advance in the flying object detection device according to this embodiment.

[0017] The sensing platform includes sensing devices such as multiple cameras (stereo cameras) 102 and radar 103 distributed around the birdie port area 101 (or possibly inside it), buildings 107, 108, 110, and a central control center 111. These sensing devices are also collectively referred to as "sensors." The sensors may also include sensors that acquire other airspace environmental data, such as a wind detection sensor 105.

[0018] In addition, the sensing platform can be used to estimate environmental conditions, such as whether it is windy, raining, snowing, or cloudy, based on information about the wind blowing within the airspace, particularly above the building 110 (hereinafter referred to as "wind information") obtained from the wind detection sensor 105, or weather and wind information received from a server installed in the weather forecast company's building 107.

[0019] The sensing platform may, for example, have edge computing capabilities and may transmit information about the airspace in which the air vehicles 104 are flying to a central control center 111. The sensing platform may also use location data obtained by GPS / GNSS 110 to globally detect and locate the air vehicles 104.

[0020] In this embodiment, the central control center 111 is provided with reliable and accurate position, orientation, and velocity information about the flying object 104, so that the central control center 111 can safely control the flight of the flying object 104. The weather forecast company building 107 is equipped with a sensing system 112.

[0021] Fig. 2 is a block diagram mainly showing an example of the hardware configuration of the flying object detection device shown in Fig. 1. The sensing platform can be configured with multiple sensors (multiple cameras 102, or a camera 102 and a radar 103, etc.) for, for example, day and night flying object 104 operation capabilities and dynamic weather operation capabilities.

[0022] The sensing platform includes, as its minimum hardware configuration, a radar (RADAR) 103, multiple cameras 102 as an example of multiple monitoring units, a wind detection sensor 105, a heat sensor 207, and a visible light sensor 208. In this embodiment, these sensors constituting the sensing platform may also be referred to as "modality sensors." The sensing platform transmits airspace environment data acquired using these sensing devices to a central control center 111. Here, the multiple monitoring units are spatially distributed to monitor the airspace, for example, so that they are arranged on the same line while being offset laterally.

[0023] The flying object detection device 1 includes a memory 202 capable of storing data and programs in a volatile manner, a storage device 206 capable of storing data and programs in a non-volatile manner, a CPU (Central Processing Unit) 201, and a communication device 203, and preferably further includes a display device 204 and an input device 205. The input device 205 is an operation unit such as a keyboard or a mouse.

[0024] 3 is a block diagram showing an example of the software configuration of the flying object detection device 1 according to this embodiment. The flying object detection device 1 includes an acquisition unit 308, a unit area identification unit 309, an attitude calculation unit 310, a risk estimation unit 311, and a model 416. Here, the model 416 used is one that has been trained in advance by machine learning. Note that the model 416 does not have to be one that has been trained in advance, and generative artificial intelligence may be used.

[0025] The combination of the acquisition unit 308 and the unit area identification unit 309 is, for example, for a first channel and a second channel. The first channel corresponds to the right camera of the multiple cameras, and the second channel corresponds to the left camera of the multiple cameras. Since the first channel and the second channel have almost the same configuration and function, the following description will focus on the acquisition unit 308 and unit area identification unit 309 of the first channel unless a distinction is particularly required.

[0026] Acquisition unit 308 receives input of airspace environment data within the airspace acquired multiple times within a predetermined period for an air vehicle detected in the airspace. Acquisition unit 308 monitors the airspace including the air vehicle 104 using multiple cameras 102 positioned at a predetermined distance (offset), and acquires images 409 of the airspace as airspace environment data based on the monitoring results of the multiple cameras 102. Images 409 reflect the parallax of the cameras 102 positioned at a predetermined distance offset.

[0027] Any of the multiple monitoring units may be, for example, a radar (RADAR), a LiDAR (Laser Imaging Detection and Ranging), or a control device that controls the flight of the flying object 104 by RF (wireless communication). Of these, the control device has a function of acquiring airspace environment data related to the flying object 104 based on a control command when controlling the flight of the flying object 104 by wireless communication.

[0028] The unit area identification unit 309 is an example of a unit area identification unit, and derives boundary area attributes related to the boundaries of each unit area, which is an example of each unit area of ​​the airspace included in a parallax image (hereinafter simply referred to as an "image") of the airspace based on each airspace environment data. Here, each boundary area attribute includes the pixel value, width, height, center, etc. of each unit area. The unit area identification unit 309 identifies a specific unit area related to the aircraft from among the unit areas.

[0029] Here, in this embodiment, the flying object 104 may include not only artificial flying objects but also flying animals such as birds, and the unit area identification unit 309 has the function of identifying the type of flying object 104 based on the image 409, for example, using image recognition technology.

[0030] The attitude calculation unit 310 inputs the specific boundary area attributes of the aircraft corresponding to the specific unit area and the range and azimuth angle of the specific unit area into the model 416, and estimates the three-dimensional attitude of the aircraft 104 using the model 416.

[0031] In addition, the attitude calculation unit 310 may input, into the model 416, the attitude of the aircraft 104 obtained using a sensor mounted on the aircraft 104, and preferably also the specific boundary area attributes corresponding to the specific unit area, in addition to the specific boundary area attributes of the aircraft 104 described above and the range and azimuth angle of the specific unit area described above, and use the model to estimate the three-dimensional attitude and speed of the aircraft 104.

[0032] In this embodiment, the concept including the three-dimensional attitude and speed of the flying object 104 may be referred to as a "four-dimensional attitude," and the concept including the three-dimensional attitude and speed, as well as the four-dimensional attitude, may be collectively referred to as a "multidimensional attitude." Therefore, multidimensional here means three or more dimensions.

[0033] The model 416 may be a deep learning model such as a Long Short-Term Memory (LSTM). LSTM is a type of recurrent neural network (RNN) architecture that can learn long-term dependencies in sequential data. A Gated Recurrent Unit (GRU) is another type of recurrent neural network designed to address the vanishing gradient problem. The GRU has fewer parameters and lower computational cost than an LSTM. The learning model may be, but is not limited to, a Transformer network, a type of neural network architecture originally proposed for natural language processing tasks, or a Bi-LSTM (bidirectional LSTM).

[0034] In addition to the specific boundary area attributes of the aircraft and the range and azimuth angle of the specific unit area, the attitude calculation unit 310 may also input the attitude of the aircraft 104 into the model 416 and use the model 416 to estimate the multidimensional attitude of the aircraft.

[0035] The risk estimation unit 311 determines whether or not a potential risk exists based on the estimated multidimensional attitude of the flying object 104. The warning notification generation unit 312 generates a warning notification based on the determination result of the risk estimation unit 311. Furthermore, if a potential risk exists, the output unit 313 outputs the warning notification generated by the warning notification generation unit 312, for example, directly or to an air traffic control system (not shown). In the flying object control system according to this embodiment, the output unit 313 outputs the warning notification generated by the warning notification generation unit 312 as described above, and can also output another warning notification depending on the behavior of the flying object 104 in response to the warning notification.

[0036] The risk estimation unit 311 identifies the type of the aircraft 104 based on the characteristics of the aircraft 104 derived from the specific boundary region attributes of the aircraft 104, and determines whether there is a potential risk according to the type of aircraft 104. Here, the potential risk according to the type of aircraft 104 can be, for example, a collision between aircraft 104s, or a collision of the aircraft 104 with a building 107, 108, 110, the central control center 111, etc.

[0037] The attitude of the air vehicle 104 is attitude data acquired by a sensor 402, which is an IMU mounted on the air vehicle 104. Specifically, the attitude calculation unit 310 receives measurements by the IMU from the air vehicle 104 via wireless communication and estimates the three-dimensional attitude of the air vehicle 104. The attitude calculation unit 310 may also have a function to estimate the drift and bias of the air vehicle 104 using the estimated multidimensional attitude of the air vehicle 104 and other available information related to the attitude of the air vehicle 104.

[0038] Here, the above-mentioned unit area identification unit 309 estimates the multidimensional attitude of the flying object 104 using equations (1) to (6) described below. In each equation, x = position along the x axis, y = position along the y axis, z = position along the z axis, x' = velocity along the x axis, y' = velocity along the y axis, z' = velocity along the z axis, φ = roll angle, θ = pitch angle, ψ = yaw angle, p = roll rate (angular velocity around the x axis), q = pitch rate (angular velocity around the y axis), r = yaw rate (angular velocity around the y axis), r = yaw angle, φ = roll ... where θ = pitch angle, ψ = yaw angle, φ = yaw angle, r = yaw angle, r = yaw rate (angular velocity around the y-axis), m = mass of the air vehicle 104, N = rotational speed of the propeller (RPM), B = number of propeller blades, A = blade area (cross-sectional area swept by one propeller blade), CTT = thrust coefficient, ρ = air density, r = radius of the propeller of the air vehicle 104, and α = angle of attack of the blade.

[0039] Equations (1) through (6) are based on the performance characteristics of the air vehicle 104's propellers and the physics of fluid dynamics. The CT coefficients depend, for example, on the design and geometry of the air vehicle 104's propellers and are typically determined experimentally or from information provided by the air vehicle 104 manufacturer. The blade's angle of attack α also plays an important role in determining these coefficients. Because it is difficult to accurately measure the angle of attack α, the angle of attack α is one of the parameters that must be adjusted. Similarly, parameters related to the air vehicle 104's dynamic model, such as the aerodynamic coefficients and the thrust model of the motors driving the propellers, may also need to be adjusted based on real-time sensor data to improve attitude estimation accuracy. If the air vehicle 104 is a quadcopter, the torque generated by all four propellers for the quadcopter can be calculated by summing the torques generated by each propeller.

[0040] x=x' (1) y=y' (2) z=z' (3) X=((T / m)*(sin(θ)·cos(ψ)+cos(θ)·sin(φ)·sin(ψ)) ···(4) Y=(T / m)* (sin(θ)·cos(ψ)+cos(θ)·cos(φ)·sin(ψ)) ···(5) Z=(T / m)* (cos(θ)·cos(φ)-g) ···(6) T=CT·ρ·A·N2 ···(7)

[0041] In this embodiment, IMU state variables such as bgyro = gyroscope bias vector (three components of the x-axis, y-axis, and z-axis) and baccel = accelerometer bias vector (three components of the x-axis, y-axis, and z-axis) can also be estimated using attitude data and are modeled as estimated parameters of the IMU.

[0042] 4 is a conceptual diagram showing an example of data processing mainly by the flying object detection device. The flying object detection device has, on the input side, a millimeter wave radar 401, a sensor 402, and ground truths 403 to 406 related to the flying object 104. The sensor 401 is, for example, a millimeter wave radar, and acquires the range and azimuth angle of the flying object 104. The sensor 402 is, for example, mounted on the flying object 104.

[0043] In this embodiment, a learning model described below is used to estimate the multidimensional attitude of the aircraft 104, using the true value regarding the multidimensional attitude of the aircraft 104, the output of sensor 401, and the output of sensor 402.

[0044] 4 is a block diagram showing an example of the system configuration of the flying object detection device 1. The flying object detection device 1 has a first channel and a second channel as channels for acquiring boundary area attributes of the flying object 104. Since the first channel and the second channel perform similar processing, the following description will mainly focus on the first channel, which performs processing on the image on the left.

[0045] As described above, the two cameras 102 are arranged on the same line but offset laterally. In the first channel, the two cameras 102 independently capture images of the airspace multiple times within a predetermined period, thereby obtaining frame images 403 and 404. These frame images 403 and 404, which have a time difference of the predetermined period, are combined by the CPU 201 to generate image 409.

[0046] In the first channel, a noise filter 410 removes salt and pepper noise from the image 409, and then the CPU 201 averages its high-intensity pixels. Next, in the first channel, the CPU 201 filters the keypoint features to obtain a keypoint feature image 411.

[0047] In the first channel, the CPU 201 updates the pixel values ​​of the pixels of the keypoint features (hereinafter also referred to as "keypoint pixels") in the keypoint feature image 411 to, for example, 255. In the first channel, the CPU 201 performs a predetermined threshold process on the filtered image in which the pixel values ​​of the keypoint pixels have been updated, to obtain a binary filtered image 412. Here, in this threshold process, if the pixel value of the keypoint pixel is greater than 180, the pixel value is set to "1," and if the pixel value of the keypoint pixel is otherwise, the pixel value is set to "0." In this embodiment, pixels with a pixel value of "1" are considered to be flying object images.

[0048] CPU 201 clusters all pixels in binary filtered image 412 that have a pixel value of "1" based on, for example, their Euclidean distance, generates a nearby keypoint feature image 413 from the clustered nearby keypoint features, and generates at least one proposed airspace image 414 as an example of a specific unit area from nearby keypoint feature image 413. This proposed airspace image 414 includes an air vehicle image that is an image of air vehicle 104.

[0049] The CPU 201 derives each boundary region attribute 415 for each proposed airspace image 414. The boundary region attribute 415 includes information such as the center, pixels, width, and height of the unit region of the detected flying object 104.

[0050] CPU 201 inputs boundary region attributes 415, range and azimuth 407 of air vehicle 104 acquired by sensor 401, and attitude 408 of air vehicle 104 acquired by sensor 402 mounted on air vehicle 104 into model 416. Model 416 estimates multidimensional attitude 418 of air vehicle 104 in response to these inputs. Multidimensional attitude 418 includes, for example, three-dimensional attitude and velocity (also referred to as "four-dimensional attitude") of air vehicle 104.

[0051] The filter 420 is a probabilistic filter such as a Kalman filter, an extended Kalman filter, or an unscented Kalman filter, and further filters the output of the model 416. Note that the filter 420 may be omitted.

[0052] The second channel performs the same processing as the first channel described above on the opposite side, that is, the right-hand image.

[0053] 5 is a diagram showing a modified example of the flying object detection device according to this embodiment. Note that in the illustrated modified example, the data processing by the first channel and the second channel is illustrated in a greatly simplified manner.

[0054] The illustrated modification uses a thermal image acquisition unit (not shown), such as a thermal imaging camera, instead of using one of the two cameras 102 (or radar 103) as described above. That is, in this modification, the other camera 102 and the thermal imaging camera are employed as multiple monitoring units. The thermal imaging camera detects the distribution of heat within the airspace and acquires a thermal image 602 that shows the heat distribution state within the airspace.

[0055] In this modification, the unit area identification unit 309 calculates boundary area attributes 604, 603 from the visible light (RGB; Re Green Blue) image 601 and the thermal image 602, respectively, in the same manner as described above. In this modification, the calculated boundary area attributes 604, 603 also include the center, pixel value, width, and height of the unit area of ​​the detected air vehicle 104. In addition, in this modification, any attribute, such as a change in pixel intensity or a change in keypoint features of the unit area over time, can be used to train the model 416 by machine learning. In addition, in this modification, the model 416 can be trained by machine learning using boundary area attributes of the detected air vehicle 104 derived from a single sensor.

[0056] FIG. 6 is a conceptual diagram showing an example of a method for training the model 416. Multiple sources can be used to train the model 416. In the illustrated example, for example, boundary region attributes 701 and 702 of the aircraft 104 are used as explanatory variables as two sources, and a true value 703 related to the multidimensional attitude of the aircraft 104 is input as a target variable to machine-train the model 416. Note that the explanatory variables are not limited to the two mentioned above, and other sources may also be used. The true value 703 can be the multidimensional attitude of the aircraft 104 depending on the required requirements, but is not limited to this.

[0057] 7 illustrates an example of a package delivery using an air vehicle 104 that can take off from a vertiport 810 in an air vehicle control system. The delivery location may be, for example, a rooftop 808 of a high-rise building 806. The illustrated example illustrates a scenario in an urban area where slower systems, such as global positioning systems, may be difficult to implement.

[0058] In this embodiment, it is also proposed to use multiple ground sensing systems 811 as the above-mentioned multiple modality sensors. The positions of the ground sensing systems 811 are managed as map information and can be derived using, for example, a three-dimensional map of an urban scenario or a database of attitude errors of the air vehicle 104 recorded while delivering packages to various locations. In this embodiment, the positions of the modality sensors of the air vehicle 104 can be selected based on the time and position history of the air vehicle 104's attitude error within the airspace.

[0059] The above is an example of the configuration of the flying object control system including the flying object detection device 1 according to this embodiment. Next, we will explain the flying object detection process as an example of an flying object detection method as an example of the operation of the flying object control system including the flying object detection device 1. Figure 8 is a flowchart showing an example of the procedure of the flying object detection process. In this flying object detection process, the multidimensional attitude of the flying object 104 is estimated.

[0060] First, an overview of the flying object detection method will be given. The flying object detection method includes a data acquisition step in which airspace environment data within an airspace acquired multiple times within a predetermined period for a flying object visually captured in the airspace is input to an acquisition unit 308, a unit area identification step in which a unit area identification unit 309 derives each boundary area attribute related to the boundary of each unit area of ​​the airspace included in a parallax image (image 409) of the airspace based on each airspace environment data, and identifies a specific unit area related to the flying object from among the unit areas, and an attitude calculation step in which an attitude calculation unit 310 inputs the specific boundary area attribute of the flying object corresponding to the specific unit area, and the range and azimuth angle of the specific unit area, into a model 416, and estimates the three-dimensional attitude of the flying object using the model 416. A specific description will be given below.

[0061] In step S901, the acquisition unit 308 receives airspace environment data in the airspace acquired multiple times within a predetermined period of time for the aircraft 104 detected in the airspace via the first channel (and the second channel) (data acquisition step). Specifically, the acquisition unit 308 receives airspace environment data acquired by, for example, an available modality sensor. The modality sensor may be, but is not limited to, a multi-modality sensor, a single-modality sensor, or a single sensor.

[0062] In step S902, the unit area identification unit 309 derives boundary area attributes (bounding box attributes) relating to the boundaries of each unit area of ​​the airspace included in the parallax image (image 409) of the airspace based on each airspace environmental data, and identifies a specific unit area related to the air vehicle 104 from each unit area (unit area identification step). The boundary area attributes include, for example, the pixel value, width, height, and center of the unit area including the air vehicle image of the air vehicle 104. Specifically, the unit area identification unit 309 detects the specific unit area including the air vehicle image of the air vehicle 104 and the type (class) of the air vehicle 104 from the image 409 based on the airspace environmental data acquired in step S901, and derives the boundary area attributes of the specific unit area.

[0063] In step S903, the attitude calculation unit 310 acquires sensor data. Specifically, the attitude calculation unit 310 acquires active sensor data acquired by a RADAR / LiDAR or the like, or sensor data acquired by an IMU mounted on the flying object 104.

[0064] In step S904, the attitude calculation unit 310 acquires attitude data relating to the three-dimensional attitude of the air vehicle 104. Specifically, the attitude calculation unit 310 combines the attitude data using the output data in steps S903 and S904 to derive the multidimensional attitude (e.g., four-dimensional attitude) of the air vehicle 104.

[0065] Fig. 9 is a flowchart showing another example of the procedure of the flying object detection process shown in Fig. 8. In this flying object detection process, the multidimensional attitude of the flying object 104 is estimated. Note that in each step shown in Fig. 9, the same processes as those in Fig. 8 are denoted by the same reference numerals, and the description thereof will be omitted.

[0066] In the illustrated flowchart, step S903A differs from step S903 in the flowchart shown in Figure 8, but the other steps are similar. In step S803A, the attitude calculation unit 310 derives specific boundary region attributes of the flying object 104 from the output data in step S902. Description of other processes will be omitted.

[0067] 10 is a flowchart showing an example of the procedure for the risk estimation process. The risk estimation process is executed by the risk estimation unit 311.

[0068] In step S1101, the risk estimation unit 311 acquires airspace data. The airspace data may include the four-dimensional attitude and map information of the flying object 104. The map information is map information of the airspace environment shown in Figure 7. The airspace environment includes information about not only the air in the airspace but also ground facilities such as a vertiport 810 and the rooftop 808 of a high-rise building 806, including latitude, longitude, and altitude.

[0069] In step S1102, the risk estimation unit 311 determines, based on the airspace data acquired in step S1101, whether any of the following warning events has occurred or is likely to occur: two or more flying objects 104 flying close to each other; an flying object 104 is unexpectedly close to a facility or building (closer than the required safe distance); or an flying object 104 is flying (or is about to fly) in a no-fly zone. If it is determined in step S1102 that no warning event has occurred or is likely to occur, the risk estimation unit 311 terminates the risk estimation process.

[0070] On the other hand, if it is determined in step S1102 that a warning event has occurred or is likely to occur, in step S1103 the risk estimation unit 311 issues an external warning of the risk of the warning event. For example, as shown in Fig. 11, the risk estimation unit 311 displays on a display that a warning event has occurred in the airspace. Note that instead of or in addition to this, the risk estimation unit 311 may output a warning sound from a speaker (not shown).

[0071] Furthermore, in step S1103, the risk estimation unit 311 may generate and output a routing signal to protect the flying object 104 from the possibility of the flying object 104 colliding with something or entering a no-fly zone (or a flight-restricted zone). The flying object detection device 1 uses such a routing signal to issue a warning to prevent the flying object 104 from getting into a dangerous situation.

[0072] The air vehicle control system according to this embodiment includes an air vehicle flying in an airspace and an air vehicle detection device 1 that detects the air vehicle. The air vehicle detection device 1 includes an acquisition unit 308 to which airspace environmental data within the airspace, acquired multiple times within a predetermined period, is input for an air vehicle 104 detected in the airspace, a unit area identification unit 309 that derives boundary area attributes related to the boundaries of each unit area of ​​the airspace included in an image 409 as an example of a parallax image of the airspace based on the airspace environmental data, and identifies a specific unit area related to the air vehicle 104 from among the unit areas, and an attitude calculation unit 310 that inputs the specific boundary area attributes of the air vehicle corresponding to the specific unit area, as well as the range and azimuth angle of the specific unit area, into a model 416 and estimates the multidimensional attitude of the air vehicle 104 using the model 416.

[0073] In this way, it is possible to safely control the traffic of aircraft 104 within the airspace by using model 416 to estimate the multidimensional attitude of aircraft 104 while ensuring reliability in the accuracy of detection of aircraft 104, without having to calibrate multiple cameras 102 according to various conditions to improve the accuracy of detection of aircraft 104.

[0074] In the flying object control system including the flying object detection device 1 according to this embodiment, the acquisition unit 308 monitors the airspace including the flying object 104 using multiple cameras 102 arranged at a predetermined distance (offset), and acquires images 409 related to the airspace as airspace environment data based on the monitoring results of the multiple cameras 102. In this way, the multidimensional attitude of the flying object 104 can be accurately estimated using the images 409 acquired with parallax using the multiple cameras 102, and traffic of the flying object 104 within the airspace can be safely controlled.

[0075] In the flying object control system including the flying object detection device 1 according to this embodiment, the unit area identification unit 309 has a function of identifying the type of flying object 104 based on the image 409, for example, by using image recognition technology. In this way, traffic of flying objects 104 within the airspace can be safely controlled according to the type of flying object 104.

[0076] In an aircraft control system including an aircraft detection device 1 according to this embodiment, any of the multiple monitoring units is, for example, a radar (RADAR), a LiDAR (Laser Imaging Detection and Ranging), or a control device that controls the flight of the aircraft 194 via RF (wireless communication).

[0077] In the aircraft control system including the aircraft detection device 1 according to this embodiment, the risk estimation unit 311 determines whether a potential risk exists based on the estimated multidimensional attitude of the aircraft 104. The warning notification generation unit 312 generates a warning notification based on the determination result of the risk estimation unit 311. Furthermore, the output unit 313 outputs the warning notification generated by the warning notification generation unit 312. In this manner, traffic of the aircraft 104 within the airspace can be safely controlled by an air traffic control system or the like. Furthermore, in the aircraft control system according to this embodiment, the output unit 313 outputs the warning notification generated by the warning notification generation unit 312 as described above, and can further output another warning notification depending on the behavior of the aircraft 104 in response to the warning notification. In this manner, traffic of the aircraft 104 within the airspace can be more reliably and safely controlled.

[0078] In the flying object control system including the flying object detection device 1 according to this embodiment, the risk estimation unit 311 identifies the type of flying object 104 based on the characteristics of the flying object 104 derived from specific boundary area attributes of the flying object 104, and determines whether there is a potential risk according to the type of flying object 104. In this way, it is possible to avoid potential risks according to the type of flying object 104, thereby safely controlling the traffic of flying objects 104 within the airspace.

[0079] In the flying object control system including the flying object detection device 1 according to this embodiment, the multiple cameras 102 are spatially distributed to monitor the airspace. In this manner, the multidimensional attitude of the flying object 104 can be more accurately estimated using images 409 acquired with sufficient parallax using the multiple cameras 102, thereby enabling safe control of the traffic of the flying object 104 within the airspace.

[0080] In an aircraft control system including an aircraft detection device 1 according to this embodiment, the attitude calculation unit 310 inputs the specific boundary area attributes of the aircraft, the range and azimuth angle of the specific unit area, as well as the attitude of the aircraft 104 into the model 416, and estimates the multidimensional attitude of the aircraft using the model 416.

[0081] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, the elements described in parallel in the present embodiment may be configured such that at least one of the elements is connected in series to the other elements. [Industrial Applicability]

[0082] The present invention can be applied to, for example, an aircraft detection device, an aircraft detection method, and an aircraft control system that can detect aircraft that may fly within an airspace. [Explanation of symbols]

[0083] 1...Flying object detection device, 308...Acquisition unit, 309...Unit area identification unit, 310...Attitude calculation unit, 311...Risk estimation unit, 416...Model

Claims

1. an acquisition unit to which airspace environment data in the airspace is input, the airspace environment data being acquired multiple times within a predetermined period for an aircraft detected in the airspace; a unit area identification unit that derives boundary area attributes related to the boundaries of each unit area of ​​the airspace included in the parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area related to the aircraft from among the unit areas; an attitude calculation unit that inputs a specific boundary area attribute of the aircraft corresponding to the specific unit area and the range and azimuth angle of the specific unit area into a learning model and estimates a multidimensional attitude of the aircraft using the learning model; An aircraft detection device comprising:

2. The acquisition unit The airspace including the aircraft is monitored by a plurality of monitoring units arranged at predetermined distances, and images of the airspace are acquired as the airspace environment data based on the monitoring results of the plurality of monitoring units.

2. The flying object detection device according to claim 1.

3. The unit area identification unit Identifying the type of the flying object based on the image 3. The flying object detection device according to claim 2.

4. Any one of the plurality of monitoring units A control device that controls the flight of radar, LiDAR, or the aircraft by wireless communication.

3. The flying object detection device according to claim 2.

5. a risk estimation unit that determines whether or not there is a potential risk based on the estimated multidimensional attitude of the aircraft; a warning notification generation unit that generates a warning notification based on a determination result of the risk estimation unit; an output unit that outputs the warning notification generated by the warning notification generation unit; 3. The flying object detection device according to claim 2, further comprising:

6. The risk estimation unit Identifying the type of the aircraft based on characteristics of the aircraft derived from specific boundary region attributes of the aircraft, and determining whether there is a potential risk associated with the type of aircraft.

6. The flying object detection device according to claim 5.

7. The plurality of monitoring units include: spatially distributed to monitor the airspace 3. The flying object detection device according to claim 2.

8. The attitude calculation unit specific boundary region attributes of the air vehicle; The range and azimuth angle of the specific unit area; and the attitude of the aircraft acquired using a sensor mounted on the aircraft are input into the learning model, and the multidimensional attitude of the aircraft is estimated using the learning model.

2. The flying object detection device according to claim 1.

9. an air vehicle flying in the airspace; an aircraft detection device that detects the aircraft, The flying object detection device an acquisition unit to which airspace environment data in the airspace is input, the airspace environment data being acquired multiple times within a predetermined period for an aircraft detected in the airspace; a unit area identification unit that derives boundary area attributes related to the boundaries of each unit area of ​​the airspace included in the parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area related to the aircraft from among the unit areas; an attitude calculation unit that inputs a specific boundary area attribute of the aircraft corresponding to the specific unit area and the range and azimuth angle of the specific unit area into a learning model and estimates a multidimensional attitude of the aircraft using the learning model; a risk estimation unit that determines whether or not there is a potential risk based on the estimated multidimensional attitude of the aircraft; a warning notification generation unit that generates a warning notification based on a determination result of the risk estimation unit; an output unit that outputs the warning notification generated by the warning notification generation unit and is capable of outputting another warning notification in accordance with the behavior of the aircraft in response to the warning notification; An aircraft control system comprising:

10. a data acquisition step in which airspace environment data in the airspace, which is acquired multiple times within a predetermined period, is input to an acquisition unit regarding an aircraft detected in the airspace; a unit area identification step in which a unit area identification unit derives boundary area attributes related to the boundaries of each unit area of ​​the airspace included in the parallax image of the airspace based on each of the airspace environment data, and identifies a specific unit area related to the aircraft from among the unit areas; an attitude calculation step in which an attitude calculation unit inputs a specific boundary area attribute of the aircraft corresponding to the specific unit area, and the range and azimuth angle of the specific unit area into a learning model, and estimates the three-dimensional attitude of the aircraft using the learning model; 1. A method for detecting an aircraft, comprising:

Images