System and method for warning a pilot of a collision risk during flight of an aircraft

Abstract

This application relates to systems and methods for warning a pilot of a collision risk during the flight of an aircraft. Systems and methods for tracking a population of flying objects using an area-scanning optical system to prevent collisions. An image of a volume of space in front of an aircraft in flight is captured using one or more cameras, and the image is then processed to determine whether the image data indicates the presence of a population. A camera-based anti-collision system is configured to detect and angularly track small objects in a population in flight at a sufficient distance to cue or warn a flight control system on an autonomous or piloted aircraft to avoid the population. A population angle tracking algorithm is used to identify a population of flying objects moving in unison in a certain direction by detecting consistent features indicative of population movement in the pixels of the captured image.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for avoiding collisions between swarms of small flying objects, such as drones or birds, and aircraft. More particularly, the present disclosure relates to analyzing images captured by one or more cameras to identify situations where a swarm is in the flight path of an aircraft. Background Art

[0002] Detecting and tracking flying objects that may collide with an aircraft is an important safety issue. The most important objects for which tracking information cannot be provided are disabled / non - cooperative aircraft, small drones, and birds. While the first object type can be reliably detected using radar, the other two types cannot be reliably detected due to their small size and lack of strong reflective materials. In contrast, cameras can reliably detect and track small drones and birds in good visibility conditions, as long as the range of the small drones and birds is not too large. Such camera systems can use ground - based systems and aircraft - based systems, and can be based on single - camera tracking or dual (stereo) camera tracking. A single - camera system tracks only by angle (azimuth and elevation), while a stereo system also tracks by distance.

[0003] The main problem with camera tracking of small objects such as birds and drones (whether in visible light, infrared, or multispectral) is that the resolution is typically not sufficient to detect and track at the ranges of interest (several miles away). At these ranges, the size of a bird or small drone is less than one pixel. Therefore, tracking an individual object of this type is unreliable. Summary of the Invention

[0004] The subject matter disclosed in detail below is directed to providing systems and methods for tracking swarms of flying objects to prevent collisions using area - scan optical systems, which may involve visible light, infrared, or multispectral imaging. According to some embodiments, images of a volume of space in front of a flying aircraft are captured using one or more cameras, and then the images are processed to determine whether the image data indicates the presence of a swarm. The advantage of camera systems is that they are completely passive, thus eliminating the problems of interference or the need for operating permits at specific airports. One or more cameras are used (as used herein, the term "camera" refers to a video camera). The camera - based anti - collision system can be located on the ground or on the aircraft.

[0005] According to one embodiment, a camera-based anti-collision system is configured to detect and angularly track small objects in a group flight at a sufficient distance away to prompt or warn a flight control system on an autonomous or piloted aircraft to avoid the group. The group angular tracking techniques disclosed herein can identify a group of flying objects moving coherently in a certain direction by detecting how the pixel values of a large number of pixels in the same area (referred to herein as a "sub-array") in a sequence of images change over time to indicate group movement (e.g., formation flight).

[0006] The group detection and tracking method proposed herein (hereinafter referred to as "group angular tracking") can track objects with a very low feature-to-noise ratio (pixels imaging the object have colors that differ little from pixels where no object is present). The group angular tracking method disclosed herein can also track objects through difficult background clutter. The group angular tracking method can also track small groups of flying objects without forming separate trajectories from individual objects, thereby improving the reliability of tracking. These features can benefit ground-based and aircraft-based anti-collision sensor systems by expanding the detection and tracking range and enabling collision alerts to be more reliable and timely. Providing a more timely alert for a potential collision with an aircraft about to take off can prevent a collision without the need for a maneuvering avoidance. Alternatively, an aircraft in the air will have a longer warning period, which will enable a safer and slower maneuvering avoidance and provide more time to warn other aircraft about the detected object.

[0007] Although various embodiments of systems and methods for tracking groups of flying objects to prevent collisions will be described in considerable detail later herein, one or more of the following aspects can characterize one or more of these embodiments.

[0008] One aspect of the subject matter disclosed in detail below is to provide a system for warning a pilot of a collision risk during flight of an aircraft, the system including: a camera system having a viewing point and a field of view, the field of view including a volume of space in front of the aircraft in flight, the camera system being configured to capture a frame of pixel data of successive images representing objects in the volume of space in response to enabling an image capture mode; an alarm system on the cockpit of the aircraft and capable of generating an alarm; and a computer system configured to perform operations including the following: (a) processing the frame to determine whether the pixel data indicates the presence of a swarm of flying objects in the volume of space; (b) in response to determining the presence of a swarm in processing operation (a), determining a current angular velocity and a current angular position of the swarm based on the pixel data; (c) extrapolating a future angular position of the swarm based on the current angular velocity and the current angular position of the swarm; (d) determining whether the future angular position of the swarm is within an angular collision region; and (e) after determining in operation (d) that the future angular position of the swarm is within the angular collision region, triggering the alarm system to generate an alarm. According to one embodiment, the computer system is further configured to determine whether a future intersection time of the swarm with the aircraft is greater than a minimum time for collision avoidance, in which case operation (e) includes: triggering the alarm system if the future angular position of the swarm is within the angular collision region and if the future intersection time is greater than the minimum time for collision avoidance.

[0009] According to a proposed implementation, operation (a) includes: meshing each frame of pixel data into an array of sub-arrays, the size of each sub-array being equal to a specified angular size of a typical swarm. Then, for each sub-array, the operation includes: extracting pixel values from a sequence of frames to form a swarm window; selecting a sequence of pixel angular velocities; calculating a swarm detection metric for each of the selected pixel angular velocities; comparing each swarm detection metric with a specified threshold, the specified threshold giving a desired false alarm rate; and for each detected swarm, generating a signal indicating to the alarm enabling module that a swarm has been detected, wherein the generated signal includes first digital data and second digital data, the first digital data representing the angular position of the sub-array containing the pixels representing the detected swarm, and the second digital data representing the angular velocity of the pixels representing the detected swarm.

[0010] Another aspect of the subject matter disclosed in detail below is to provide a method for warning a pilot of a collision risk during the flight of an aircraft, the method comprising the steps of: (a) using at least one camera to capture a frame of pixel data representing successive images of objects in a volume of space; (b) processing the frame in a computer system to determine whether the pixel data indicates the presence of a population of flying objects in the volume of space; (c) in response to determining the presence of a population in step (b), determining a current angular velocity and a current angular position of the population based on the pixel data; (d) extrapolating a future angular position of the population based on the current angular velocity and the current angular position of the population; (e) determining whether the future angular position of the population is within an angular collision zone; and (f) generating an alarm after determining in step (e) that the future angular position of the population is within the angular collision zone.

[0011] Another aspect of the subject matter disclosed in detail below is to provide an aircraft comprising: a camera that points in a forward direction along the centerline of the aircraft and has a field of view that includes a volume of space in front of the aircraft during flight, the camera being configured to capture a frame of pixel data representing successive images of objects in the volume of space in response to enabling an image capture mode; a cockpit of the aircraft that includes an alarm system capable of generating an alarm; and a computer system configured to perform operations including: (a) processing a frame received from the camera to determine whether the pixel data indicates the presence of a population of flying objects in the volume of space; (b) in response to determining the presence of a population in processing operation (a), determining a current angular velocity and a current angular position of the population based on the pixel data; (c) extrapolating a future angular position of the population based on the current angular velocity and the current angular position of the population; (d) determining whether the future angular position of the population is within an angular collision zone; and (e) triggering the alarm system to generate an alarm after determining in operation (d) that the future angular position of the population is within the angular collision zone.

[0012] Other aspects of systems and methods for tracking populations of flying objects to prevent collisions are disclosed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The features, functions, and advantages discussed in the previous sections may be implemented independently in various embodiments or may be combined in other embodiments. In the following, for the purpose of illustrating the above and other aspects, various embodiments are described with reference to the accompanying drawings. The drawings briefly described in this section are not drawn to scale.

[0014] Figure 1 is a top view of an aircraft showing possible positions of cameras for object detection and tracking on the aircraft according to two embodiments.

[0015] Figure 2 is a flowchart identifying steps of a group angle tracking algorithm according to one embodiment.

[0016] Figure 3 is a diagram showing a group grid window and a corresponding array of group detection metrics for a one-dimensional grid.

[0017] Figure 4 is a three-dimensional graph of group detection metrics according to frame count and number of one-dimensional pixels, showing the group detection results generated over time in a simulation, where the speed per frame is assumed to be between -1 pixel and 1 pixel in two directions.

[0018] Figure 5 is a three-dimensional graph of detection metrics according to number of one-dimensional pixels and frame count, showing the group detection results generated over time in a simulation, where the speed per frame is assumed to be between -1 pixel and 1 pixel in two directions.

[0019] Figure 6 is a graph showing the group detection metric values along the bottom boundary of the group detection metric array after 300 frames, where in the simulation, the sub-pixel noise at the object / noise is equal to -5 dB. The solid line represents the metric with a speed of 1 pixel / frame; the dashed line represents the metric with a speed of -1 pixel / frame.

[0020] Figure 7 is a graph showing the group detection metric values along the positive speed boundary of the group detection metric array, where in the simulation, the sub-pixel noise at the object / noise is equal to -5 dB. The solid line represents the metric with a speed of 1 pixel / frame; the dashed line represents the metric with a speed of -1 pixel / frame.

[0021] Figure 8 identifies the Figure 1 block diagram of components of an object detection and anti-collision system located on the aircraft depicted, the system including: a camera mounted on the aircraft nose, and a machine vision processing unit configured (e.g., programmed) to process the acquired image data using a group angle tracking algorithm.

[0022] Hereinafter, the accompanying drawings are described, where like elements in different figures have the same reference numerals. Detailed Embodiments

[0023] For purposes of illustration, a system and method for tracking a group of flying objects to prevent collisions will now be described in detail. However, not all features of the actual implementation are described in this specification. Those skilled in the art will recognize that many specific implementation decisions must be made in developing any such implementation to achieve the developer's specific goals (such as meeting system-related constraints and business-related constraints), which will vary from one implementation to another. In addition, it should be recognized that such development work may be complex and time-consuming, but for those of ordinary skill in the art who benefit from this disclosure, it is merely a routine task.

[0024] In the following, innovative techniques are disclosed in considerable detail that address the problem of in-air collisions between aircraft and small objects such as birds and small unmanned aerial vehicles (usually quadcopters). These two types of objects have many similarities: (1) Birds and small unmanned aerial vehicles are respectively similar in size and weight, and thus pose similar risks to aircraft. (2) Birds and small unmanned aerial vehicles can pose risks when flying in formation (as part of a group or flock). (3) Birds and small unmanned aerial vehicles are warmer than the background sky and can thus be detected using infrared imaging.

[0025] In addition, there are similarities in how a region-scanning imager (e.g., a camera) can detect small unmanned aerial vehicles and birds. In visible light, a distant bird or small unmanned aerial vehicle appears as a moving speck, where, in visible light, the color of the moving speck is typically darker than the color of the sky. For the case of infrared imaging, a distant bird or unmanned aerial vehicle appears as a moving speck, where, in infrared light, the color (temperature) of the moving speck is lighter than the color of the sky. In addition, the formation or group of any type of flying object appears similar at a distance and requires similar detection and tracking methods. In either case (birds or small unmanned aerial vehicles), the basic problems of detection and tracking are the same, and the performance of detection and tracking depends to a large extent on the size of the object and the difference between the color of the object and the background color, and on the size of the group.

[0026] Figure 1 FIG. is a diagram showing an aircraft 10 and a bird 104 in flight. The aircraft 10 includes: a fuselage 12, a pair of wings 14a and 14b, a horizontal stabilizer 16, a pair of engines 18a and 18b, and a vertical stabilizer 20. In Figure 1 the depicted example, the aircraft 10 may be flying in the direction indicated by arrow 102. The bird 104 may be flying in the direction indicated by arrow 106. In Figure 1The aircraft 10 and the bird 104 are not shown to scale. Depending on the position and relative movement of the aircraft 10 and the bird 104, the bird 104 may strike the aircraft 10. The bird 104 may be a member of a group of similar birds flying in formation, in which case the aircraft 10 may collide with a number of birds. The camera-based techniques proposed herein are designed to detect a group in motion in sufficient time to warn the flight crew to perform a maneuver to avoid collision.

[0027] Figure 1 Possible locations of cameras mounted to the aircraft 10 are shown. According to one embodiment, a forward-facing video camera 2 (hereinafter referred to as "camera 2") is mounted in the nose of the aircraft 10. Camera 2 is directed in the forward direction indicated by arrow 102 with a field of view 108. According to an alternative embodiment, the camera system in the nose of the aircraft 10 may include a plurality of video cameras with a field of view 108. In this example, the field of view 108 is defined by a volume of space (e.g., generally conical) between the dashed line 110 and the dashed line 112 in front of the aircraft 10. In the case of a conical volume, the axis of the cone may be coaxial with the direction of movement of the aircraft 10. Video images from the camera system on the aircraft 10 can be used to determine whether the bird 104 is likely to strike the aircraft 10 or has struck the aircraft 10. Additionally, video images from the camera system can be used to identify various characteristics of a bird strike. The bird 104 is within the field of view 108. Accordingly, the video images from the camera system on the aircraft 10 will include an image of the bird 104. According to an alternative embodiment, a pair of forward-facing stereo cameras 2a and 2b are mounted in the leading edges of the tips of the wings 14a and 14b, respectively. The stereo cameras 2a and 2b can be arranged to have at least partially overlapping fields of view. However, a pair of stereo cameras can be mounted elsewhere on the aircraft 10 to track a group during flight.

[0028] According to Figure 1 the illustrated exemplary embodiment, if a bird is close enough to the aircraft, video frames from the camera system on the aircraft 10 can be processed to identify an image of the bird 104. However, if the bird 104 is far away, the image of the bird 104 may occupy only a single pixel in the video frame. The situation becomes more complex when the bird 104 is a member of a flock of birds located in front of and far from the aircraft.

[0029] The system proposed herein uses group tracking techniques to detect and track groups by angle. In an alternative embodiment, this group angle tracking can be supplemented by deriving range (the distance separating the aircraft from the group) to help improve group tracking. According to some embodiments, one or more video cameras on the aircraft 10 capture image data frames, which are processed to detect the presence of a group of small flying objects, determine the likelihood of a collision with the group, and then alert the pilot if the likelihood of a collision exceeds a specified threshold. According to some other alternative embodiments, one or more cameras can be mounted on a tower near the runway. In the latter case, the video images can be processed on the ground by a computer system configured (e.g., programmed) to alert the aircraft when an arriving or departing aircraft may be at risk of colliding with a group.

[0030] Figure 2 is a flowchart identifying the steps of a group angle tracking algorithm 50 according to one embodiment. As the aircraft flies along its flight path, a video camera (mounted to the aircraft or a tower at the airport) continuously captures images of the volume of space in front of the aircraft's current position (step 52). The video camera can be configured to detect visible light, infrared radiation, or a multi-spectral band and output frames at the camera's frame rate. The resulting image video stream is then sent to a machine vision processing unit that performs Figure 2 the other steps identified therein.

[0031] The machine vision processing unit extracts frame f at time t t (step 54). Each frame consists of a two-dimensional array of pixel values. These values can be a single integer c indicating intensity at a certain optical frequency, or a vector consisting of a set of integers representing the intensities of the corresponding colors, e.g., c = (red, green, blue).

[0032] Then, the extracted frame f t undergoes image preprocessing (step 56). There are many different possible image preprocessing steps that can be used to enhance camera resolution, reduce artifacts, repair bad pixels, calculate image gradients, etc. Image preprocessing also includes image registration with the previous frame f in the case of camera motion (if the camera is mounted on the aircraft rather than a tower at the airport). t-1

[0033] Next, the machine vision processing unit updates the pixel noise by updating the pixel variance based on a time window (step 58). If frame f t is a two-dimensional pixel array C = {c ij(t)}, where i = 1, …, N and j = 1, …, M vary with the size of the N×M array at time t. Then, a windowing technique similar to the windowing technique described in U.S. Patent No. 10,305,701 is used to calculate the pixel variance update, but modified for the image frame to eliminate non-noise image changes due to moving objects. The machine vision processing unit calculates a sequence of variances {σ(t) 2} for each pixel within a time window of length m based on the following two iterative equations for the input pixel sequence {c(t)}:

[0034] S(t) = S(t - 1) + c(t) - c(t - m)

[0035] σ 2 (t) = σ 2 (t - 1) + c(t)(c(t) - S(t - 1)) - c(t - m)(S(t) - c(t - m))

[0036] This gives the variance for each of the N×M pixels in a frame. Then, the final pixel noise estimate calculates the mean μ(σ ij (t) 2 ) for all pixels i, j in the current frame, and then calculates the mean over the means:

[0037]

[0038] Finally, the pixel noise estimate is the mean of all pixel variances less than μ 2 :

[0039]

[0040] Then the current frame is gridded from the original pixel array into an array G(f t )(step 60). The size of each sub-array is pre-selected to be equal to the specified angular size of a typical population.

[0041] Each sub-array g t is extracted from the array G(f t )(obtained from the image data collected at time t)(step 62), and then each sub-array is processed in sequence.

[0042] For each sub-array, pixel values are extracted from consecutive frames to form a population window W g (t). The population window W g (t) is for each sub-array g t that includes the last m sub-arrays {g(t - m, …, g(t)} at that grid location up to the current time t.A time window associated with (also referred to as g(t) hereinafter). In this step and the next step, a new sub-array g(t) is added to the old time window W g (t - 1) (step 64), and the oldest sub-array g(t - m - 1) is deleted (step 66), retaining the current set W g (t) = {g(t - m + 1), …, g(t)}.

[0043] A sequence of pixel angular velocities (vx 1 、vy 1 )、(vx 2 、vy 2 )、…、(vx d 、vy d )} is selected, and this sequence of pixel angular velocities spans the range of the angular pixel velocity of the object being tracked (step 68). The machine vision processing unit assumes that in both directions, the angular velocity range is between -1 pixel and +1 pixel per frame.

[0044] Then, the swarm angle tracking module calculates the corresponding swarm detection metric S W (v) = S W (vx, vy) (step 70). The swarm detection metric is a measure of the degree to which a large number of pixel values in a sub-array of a sequence of frames change over time in a manner indicating swarm motion (such as formation flight). This calculation of the swarm detection metric is performed by executing the swarm angle tracking algorithm described in more detail below.

[0045] If the swarm detection metric S W (v) is higher than a selected threshold T that gives the desired false alarm rate, then for each detected swarm, a signal indicating the detected swarm is generated and sent to the alarm enabling module (step 72). The signal output to the alarm enabling module includes: digital data representing the angular position of the sub-array g that contains the pixels representing the detected swarm; and digital data representing the angular velocity v of the pixels within the sub-array g that represent the detected swarm.

[0046] Figure 2The last part shown in the figure is the process of issuing an alarm or a warning for a possible conflict with the detected group. The process of enabling the alarm (step 78) is specific to each aircraft or is part of airport operations. Generally speaking, assuming the current angular position g and the current angular velocity v of the group, this process extrapolates the group angular position into the future for a given number of seconds s (time t + s). It is determined whether the extrapolated (future) angular position (in the reference frame of the grid array) is within the future angular collision area (in the reference frame of the grid array) (step 74). (The future angular collision area generally corresponds to the current heading of the aircraft or the area around the predicted or assigned future trajectory of the aircraft within the next s seconds.) On the one hand, if it is determined in step 74 that the extrapolated angular position is not within the future angular collision area, the alarm is not enabled (step 80). On the other hand, if it is determined in step 74 that the extrapolated angular position is within the future angular collision area, it is determined whether s > S (step 76), where S is the selected alarm time, which allows for maneuvering or otherwise changing the flight plan. If s > S (meaning the pilot has enough time to take evasive action), an alarm or a warning is transmitted to the relevant party or system (step 78). If s is not greater than S, the alarm is not enabled (step 80).

[0047] Group detection metric S W (v) = S W (vx, vy) is calculated based on the group window W of the current sub - array g g and the selected angular velocity v. The group window has a set of m frame times f ∈ {t - m + 1, t - m + 2, …, t} and the last m sub - arrays {g(t - m + 1), g(t - m + 2), …, g(t)} up to the current time. As mentioned before, it is assumed that the pixel angular velocity (motion in pixels per second) is bounded by - 1 ≤ vx and vy ≤ 1. First, the group detection metric for a one - dimensional N×1 pixel grid (each sub - array g(f) is an N×1 pixel grid) working with dark (high - value) pixel objects will be described. Then, an indication of how this technique works for a two - dimensional pixel array will be provided. Hereinafter, the parameter v represents the one - dimensional angular velocity.

[0048] For each pixel index r in the one - dimensional sub - array g(f) at time f, the processor uses both the pixel value c r (f) and the following interpolated pixel values as inputs:

[0049] <c r (f)> v

[0050] The interpolated pixel value is calculated based on the pixel and its neighbors as:

[0051] <cr (f) > v = vc r-sgn(v) (f) + (1 - v)c r (f)

[0052] Where, if v > 0, then sgn(v) represents 1; if v < 0, then sgn(v) represents -1. Moreover, the processor uses the same pixel value at the previous time f - 1 as the input, i.e., c r (f - 1) and the following interpolated pixel values:

[0053] < c r (f - 1) > v

[0054] Then, based on these values on the population window, the processor recursively calculates the population detection metric array d (the size of this array is N × m) over time as:

[0055] d r (f) = d r (f - 1) + max{( < c r (f) > v - < c r (f - 1) > v - (c r (f) - c r (f - 1)), 0}

[0056] Where,

[0057] f ∈ {t - m + 1, t - m + 2, …, t}

[0058] And d r (t - m) = 0, i.e., this metric starts from zero at the beginning of this recursive calculation.

[0059] Figure 3 is a diagram showing the population grid window (left side) for a one - dimensional sub - array and the corresponding population detection metric array (right side). More specifically, Figure 3 illustrates how the population detection metric d is calculated over time for a one - dimensional sub - array. Each row in the population grid window represents the pixel data collected by the one - dimensional sub - array g at the corresponding time, where the oldest (in time) pixel data is in the top - most row and the newest (current) pixel data is in the bottom - most row. The arrow ( Figure 3 on the left side) in the population grid window indicates the velocities of six objects, one object moving to the left and the other five objects moving to the right. On Figure 3 the right side, the labels L 1:m (t) and R 1:m (t) indicate the left and right boundaries of the population detection metric array respectively, while the label B 1:N(t) indicates the bottom boundary of the population detection metric array. The population detection metric array can be depicted on a three-dimensional graph of the type seen in Figure 4 and Figure 5 where the horizontal axes indicate one-dimensional pixel count and frame count respectively, and the vertical axis indicates the value of the population detection metric according to the pixel count and frame count. In a final step, the population detection metric (shown before calculating the final average of the mean above the mean) is calculated by using the bottom boundary B 1:N (t) and the right boundary R 1:m (t) (since the velocity is positive in this example). No normalization of the population detection metric shown in Figure 3 is performed to illustrate the necessity of normalization.

[0060] It can be realized that N is the number of pixels in a one-dimensional subarray, and m is the number of time steps in the population calculation. Thus, the left and right metrics have m time steps, and the bottom metric has N pixels. Consequently, Figure 3 the left and bottom sides of the calculation represented in

[0061] Figure 4 and Figure 5 have a total of N + m values; similarly, the right and bottom sides of this calculation have a total of N + m values. Figure 4 Figure 5

[0062] Figure 6 Figure 7 The third step of the population detection process is to take the population detection metric from the left, right, and bottom boundaries of the population detection metric array (see L, R, and B defined below). Figure 6 and Figure 7It is a graph showing the population detection metric values along the bottom boundary and along the positive velocity boundary respectively after 300 frames with sub-pixel noise / noise equal to -5 dB at the object in the simulation. The solid line represents the metric with a velocity of 1 pixel / frame; the dotted line represents the metric with a velocity of -1 pixel / frame. The peaks of each of the six moving objects can be clearly seen. Therefore, the final step is to take the boundary values of the population detection metric array of this population grid window to form the final population detection metric. Specifically, the current time boundary (bottom side) will be d 1:N (t) (this time boundary will be used for all velocity values v); the positive velocity pixel boundary (right side) will be d N (1:m) (the positive velocity values with v≥0 will use this positive velocity pixel boundary); and the negative velocity pixel boundary (left side) will be d 1 (1:m) (the negative velocity values with v<0 will use this negative velocity pixel boundary).

[0063] The population detection metric is calculated using the pixel noise p calculated above. Specifically, the population detection metric S is calculated from the following one-dimensional boundary array W (v): This one-dimensional boundary array is defined as the concatenation of two of the three vectors that have been normalized to eliminate the non-constant noise background. These three vectors are:

[0064] For j = 1:N, for v≥0, set B j (t) = d j (t) - min(m - v(m - j), m)p 1 / 2 (for v<0, set B j (t) = (d j (t) - min(m + v(j), m)p 1 / 2 ).

[0065] For j = 1:m, for v≥0, set R j (t) = d N (j) - (m - v(m - j))p 1 / 2 (for v<0, set R j (t) = d N (j) - (m + v(j))p 1 / 2 ).

[0066] For j = 1:m, for v≥0, set L j (t) = d 1 (j) - (m - v(m - j))p 1 / 2 (for v<0, set L j (t) = d 1 (j) - (m + v(j))p 1 / 2 ).

[0067] Then, define the population detection metric S as follows W (v). Take the value C of the following vector of length N + m W (v), the number of these values that are greater than their mean above the mean:

[0068] For v ≥ 0, [B 1:N (t), R 1:m (t)]

[0069] For v < 0, [L 1:m (t), B 1:N (t)]

[0070] Then S W (v) = C W (v) / (N + m).

[0071] What has been described is for a one - dimensional N×1 sub - array as a grid. For a general N×M sub - array, the angular velocity v and the index r will have two components, the population detection metric array d will be of size N×M×m, and the mean above the mean will be changed to be calculated on a two - dimensional boundary array.

[0072] Each of the embodiments disclosed herein includes: one or more sensors, a computer system configured (e.g., programmed) to process image data, and an alert system that notifies a pilot of a potential collision. According to one embodiment, the sensor is a video camera mounted to an aircraft or a tower on the ground, and the computer system is a machine vision processing unit. Configure the machine vision processing unit to process the image data captured by the camera using the population angle tracking algorithm disclosed herein. The camera can be a visible - light camera, an infrared camera, or some combination that allows operation during both day and night. Various known mounts for these cameras can be employed, and they are not described in detail here. Those skilled in the art of camera systems can readily appreciate that various types of cameras can be used. For example, a low - light or infrared / thermal imaging camera can be used for night operations.

[0073] The aircraft 10 can move along a path (e.g., on a runway or in the air). A small flying object population ( Figure 1 not shown in the figure) may move along the same or different (e.g., intersecting) paths. Depending on the relative position and / or relative movement of the aircraft 10 and / or the population, there may be a risk of the aircraft 10 colliding with some members of the population. The anti - collision system disclosed herein can be installed on and Figure 1on different types of aircraft as depicted. For example, the aircraft can be a commercial airliner operated by an airline, a cargo aircraft operated by a private or public entity, a military aircraft operated by a military or other government organization, a private aircraft operated by an individual, or any other type of aircraft operated by any other aircraft operator.

[0074] According to one embodiment, the object detection and collision avoidance system further includes Figure 8 the additional components identified in, the additional component including a machine vision processing unit 22, which is located inside the aircraft fuselage but communicatively coupled to the camera 2 (or cameras) to receive image data in the form of a corresponding video frame stream from the camera 2. The machine vision processing unit 22 can be configured to recover three-dimensional shape and color information from the captured object images using any of a variety of methods known in the art.

[0075] According to one embodiment, the machine vision processing unit 22 includes a computer system that executes software configured to process the received image data to determine whether any group in the vicinity poses a collision risk. Additionally, the software executed by the machine vision processing unit 22 is configured to calculate the range of each potential collision group present in the field of view of the camera 2 based on the image data in the video frames captured by one or more cameras and also based on the aircraft geometry data retrieved from the aircraft model profile 24 and the aircraft state data (such as ground speed, heading) received from the air data inertial reference unit 28 ( Figure 8 the ADIRU in). The aircraft model profile 24 contains information about a specific aircraft model (such as the aircraft dimensions), which is required to determine whether an indication needs to appear in the cockpit / when an indication needs to appear in the cockpit. The aircraft dimensions describe the size and shape of the entire exterior of the aircraft.

[0076] The machine vision processing unit 22 executes instructions stored in a non-transitory tangible computer-readable storage medium such as an internal data storage unit, an external data storage unit, or a combination thereof ( Figure 8 not shown in). The machine vision processing unit 22 can include any of a variety of types of data processing techniques. For example, the machine vision processing unit 22 can include a dedicated electronic processor or a general-purpose computer. Other types of processors and processing unit technologies are also possible. Similarly, the non-transitory tangible computer-readable storage medium can include any of a variety of types of data storage techniques. For example, the non-transitory tangible computer-readable storage medium can include random access memory, read-only memory, solid-state memory, or any combination thereof. Other types of memory and data storage unit technologies are possible.

[0077] According to a proposed implementation, the machine vision processing unit 22 includes a computer system that executes software configured to process consecutive video frames. More specifically, the current frame is gridded into an array of sub-arrays that are roughly the angular size of a typical flock. Individual sub-arrays are extracted from the array and then processed sequentially using a flock angle tracking algorithm to detect when a flock appears in the image and then determine the current angular position and angular velocity of the flock. The machine vision processing unit 22 is also configured to extrapolate the angular position of the flock at a future time based on the current angular position and angular velocity of the flock and then determine whether the extrapolated angular position is within an angular collision zone.

[0078] If the machine vision processing unit 22 determines that the extrapolated angular position is within the angular collision zone and determines that there is sufficient time to maneuver the aircraft to avoid a collision, an alert is enabled. When a detection threshold is exceeded, an auditory cue trigger signal is sent to the cockpit auditory system 34, thereby enabling the cockpit auditory system to emit a cue sound through the speaker 36. The particular auditory cue emitted is a function of the proximity of the object or the nearness of a potential collision. According to one implementation, as the time to collision decreases, corresponding detection thresholds are exceeded, which triggers corresponding auditory indicators representing the corresponding risk levels. For example, a pilot warning sound can be emitted based on the nearness of a potential collision, and an impending collision will emit a louder sound cue (i.e., greater volume or amplitude) or a faster auditory cue (i.e., greater repetition rate).

[0079] Still referring to Figure 8 , the machine vision processing unit 22 also records images from the camera 2. The sequence of images is sent to the cockpit display system or the electronic flight bag 30 and then displayed on the cockpit display 32 for the pilot to view. Additionally, the machine vision processing unit 22 can be configured to send time-to-collision data to the cockpit display system or the electronic flight bag 30 for display on the cockpit display 32.

[0080] The machine vision processing unit 22 can execute the code of various software modules stored in a non-transitory tangible computer-readable storage medium. The machine vision processing unit 22 can include a number of processing units that execute the code of the corresponding software modules. The encoded instructions implementing the detection method can be stored in a mass storage device, volatile memory, non-volatile memory, and / or a removable non-transitory tangible computer-readable storage medium such as an optical disc that stores digital data.

[0081] The swarm angle tracking algorithm can be implemented using machine-readable instructions, which include a program executed by the machine vision processing unit 22. The program can be embodied as software stored on a non-transitory tangible computer-readable storage medium (such as a CD, floppy disk, hard drive, or memory associated with the anti-collision processor), but all or portions of the program can alternatively be executed by means implemented as firmware or dedicated hardware.

[0082] Optionally, a process of modeling the structure of a flock of birds or a swarm of drones can be performed. Based on modeling studies using the wingspan of the flying objects as a modeling parameter, the cumulative frequency of impacts of a given number of birds, 1,000 per swarm, on a jet engine with a diameter of 100 inches can be modeled. Based on this study and other studies, the hazards of various types of birds, the size of a typical swarm, migration behavior, and many other factors can be used to adjust the parameters of the systems disclosed herein. Such factors can affect the grid size, the camera's area of interest, detection and alert thresholds, etc.

[0083] The aircraft-mounted swarm detection and anti-collision system disclosed herein can be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, the aircraft-mounted swarm detection and anti-collision system can be implemented using analog or digital circuits, logic circuits, programmable processors, application-specific integrated circuits, programmable logic devices, or field-programmable logic devices.

[0084] In this document, certain systems, devices, applications, or processes have been described as including multiple modules. Except for those modules that are preferably implemented as hardware or firmware to enable stream computing as disclosed herein, a module can also be a unit of unique functionality that can be implemented as software, hardware, or a combination thereof. When the functionality of a module is executed in any part by software, the module can include a non-transitory tangible computer-readable storage medium.

[0085] Although systems and methods for tracking a flying object swarm for anti-collision have been described with reference to specific embodiments, those skilled in the art should understand that various changes can be made without departing from the scope of the teachings herein, and its elements can be replaced with equivalents. Additionally, many modifications can be made without departing from the basic scope of the teachings herein to adapt the teachings to a particular situation. Therefore, it is intended that the claims set forth herein are not limited to the disclosed embodiments.

[0086] As used herein, the term "computer system" should be broadly construed to cover a system having at least one computer or processor, and may have multiple computers or processors communicatively coupled by means of a network or a bus. As used in the foregoing sentence, the terms "computer" and "processor" both refer to a device that includes a processing unit (e.g., a central processing unit) and a form of memory (i.e., a non-transitory tangible computer-readable storage medium) that stores a program readable by the processing unit. As used herein, the term "camera system" should be broadly construed to cover a single camera or multiple cameras having at least partially overlapping fields of view.

[0087] The methods described herein may be encoded as executable instructions embodied in a non-transitory tangible computer-readable storage medium (including, but not limited to, storage devices and / or memory devices). When such instructions are executed by a processor or computer, they cause the processor or computer to perform at least a portion of the methods described herein.

[0088] The method claims set forth below should not be construed as requiring that the steps recited herein be performed in alphabetical order (any alphabetical ordering in the claims is used solely for the purpose of referencing previously recited steps) or in the order in which they are recited, unless the claim language expressly specifies or states conditions indicating a particular order for performing some or all of those steps. Nor should the method claims be construed as precluding any portion of two or more steps from being performed simultaneously or alternately, unless the claim language expressly states conditions precluding such an interpretation.

[0089] Note: The following clauses describe other aspects of the present disclosure.

[0090] Clause 1. An aircraft, the aircraft comprising:

[0091] A camera that points in a forward direction along a centerline of the aircraft and has a field of view that includes a volume of space in front of the aircraft during flight, the camera being configured to capture a frame of pixel data of successive images representing objects in the volume of space in response to enabling of an image capture mode;

[0092] A cockpit of the aircraft, the cockpit including an alarm system capable of generating an alarm; and

[0093] A computer system configured to perform operations including the following operations:

[0094] (a) Process frames received from the camera to determine whether the pixel data indicates the presence of a population of flying objects in the volume of space;

[0095] (b) In response to determining the existence of a group in process operation (a), determine the current angular velocity and current angular position of the group based on the pixel data;

[0096] (c) Extrapolate the future angular position of the group based on the current angular velocity and current angular position of the group;

[0097] (d) Determine whether the future angular position of the group is within an angular collision region; and

[0098] (e) After determining in operation (d) that the future angular position of the group is within the angular collision region, trigger the alarm system to generate an alarm.

[0099] Clause 2. The aircraft according to Clause 1, wherein the computer system is further configured to determine whether the future intersection time of the group and the aircraft is greater than the minimum time for collision avoidance, and wherein operation (e) includes: if the future angular position of the group is within the angular collision region and if the future intersection time is greater than the minimum time for collision avoidance, then trigger the alarm system.

[0100] Clause 3. The aircraft according to Clause 1, wherein operation (a) includes:

[0101] Grid the pixel data for each frame into an array of sub-arrays, the size of each sub-array being equal to the specified angular size of the group; and

[0102] Detect the manner in which a large number of pixel values in the sub-arrays of a sequence of frames change over time to indicate group movement.

Claims

1. A system for warning a pilot of a collision risk during flight of an aircraft, the system include: a camera system having a viewpoint and a field of view, the field of view comprising a volume of space forward of an aircraft in flight, the camera system being configured, in response to activation of an image acquisition mode, to capture frames of pixel data representing successive images of objects in the volume of space; an alarm system, the alarm system being on the cockpit of the aircraft and capable of generating an alarm; as well as A computer system configured to perform operations comprising: (a) processing the frame to determine whether the pixel data indicates the presence of a group of flying objects in the volume of space; (b) in response to determining in processing operation (a) that a group exists, determining a current angular velocity and a current angular position of the group based on the pixel data; (c) extrapolating a future angular position of the population based on the current angular velocity and the current angular position of the population; (d) determining whether the future angular position of the population is within an angular collision region; and (e) after determining in operation (d) that the future angular position of the swarm is within the angular collision zone, triggering the alarm system to generate an alarm, wherein operation (a) comprises: gridding each frame of pixel data into an array of sub-arrays, each sub-array having a size equal to a specified angular size of the group, and further wherein operation (a) further comprises: detecting that a number of pixel values ​​in the sub-arrays of a sequence of frames vary over time in a manner indicative of movement of the group, The operation (a) further includes, for each subarray: Extracting pixel values ​​from a sequence of frames to form a population window; Selecting a sequence of pixel angular velocities; calculating a group detection metric for each selected pixel angular velocity; comparing each population detection metric to a specified threshold that gives a desired false alarm rate; and A signal is generated for each population detected to indicate to an alarm enabling module that a population has been detected, wherein the generated signal includes first digital data representing an angular position of a subarray containing pixels representing the detected population and second digital data representing an angular velocity of the pixels representing the detected population.

2. The system according to claim 1, in, The computer system is also configured to determine whether a future intersection time of the group and the aircraft is greater than a minimum time for collision avoidance, and wherein operation (e) includes triggering the alarm system if the future angular position of the group is within the angular collision area and if the future intersection time is greater than the minimum time for collision avoidance.

3. The system according to claim 1, in, The computer system comprises: A group angle tracking module, the group angle tracking module performing operation (a) and operation (b); and The alarm enabling module, the alarm enabling module performs operations (c) to (e).

4. The system according to claim 1, in, The population detection metric is a measure of the degree to which a large number of pixel values in a sub-array of a sequence of frames vary over time in a manner indicative of population movement. Further, the population detection metric for each sub-array is calculated over a population window having a large number of frame times for a selected pixel angular velocity.

5. The system according to claim 4, wherein, the population detection metric for each sub-array is calculated recursively over time using a large number of pixel values and interpolated pixel values obtained from the large number of pixel values as inputs.

6. A method for warning a pilot of a collision risk during flight of an aircraft, the method comprising the steps of: (a) capturing, using at least one camera, a frame of pixel data representing successive images of objects in a volume of space; (b) processing the frame in a computer system to determine whether the pixel data indicates the presence of a population of flying objects in the volume of space; (c) in response to determining in step (b) the presence of a population, determining a current angular velocity and a current angular position of the population based on the pixel data; (d) extrapolating a future angular position of the population based on the current angular velocity and the current angular position of the population; (e) determining whether the future angular position of the population is within an angular collision zone; and (f) generating an alert after determining in step (e) that the future angular position of the population is within the angular collision zone, step (b) includes: gridifying each frame of pixel data into an array of sub-arrays, the size of each sub-array being equal to a specified angular size of the population, wherein step (b) further includes: detecting a large number of pixel values in sub-arrays of a sequence of frames to vary over time in a manner indicative of population movement, wherein step (b) further includes, for each sub-array: extracting pixel values from a sequence of frames to form a population window; selecting a sequence of pixel angular velocities; calculating a population detection metric for each selected pixel angular velocity; comparing each population detection metric with a specified threshold, the specified threshold giving a desired false alarm rate; and generating, for each detected population, a signal indicating to an alert enabling module that a population has been detected, wherein the generated signal includes first digital data and second digital data, the first digital data representing the angular position of the sub-array containing the pixels representing the detected population, and the second digital data representing the angular velocity of the pixels representing the detected population.

7. The method according to claim 6, the method further comprising the steps of: determining whether a future intersection time of the population with the aircraft is greater than a minimum time for collision avoidance, and wherein step (f) includes: generating the alert if the future angular position of the population is within the angular collision zone and if the future intersection time is greater than the minimum time for collision avoidance.

8. The method according to claim 6, wherein, the computer system includes: a population angle tracking module that performs steps (a) and (b); and the alert enabling module that performs steps (c) to (e).

9. The method according to claim 6, wherein, the population detection metric for each sub-array is calculated over a population window having a large number of frame times for a selected pixel angular velocity.

10. The method according to claim 9, wherein, the population detection metric for each sub-array is calculated recursively over time using a large number of pixel values and interpolated pixel values obtained from the large number of pixel values as inputs.

Images