Determining relative position data of objects

Abstract

A method implemented by a controller (100) of a first object (200) with respect to a second object includes repeatedly obtaining (306) a stream of position data of the second object by: i) receiving (302) infrared (IR) signals emitted by one or more IR emitters on the second object; and ii) using a pattern created by the sensed IR signals to determine (304) position data of the second object.

Description

[Technical Field]

[0001] The disclosure herein relates to determining relative position data of moving objects. The disclosure further relates to, among other things, detecting first and second objects relative to each other for purposes of tracking, performing coordinated operations, and communicating with each other. [Background technology]

[0002] There is an increasing number of aerial vehicles, including manned and unmanned aerial vehicles, such as airplanes and drones. Individual drones can be used for purposes such as package delivery, traffic monitoring, and aerial photography. There is also growing interest in groups of drones working together, such as drone fleets. Groups of drones can work together for many purposes. For example, in entertainment, groups of drones can be equipped with light-emitting diodes (LEDs) and flown in sync to create light shows. Drones can also be used in disaster management scenarios, such as to map and / or extinguish wildfires. In another example, groups of drones can be equipped with thermal imaging capabilities and used to survey large areas for human body heat signatures after an earthquake. Drones and groups of drones have myriad other uses.

[0003] Coordination of aerial vehicles such as drones presents a significant challenge in terms of coordinating drones not only to produce a desired result (e.g., a successful ground survey or light show), but also to prevent collisions between drones within a fleet and with other aircraft.

[0004] Boats and vessels face similar problems, and as their numbers continue to grow, coordination of such vessels becomes more important and more difficult, especially when operations need to be carried out, for example, in confined spaces and / or between pairs or groups of vessels.

[0005] There are also issues related to land vehicles such as driverless cars, which require the car to be equipped with sensors that enable onboard algorithms to accurately and safely perform maneuvers and prevent collisions.

[0006] Many, if not most, of the vehicles in the above ground examples rely on positioning, navigation, and timing (PNT) systems such as GPS, as described in the U.S. Department of Transportation document, "GPS Dependencies in the Transportation Sector. An Inventory of Global Positioning System Dependencies in the Transportation Sector, Best Practices for Improved Robustness of GPS Devices, and Potential Alternative Solutions for Positioning, Navigation, and Timing" (Volpe (2016)). However, in many situations, existing systems and methods are not sufficiently accurate or reliable, for example, when the speed of moving objects is not low and proximity is relatively close, or when the environment is not favorable.

[0007] As discussed above, the increasing number of vehicles and the increasing complexity of group operations conducted in land, water, and air scenarios creates unique challenges associated with coordinating such vehicles to safely and accurately perform tasks and prevent accidents. To this end, one challenge is obtaining real-time position data of objects, such as vehicles, with sufficient accuracy to provide accurately coordinated, safe operations.

[0008] Currently, location information is obtained in a variety of different ways.

[0009] Manned air vehicles traditionally use radio navigation aids such as radio beacons and radar control by air traffic control for navigation.

[0010] Other methods of determining position data for air (and land or water) vehicles include the use of satellite navigation systems such as the Global Positioning System (GPS) or Global Navigation Satellite System (GNSS), which triangulate the position of a transceiver on the vehicle via satellites, as described by Abulude et al. (2015): “Global Positioning System and its wide applications” Continental J. Information Technology 9(1):22-32, 2015.

[0011] Autonomous aerial vehicles such as drones may also navigate autonomously using GPS / GNSS receivers and other systems such as inertial navigation systems (INS), LiDAR scanners, ultrasonic sensors, and / or visual cameras. Other techniques include simultaneous localization and mapping, which involves creating a map of the drone's surroundings in real time and simultaneously updating the map with the drone's location information.

[0012] Driverless vehicles often have on-board cameras that capture real-time video streams of the vehicle's environment and process them in real time using image processing techniques such as AI to determine the relative positions and velocities of other objects / vehicles around them for appropriate and safe operation. A survey of various methods is presented in Gupta et al., (2021): "Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues."

[0013] While many of these techniques are well developed, their accuracy and / or ability to operate in near real time tends to be limited. For example, the U.S. government makes available the Global Positioning System Wide Area Augmentation System (GPS-WAAS) for aviation purposes with an accuracy of 1.6 m (horizontal and vertical). For details, see Table 3.3-1 of the "Global Positioning System Wide Area Augmentation (WAAS) Performance Standard," First Edition, published by the U.S. Department of Transportation on October 31, 2008. See also the citation of Volpe (2016) in the Background section above. Additionally, it should be noted that GPS systems generally experience reduced accuracy near buildings, trees, and other high topographical features that can cause signal reflections.

[0014] While this level of accuracy is sufficient for many purposes, it limits, for example, the performance of fleets or group drones operating dynamically in close proximity to one another or in complex formations, as well as the ability of ships and other vessels, for example, to conduct operations at close range. Summary of the Invention [Problem to be solved by the invention]

[0015] For these reasons and others discussed below, more precise positioning systems are desirable for use in the precise operation and coordination of multiple vehicles (such as fleets of drones, driverless automobiles, and autonomous surface vessels). [Means for solving the problem]

[0016] Thus, according to a first aspect of the present specification, there is provided a method for obtaining position data of a second object, implemented by a controller of a first object, comprising repeating the steps of: i) receiving infrared (IR) signals emitted by one or more IR emitters on the second object; and ii) using a pattern created by the sensed IR signals to determine relative position data of the second object. Steps i) and ii) are repeated to obtain a stream of position data of the second object.

[0017] Thus, in embodiments herein, a pattern of IR signals emitted by one or more emitters on a second object is used by a first object to determine location information of the second object. This is because the one or more signals vary as a function of distance (e.g., 1 / d 2 The present invention operates on the principle that the first and second emitters are received at an intensity (decreasing as the second emitter approaches the second object) and / or create different geometric patterns according to the viewing angle and the distance between the objects. The created patterns can then be used to determine both the distance to a second object and the orientation of the second object, particularly but not exclusively when the positions of the emitters on the objects are known in advance relative to one another.

[0018] The location of IR signals can be determined with much greater accuracy compared to traditional methods (e.g., GPS, etc.). While current technology is estimated to be accurate to better than 10 cm at a range of 100 meters, current NIR technology is on a rapid growth curve, improving accuracy over time and resulting in more accurate positioning systems and therefore improved maneuverability and coordination of mobile objects. The methods herein can be used by multiple objects to maintain constant (e.g., near real-time) situational awareness of relative position data (e.g., in up to six degrees of freedom) of other objects or vehicles in their vicinity, without requiring wireless RF communication between them or assistance from an external Positioning, Navigation, and Timing (PNT) system such as a Global Positioning System (GPS). In fact, the present disclosure adds a new, independent PNT functional layer to those existing systems, making the combination more robust, reliable, and accurate. It should be understood that the above example is a much more accurate location system than is possible using GPS.

[0019] In embodiments herein, steps i) and ii) may be repeated continuously or near-continuously on the real-time IR signal to obtain a real-time stream of position information of the second object.

[0020] In embodiments where the first and second objects are moving objects, the stream of position data can be used to cause the first object to create a trajectory for the second object based on the stream of position data for the second object. Thus, the methods herein enable highly accurate tracking of moving objects.

[0021] In some embodiments, the method includes causing the first object to follow the second object by maintaining a fixed spacing and / or orientation between the first and second objects based on the stream of position data, thereby enabling precise (and automated) coordinated movement of the first and second objects.

[0022] In some embodiments, the method includes using the stream of position data of the second object to cause the first object to maintain a spacing between the first object and the second object greater than a first threshold spacing, and / or maintaining a spacing between the first object and the second object less than a second threshold spacing, the second threshold spacing being greater than the first threshold spacing. Thus, the stream of position data can be used to maintain a fixed or near-fixed positioning between the first object and the second object (e.g., within a first and second threshold distance). Thus, embodiments herein enable the first object to autonomously follow the second object in real time, even at close range.

[0023] In some embodiments, the method further includes using the stream of positional data to cause the first object to perform a coordinated operation with the second object. For example, one or more IR emitters may be located on a hose protruding from the second object, and the operation may be a refueling operation, in which the first object connects to the hose and receives fuel from the hose. The stream of positional data of the emitters on the hose is used to guide the first object to a position for connecting with the hose. In this manner, refueling can be performed autonomously, safely, and reliably, even when the first and second objects are airborne.

[0024] In some embodiments, the method includes using the pattern of the sensed first plurality of IR signals to classify the second object according to type, such as the type of object (e.g., automobile, drone, aircraft), the make or model of the object, etc. In this manner, the pattern can be used to identify the object.

[0025] In some embodiments, the method includes determining a position of a laser optical communications transceiver on the second object based on the determined position data of the second object. The method may further include initiating free-space laser optical communications with the second object by commanding a pointing mechanism to align the transceiver on the first object with the transceiver on the second object using the determined position of the laser optical communications transceiver. In this manner, the high-precision, real-time spatial positioning methods described herein can be used to conduct free-space laser communications even between pairs of moving objects, such as pairs of drones or other aircraft.

[0026] In some embodiments, the first object is stationary and forms part of the road infrastructure, and the second object is a vehicle on the road. In such embodiments, free-space laser optical communication may be used to transfer data to instruct the vehicle to perform a maneuver or to provide traffic data to the vehicle. Thus, high-precision location information obtainable using the methods herein may be applied to "smart road" communication between roadside infrastructure and passing vehicles.

[0027] In some embodiments, the method further includes repeating the steps of receiving infrared (IR) signals emitted by one or more IR emitters on the third object and determining position data of the third object using a pattern created by the received IR signals to obtain a stream of position data for the third object. The method may then further include causing the first object to perform coordinated operations with the second and third objects using the streams of position data for the second and third objects. Thus, the method may be performed with respect to multiple objects to facilitate tracking and / or coordinated movement among a group or fleet of objects, such as drones.

[0028] In some embodiments, the pattern relates to one or more of the size or luminosity of the IR signal as seen by the first object, the spatial arrangement of the IR signal as seen by the first object, the frequency of the IR signal, and a code embedded in the IR signal.

[0029] In some embodiments, step ii) comprises determining a transformation between the coordinates of the IR signal and a known geometric configuration of the IR emitters on the second object.

[0030] In some embodiments, step ii) includes comparing the pattern to a plurality of stored patterns, each stored pattern in the plurality of stored patterns representing an exemplary object at an exemplary known location, the method further including determining location data based on the closest matching stored pattern and the corresponding known location of the closest matching stored pattern.

[0031] In some embodiments, step ii) includes predicting the position data of the second object using a neural network trained on the previously sensed IR patterns and corresponding known position data of the previously sensed IR patterns.

[0032] In some embodiments, the method includes moving the first object closer to the second object in response to detecting a noise level in the first plurality of IR signals that exceeds a first threshold level or a change in light intensity that exceeds a first threshold change level. Thus, the noise level threshold may be used (e.g., in various weather conditions) to ensure that the two objects remain safely within range of each other.

[0033] In some embodiments, the position data includes one or more of an orientation of the second object, a position of the second object, a direction to the second object, a distance to the second object, a direction to a point on the surface of the second object, and a pitch, roll, and / or yaw of the second object.

[0034] In a second aspect, there is a controller including one or more processors collectively configured to perform the method of the first aspect.

[0035] In a third aspect, there is provided an object including at least one IR sensor for sensing IR signals and the controller of the second aspect. The object may further include one or more IR emitters for emitting IR signals. This can facilitate mutual tracking of the first object and the second object.

[0036] In some embodiments, the object may include at least four emitters that are offset in at least two different geometric planes, allowing the object to be tracked within a three-dimensional volume.

[0037] In some embodiments, the object is an air vehicle, a land vehicle, or a water vehicle. In any of the embodiments herein, the objects herein can be either manned or unmanned.

[0038] In some embodiments, the object further includes a laser optical transceiver for free-space laser optical communication and a pointing mechanism for changing the direction in which the laser optical transceiver points.

[0039] According to a fourth aspect, there is provided a first object including a memory including instruction data representing an instruction set, and a processor in communication with the memory and configured to execute the instruction set. The instruction set, when executed by the processor, causes the processor to repeatedly: i) receive infrared (IR) signals emitted by one or more IR emitters on a second object; and ii) determine position data of the second object relative to the first object using a pattern created by the received IR signals. Steps i) and ii) are repeated to obtain a stream of position data for the second object.

[0040] The first object may be further configured to perform any of the method embodiments of the first aspect.

[0041] According to a fifth aspect, there is provided a system including a first object and a second object. The first object is configured to perform the method of the first aspect to obtain a first stream of relative position data for the second object. The second object is configured to perform the method of the first aspect to obtain a second stream of relative position data for the first object. In this manner, mutual tracking is facilitated.

[0042] According to a sixth aspect, there is a computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of the first aspect.

[0043] According to a seventh aspect, there is a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of the first aspect.

[0044] The first and second objects may generally be any type of vehicle, such as a manned or unmanned air vehicle, a manned or unmanned water vehicle, or a manned or unmanned road vehicle, for example a driverless car.

[0045] The first and second objects may be first and second drones. The drones may be part of a fleet or group of drones, and the method may be used to coordinate the fleet or group of drones.

[0046] The method may be performed simultaneously by both the second object and the first object and any subsequent objects, thus facilitating mutual tracking and coordination of multiple objects.

[0047] Exemplary embodiments are described herein with reference to the following drawings: [Brief explanation of the drawings]

[0048] [Figure 1a] FIG. 1 is a schematic block diagram illustrating an example controller according to some embodiments herein. [Figure 1b]FIG. 1b is a schematic block diagram illustrating connectivity between the controller of FIG. 1a and other electronic components on the first object. [Figure 2] 1 is a schematic diagram illustrating an example object including a controller according to some embodiments herein. [Figure 3a] 1 is a flow diagram illustrating an example method implemented by a first object to obtain a stream of relative position data of a second object, according to some embodiments herein. [Figure 3b] FIG. 3B is a block diagram illustrating a first alternative method for performing step 304 of FIG. 3A, i.e., determining position data of a second object using the pattern produced by the received IR signal. [Figure 3c] FIG. 3B is a block diagram illustrating a second alternative method for performing step 304 of FIG. 3A, i.e., using the pattern created by the received IR signal to determine position data of a second object. [Figure 3d] FIG. 3B is a block diagram illustrating a third alternative method for performing step 304 of FIG. 3A, i.e., using the pattern produced by the received IR signal to determine position data of the second object. [Figure 4] 1A and 1B show exemplary configurations of receivers and emitters on respective exemplary first and second objects, and exemplary patterns detectable by the first object using said configurations. [Figure 5a] FIG. 1 is a schematic diagram illustrating the application of the methods described herein to the mutual tracking of a pair of airborne vehicles. [Figure 5b] 5b is a flow chart illustrating a method performed by the vehicle of FIG. 5a. [Figure 6a] FIG. 1 is a schematic diagram illustrating the application of the methods described herein to a refueling operation between two aircraft in motion. [Figure 6b] 1 is a flow chart illustrating a method for conducting a refueling operation between two moving aircraft. [Figure 7a] FIG. 1 is a schematic diagram showing two aircraft tracking each other using the methods described herein. [Figure 7b]7b is a flow chart illustrating a method for mutual tracking of the aircraft of FIG. 7a. [Figure 8a] 1 is a schematic diagram showing two vessels in a convoy at sea using the mutual tracking and orientation measurement methods described herein. [Figure 8b] 8b is a flow diagram illustrating the method of mutual tracking and orientation measurement of FIG. 8a. DETAILED DESCRIPTION OF THE INVENTION

[0049] As noted above, the disclosure herein relates to moving objects, such as vehicles and / or objects that interact with or operate in relatively close proximity to other moving objects, and obtaining precise position information for such moving objects to facilitate processes such as precise tracking, complex manipulation, enhanced cooperative capabilities and / or close-range manipulation.

[0050] The systems and methods described herein enable vehicles and other objects to detect, identify, and track one another while moving dynamically in relatively close proximity. More particularly, but not exclusively, the present disclosure is directed to improvements in or relating to systems and methods for mutual detection and tracking of manned, remotely controlled, semi-autonomous, or autonomous vehicles, including road vehicles, aircraft, drones, and watercraft, operating in relatively close formation. These improvements ensure that two or more vehicles moving in relatively close proximity can at least constantly measure each other's relative position and orientation in near real time and therefore track each other with high accuracy. This has many advantages. For example, in the case of a drone or aircraft, these precise real-time tracking systems and methods can be combined with its flight control systems and methods to ensure that safe separation is always maintained, ensure that maximum separation is not exceeded, and perform coordinated maneuvers that would not normally be possible.

[0051] 1a shows a controller (e.g., a computing controller) for an object. The controller may be included within the object for use in controlling the movement or other functionality of the object.

[0052] Controller 100 may generally be configured (e.g., enabled) to perform any of the methods and functions described herein, such as methods 300, 600, 700, and 800 described below. Controller 100 includes a processor 102, a memory 104, and an instruction set 106. The memory holds instruction data (e.g., compiled code, etc.) representing instruction set 106. The processor may be in communication with the memory and configured to execute the instruction set. The instruction set, when executed by the processor, causes the processor to perform any of the methods herein, such as methods 300, 600, 700, and / or 800 described below.

[0053] The processor (e.g., processing circuitry or logic) 102 may be any type of processor, such as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or any other type of processing device. The processor 102 may include one or more sub-processors, processing units, multi-core processors, or modules configured to cooperate in a distributed manner to control the controller as described herein.

[0054] The controller 100 may include a memory 104. In some embodiments, the memory 104 of the controller 100 may be configured to store program code or instructions that may be executed by the processor 102 of the controller 100 to perform the functions described herein. The memory 104 of the controller 100 may be configured to store any data or information referred to herein, such as, for example, requests, resources, information, data, signals, etc., as described herein. The processor 102 of the controller 100 may be configured to control the memory 104 of the controller 100 to store such information.

[0055] In some embodiments, controller 100 may be a virtual controller, such as a virtual machine or any other container-type computer controller. In such embodiments, processor 102 and memory 104 may be part of larger processing and memory resources, respectively. The controller is configured to acquire a data stream of relative position data of the second object by repeatedly performing the following steps: i) receiving infrared (IR) signals emitted by one or more IR emitters on the second object, and ii) determining relative position and orientation data of the second object using a pattern created by the sensed IR signals (which may include the location of the sensed signal and other measurements such as received signal strength). As described in more detail below, controller 100 may be configured to cause the first object to move relative to the second object (e.g., by sending signals or commands to associated dynamic components of the first object) based on the acquired data stream of relative position data of the second object. In this manner, the controller may cause the first object to track and / or manipulate in response to the second object. The use of such IR signals is highly accurate and therefore facilitates highly precise manipulation between a first object and a second object.

[0056] It should be understood that controller 100 may include components other than those shown in Figure 1a. For example, controller 100 may include a power source (e.g., mains power or battery power). Controller 100 may further include a wireless transmitter and / or wireless receiver for wirelessly communicating with other computing controllers and / or sensors on objects, such as an IR sensor for detecting IR signals. In some embodiments, controller 100 may have a wired connection for communicating with other computing controllers.

[0057] The controller may be included in (e.g., form part of) a first object. The controller may be used to control the first object to move in conjunction with or relative to a second object. A first object as described herein may be a vehicle. Examples of vehicles include, but are not limited to, air vehicles (such as airplanes, drones, helicopters, airships, gliders, and / or any other air vehicle), land vehicles (such as manned or unmanned automobiles, trucks, motorcycles, vans, and / or any other road vehicle), and water vehicles (such as boats, container ships, liners, sailboats, and / or any other water vehicle). Vehicles herein may generally be manned or unmanned vehicles.

[0058] The controller 100 may interface with other components on the first object, as shown in FIG. 1b, such that the processor 102 receives IR sensor data from an IR sensor 204 (also referred to herein as a receiver or detector). The IR sensor 204 is used to receive signals emitted by one or more IR emitters on the second object. The processor may also further control the one or more IR emitters 202 on the first object to emit IR signals receivable by the second object (e.g., for use in mutual tracking). In such an embodiment, the processor 102 may control the intensity, radiation frequency, and / or any code or pulse emitted by the IR emitter 202. The processor 102 may further interact with a navigation control module 108 to control the movement (e.g., trajectory and speed) of the first object based on the sensed stream of position information obtained regarding the second object.

[0059] An exemplary first object in the form of a drone 200 including controller 100 is shown in FIG. 2. Drone 200 further includes four IR emitters 202a, 202b, 202c, and 202d positioned on the nose, tail, and wingtips of drone 200. Additionally, in this example, drone 200 further includes receivers 204a, 204b, and 204c for receiving IR signals from other drones. In this manner, drone 200 is configured to determine streams of position data of other drones (e.g., according to method 300 below) and also to emit IR signals to other drones for use in mutual tracking and coordination operations.

[0060] The second object may also be a vehicle, such as any of the types of vehicles described above with respect to the first object. The first object may be the same type of vehicle as the second object; for example, the principles described herein may apply to first and second drones, first and second ships, first and second unmanned automobiles, and / or any other vehicle pair. Similarly, the first and second objects may be different types of vehicles, such as drones that interact with a ship or automobile (e.g., follow the ship or automobile, pursue the ship or automobile, land on the ship or automobile, or perform some other operation on the ship or automobile).

[0061] Alternatively, one of the first object and the second object may be a stationary object associated with, for example, infrastructure. As an example, the second object may be an unmanned vehicle (e.g., a car or truck, etc.), and the first object may be part of roadside infrastructure, such as a gantry, sign, or other roadside infrastructure. The roadside infrastructure may use the methods described herein to determine a real-time data stream of location information for the unmanned vehicle or vehicles passing by it. As described in more detail below, such location information may be used to conduct free-space optical communications or other forms of communication with the vehicles, for example, to coordinate or otherwise direct the vehicles.

[0062] As another example, one (or both) of the objects may be a person. For example, the first object may be a vehicle and the second object may be a person, such as a road worker. In such an example, the methods herein may be used by the vehicle to detect the movement of the person to avoid a collision.

[0063] As another example, the second object may be a moving part of a stationary object, for example a moving structure such as a wind turbine.

[0064] The second object may be a structure that should not move, and the methods herein may be used to determine whether (undesired) movement has occurred. For example, in one embodiment, the first object is a first drone that uses the methods described herein to monitor a second object in the form of a building or other structure, and monitors movement of said building or structure due to, for example, an earthquake or subsidence.

[0065] In general, the methods herein are well suited to monitoring any "collaborative," "designed," or "anticipated" relative movement between two objects (e.g., where the owners or designers of the objects wish to coordinate with each other).

[0066] As described above, the first object has at least one IR sensor for sensing IR signals emitted by an IR emitter on the second object. As an example, the IR sensor (which may also be called an IR receiver) may include one or more IR cameras that can be used to capture the signals and generate a photograph or image of the pattern. IR cameras and / or video equipment may also be used to capture IR signal patterns in a video stream.

[0067] The IR sensor can be monocular, binocular, or multi-eye. While monocular sensors can be used in any embodiment herein, binocular and multi-eye sensors offer the benefit of additional tracking accuracy over monocular sensors.

[0068] In embodiments where the emitter emits pulsed or coded radiation, these emissions can also be detected using an IR camera. As an example, a camera frame rate of, say, 100 Hz can decode a signal of up to 100 Hz. In this case, each video frame will recognize either an "on" or "off" from the emitter.

[0069] The IR sensor may be positioned on a first object so as to be able to sense IR signals emanating from a second object at any position relative to the first object (e.g., to enable full-spherical field-of-view sensing). However, the sensor may be positioned to suit a particular application, e.g., in an application where the second object is always within a particular hemispherical field of view of the first object, the sensor may be positioned to enable hemispherical sensing.

[0070] With respect to the second object, the number of emitters employed generally depends on the particular application and the number of dimensions in which the second object can move. For example, if the second object is constrained to a linear geometry (e.g., a train), a single IR emitter may be placed on the second object to enable tracking. As another example, if the second object is constrained to move within a 2D plane or along a known surface (e.g., a road vehicle), two emitters may be sufficient to determine position and orientation. However, if the first and second objects are free to manipulate in 3D space, at least four emitters that are not in the same planar plane may be used. Furthermore, it should be understood that additional emitters beyond the minimum number above may improve overall system performance.

[0071] In general, when multiple emitters are used, the positions of each IR emitter can be selected so that it is in a different geometric plane than the other IR emitters, which has the advantage of being able to create a wider variety of patterns from different viewing angles of each object.

[0072] In some configurations, the IR emitters may be located at or near the edges of each object. As an example, the IR emitters may be located on the wingtips and tail of an aircraft or drone. While locating the IR emitters at the edges is not required, such emitter placement has the advantage of making it easier to determine the extent of each object and also maximizes the potential variation in the observable pattern created across different viewing angles.

[0073] The IR emitter may be a broadband IR emitter, a narrowband IR emitter, or any other type of IR emitter. Emitters and sensors operating in other portions of the electromagnetic spectrum may be used. In principle, any portion of the IR or broader EM spectrum may be used. In some embodiments, near-infrared (NIR) is used, which may provide a higher signal-to-noise ratio, allowing for accurate detection of emissions at higher altitudes. In embodiments where two or more IR emitters are employed, the emitters may have different frequencies and / or emit pulsed radiation that may be added to the patterns described herein, allowing the emitters to be distinguished from one another.

[0074] In general, it should be appreciated that a wide variety of different arrangements of IR emitters and IR sensors may be employed depending on the requirements of the system and the accuracy of the relative position data that may be obtained in various scenarios.

[0075] In a scenario in which both a first object and a second object implement the methods described herein, e.g., in a mutual tracking scheme, both the first object and the second object may have both an IR emitter and an IR receiver. As an example, a small array of IR emitters and IR sensors may be deployed on each object or vehicle (e.g., the first object, the second object, and / or subsequent objects if the method is applied to a fleet of objects) as follows: An array of IR sensors on each object or vehicle provides the amount of spherical sensing coverage required for the application (up to full spherical coverage in some instances). An array of IR sensors on each object or vehicle provides the amount of spherical IR illumination required for the application (up to complete spherical IR illumination in some instances).

[0076] Generally, a sufficient number of IR emitters should be present at fixed, known locations on the second object or vehicle so that a sensor on the first object or vehicle can calculate parameters such as the direction to, distance to, and orientation of the second object or vehicle based on information collected by the sensor and, preferably, but not necessarily, knowledge of the locations of the emitters on the second object. This is typically done by sensing equipment on the first object, which includes a sensor for detecting the specific size and arrangement of sensed IR points received from the second object. This size and arrangement of the IR points forms a pattern that can be used to complement the direction vector measured by the sensor and geometrically quickly calculate parameters such as the distance to and orientation of the second object, as well as the direction to any point on the surface of the second object.

[0077] The use of IR emitters and sensors described herein is one example of a more general use case for electromagnetic (EM) radiation emitters and sensors operating at any frequency or across any frequency range, or combinations thereof. A distinctive, beneficial, and novel combination of features and attributes of IR, particularly near-infrared (NIR), emitters and sensors—for which mature technology currently exists with suitable size, weight, power, cost, performance, and reliability—enables sensors to determine the precise direction of target emitters in near real time at operationally useful ranges and across a wide range of operationally relevant environmental conditions (e.g., atmospheric, weather, spray, sunlight, darkness, etc.). High-resolution cameras with sensitivity in the NIR range, combined with appropriate lenses and filters, can form sensors capable of detecting properly tuned emitters and calculating the emitter's relative orientation with very high accuracy using computational resources (hardware and software) that are neither too complex nor too expensive. The combination of emitters on an object creates a pattern that can be similarly detected and analyzed in near real time using modest computational resources, thereby making the challenge of highly accurate relative and mutual tracking feasible for the first time. In the future, other technologies may emerge that allow these benefits to be obtained using other parts of the EM spectrum, such as optical and ultraviolet wavelengths.

[0078] It should be understood that the first and second objects may be equipped with additional equipment or hardware in addition to that described above. By way of example, in some embodiments, as described below, the first and second objects may be further configured to conduct free-space optical communications based on the determined position information streams and thus may be configured with optical communications transceivers. Such transceivers may employ pointing mechanisms, such as, but not limited to, a gimbal and mirror combination, to enable the transceivers to be pointed at one another continuously, in near real time, and with the required precision.

[0079] 3a, a method 300 for obtaining relative position data of a second object, performed by a controller of a first object, is shown. The method may be performed by the first object 200, as described above. In a first step, the method 300 includes i) receiving infrared (IR) signals emitted by one or more IR emitters on the second object, and ii) using a pattern created by the sensed IR signals to determine the relative position data of the second object. The method includes repeating steps i) and ii) to obtain a stream of relative position data of the second object.

[0080] Steps i) and ii) may be repeated continuously or near-continuously (e.g., within the limits of computing power suitable for incorporation into the first object or, if not, accessible to the first object) to obtain 306 a real-time or near-real-time data stream of relative position data. "Near real-time" in this sense is used to indicate that the relative position data is obtained as close to real-time as possible given the technical limitations of the computer system and the timing and performance requirements of the application, e.g., the time delays between transmitting, receiving, and processing IR signals associated with the demands of high performance acrobatic flight maneuvers.

[0081] It should also be appreciated that if periodic measurements, which may have latency, are sufficient for a particular application, steps i) and ii) may similarly be repeated in a periodic manner, which may have less real-time latency.

[0082] As mentioned above, in step i), the IR signals are received by one or more IR sensors (or receivers) on the first object (e.g., IR cameras, video equipment, etc.), which may be positioned to enable spherical sensing of the IR signals. Those skilled in the art will be familiar with how to process the IR signals to determine parameters such as the direction of the source of each IR signal, the relative positions of each IR signal, and their relative (and absolute) intensities.

[0083] In step ii), the pattern can be derived from various features of each IR signal, such as the absolute size, signal strength, or luminosity. These features depend on the strength of the IR signal emitted by the second object and the distance of the second object from the first object (e.g., 1 / d 2 (The relative luminance of the IR signals may also vary with the viewing angle and distance to the second object, creating a pattern. The viewing direction and angle of the first and second objects may also distort the relative spatial location of the IR signals (e.g., appearing further apart when viewed head-on than when at an angle), and thus the spatial distribution or arrangement of the IR signals (relative to the field of view of each IR sensor) may also contribute to the pattern.

[0084] Additionally, the IR signals from the emitters on the first object may be emitted at different frequencies, pulses, or codes, which may be used to distinguish the signals emitted from each respective emitter and thus may also contribute to the pattern of the IR signals received by the first object. In other words, the pattern may be a pattern of different frequencies or different codes of the IR signals in addition to or as an alternative to a pattern related to the intensity and / or placement of the IR signals.

[0085] The relative position data may be any type of data relating to any of the six degrees of freedom of the second object. Thus, the position data may indicate the position and / or orientation of the second object relative to the first object. In general, the relative position data may include one or more of the orientation of the second object, the position of the second object, the direction to the second object, the distance to the second object, the direction to a point on the surface of the second object, and the pitch, roll, and / or yaw of the second object.

[0086] As shown in Figures 3b-3d, the pattern can be processed in various ways to obtain relative position data. Briefly, the position data can be calculated by transforming the vector coordinates of the detected emitter with the known geometric configuration of the emitter array. In another example, the pattern can be compared to a plurality of stored patterns providing a library of different types of objects or vehicles to determine the type of second object and the orientation, direction, and distance of the second object from the first object. In another embodiment, when the distance and orientation are known, the sensed pattern can be provided to a neural network (part of the second object sensing device) that has been pre-trained with such sensed data. Based on its training, the neural network (typically in the form of a dedicated AI chip such as the A15 Bionic chip) can relatively quickly determine the distance and orientation for a wide range of potential objects.

[0087] More specifically, as shown in FIG. 3b, in some embodiments, a mathematical transformation may be determined (304a) between the known (actual) emitter geometry of the second object and the detected pattern in step ii) to determine relative position information. Additionally, the known luminous intensity of the detected emitter on the vehicle may be compared to the detected luminous intensity (e.g., 1 / d 2Additional or supporting information about distances can be inferred (according to). As of this writing, an introduction to the mathematics of forward transforms (transforming from a 3D real-world point cloud to a 2D image plane point set) is available in Scratchapixel 3.0 in the section "Computing the Pixel Coordinates of a 3D Point," available at the following URL: https: / / www.scratchapixel.com / lessons / 3d-basic-rendering / computing-pixel-coordinates-of-3d-point / mathematics-computing-2d-coordinates-of-3d-points.html. The process described herein can be modified to perform the above-mentioned transforms (the inverse of the transforms described in the references).

[0088] In another embodiment, as shown in FIG. 3c, the pattern detected in step ii) can be compared to a plurality of stored patterns (304b). The plurality of stored patterns can be a library of stored patterns or can be formatted as a lookup table, etc. Each stored pattern in the plurality of stored patterns represents an exemplary object at an exemplary known location. The patterns can be obtained by simulation, for example, by creating a model of the second object in a computer-aided design (CAD) program, adding an IR emitter position, and moving the model of the second object through various orientations, rotations, and distances to determine / simulate the pattern generated by the IR emitter at each location and distance. In another example, the stored patterns can be obtained via sensor readings taken by a pair of real objects at various known distances and orientations (e.g., measured using another measurement method). For example, exemplary patterns and relative position data can be obtained by moving the first and second objects through a series of preset maneuvers to build a library of various patterns taken at various locations.

[0089] It should be understood that the library or lookup table may be multidimensional; for example, the library may further include example patterns acquired under various (simulated or real) weather and / or atmospheric conditions. In such examples, the patterns may be accompanied by additional information, such as expected noise that may be associated with each particular position of the second object under various weather and / or atmospheric conditions. The library may also include patterns resulting from various IR emitter combinations or characteristics. For example, as described above, different IR emitters may be configured to emit at different frequencies, pulses, or codes, which may also be added to simulated or real data collection scenarios.

[0090] Additionally, patterns can be associated with different types of objects. For example, different types of objects (e.g., drones of different makes or models) can be configured with different configurations of IR emitters, for example, located in different locations (as described above), having different frequencies, or emitting different pulses or codes. Thus, the pattern can be further used to determine the type of object. This information can be another dimension of a multi-dimensional lookup table or can also be determined from the pattern.

[0091] In use, the controller can match the (real / detected) pattern obtained in step i) with patterns in a library or look-up table, which can be done by comparing each stored pattern in turn with the detected pattern.

[0092] It should be understood that the complete library of stored patterns is not necessarily searched in each iteration.

[0093] For example, it should be understood that the pattern created depends on the source intensity of the IR signal and the geometric configuration of the emitter on the second object. Thus, in some embodiments, the input to the matching algorithm is data related to an indication of the IR intensity signal and / or the geometric configuration of the emitter. In some embodiments, an input parameter to the matching step may be the type of object being tracked (which may be linked to the intensity and / or the geometric profile of the object). This information may be used to select a subset of patterns in a stored library or a subset of patterns in a multidimensional lookup table that corresponds to the IR signal intensity and geometric configuration of the second object.

[0094] As another example, the position of a second object at time t+1 may be assumed to be directly related to the position determined at time t. In other words, the second object may be assumed to move continuously through a series of locations and positions according to the laws of physics, rather than, for example, jumping instantaneously from one location to the next. Thus, if the stored patterns are stored in a multidimensional lookup table or library, the position at time t+1 may be expected to be found in a portion of the lookup table or library adjacent to or related to the position at time t.

[0095] The speed of the matching process can also be increased by extrapolating the position of the second object based on previous relative position data (e.g., speed, direction, orientation, etc.), for example, to predict a subset of neighboring portions of the lookup table that should be searched first.

[0096] Any possible double mapping of patterns (e.g., in scenarios where one pattern is associated with multiple sets of position information) can also be mitigated by searching only the most likely portion of the parameter space based on previous relative position data. Problems with double mapping can also be mitigated at the design stage, for example, using design principles of uniqueness requirements to ensure that each pattern only occurs with one type of object, for example, at one position and orientation.

[0097] Once a matching pattern (or closest matching pattern) is determined, relative position data is determined 304c from the closest matching stored pattern and the corresponding known location of the closest matching stored pattern. For example, this can be thought of as the corresponding known stored location of the closest matching stored pattern, or it can be calculated by transforming the vector coordinates of the detected emitter and the known geometric configuration of the array of emitters on the detected vehicle.

[0098] As mentioned above, the use of a (multidimensional) lookup table or library of stored patterns and direct calculation from the direction vector and the dimensions of the target array are just two ways to determine position data from a detected pattern. In another example, as shown in Figure 3d, in step ii), a machine learning model such as a neural network can be used to predict position data from the pattern (304d).

[0099] Those skilled in the art will be familiar with machine learning and how to train models using machine learning processes. However, briefly, a model, which may also be referred to as a "machine learning model," includes a set of rules or (mathematical) functions that can be used to perform tasks related to data input to the model. Models can be taught to perform a variety of tasks on input data. Examples include, but are not limited to, determining labels for the input data, performing transformations on the input data, making predictions or estimates of one or more output parameter values ​​based on the input data, or generating any other type of information that can be determined from the input data.

[0100] In supervised machine learning, a model learns from a set of training data that includes example inputs and corresponding ground truth (e.g., "correct") outputs for each example input. Generally, the training process involves learning the weight or bias values ​​of the model to tune the model to reproduce the ground truth outputs for the input data. Various machine learning processes are used to train different types of models; for example, machine learning processes such as backpropagation and gradient descent may be used to train neural network models.

[0101] A model herein may generally be any type of machine learning model that can be trained to receive as input a pattern of IR signals (or data indicative of such a pattern) and output a prediction of location data, as described above. Examples of suitable models include, but are not limited to, neural network models, linear regression models, and decision tree models.

[0102] In some examples, the model is a neural network. There are various possible combinations of input and output parameters. As one example, the input may be in the form of an image, e.g., an image showing an IR signal pattern. Such an image may be supplemented with additional data, such as the direction or orientation relative to the first object from which the image was taken. As another example, the input may be a list of vectors corresponding to the center points of each detected IR signal and their respective brightness values. In yet another example, the input may be raw data from an IR sensor. These are merely examples, and those skilled in the art will understand that other inputs may be provided in addition to or in place of the inputs described above. For example, any of the exemplary data types may be further supplemented with weather data, data regarding the atmospheric conditions in which the IR signals were transmitted and sensed, and / or any other data that may affect the strength or pattern of the detected IR signals.

[0103] The neural network can also receive as input data related to the second object's previous location information (such as its trajectory or last known position), which can improve the neural network's predictions because this information has a strong causal relationship to the second object's current location.

[0104] In terms of output parameters, the neural network may be trained to output any type of position data, such as any one or more of the following: orientation of the second object, position of the second object relative to the first object, position of the second object relative to a fixed point, direction to the second object, distance to the second object, direction to a point on the surface of the second object, pitch, roll and / or yaw of the second object.

[0105] The neural network can be trained to output other information about the second object that can be inferred from the pattern, such as the type of the second object (e.g., the type or make of drone, aircraft, vehicle, etc.), its size, or range.

[0106] As described above, a neural network can be trained using training data including example inputs and "correct" or ground truth outputs for the example inputs. A training dataset can be constructed using the same techniques described above to create a library of stored patterns. This is done, for example, by simulating an object at different positions, such as at different distances, angles, and orientations from a single viewpoint, and simulating the corresponding pattern for each position. A training dataset can be systematically constructed by sequentially sampling the complete "position" space available for a second object relative to a first object. In this way, a training set can be constructed that can train a neural network with high accuracy in a wide range of flight or object movement scenarios. It should be understood that if additional input parameters are provided, the training dataset can be expanded to sample possible positions and resulting patterns created by considering additional parameters (and parameter space) (e.g., patterns created by different weather and / or atmospheric conditions or the use of different IR emitters emitting pulsed or coded signals at different frequencies, etc.).

[0107] It should further be understood that the training data set can also be constructed from real measurements, for example, by commanding a first object and a second object to perform different operations and recording the resulting patterns and position data for use in constructing the training data set. It should further be understood that the training data set can also be comprised of a combination of real and simulated data.

[0108] Appendix I shows several example rows of training data for the example system shown in FIG. 4. In this example, the second object 400 is an airborne object with four emitters 402a, 402b, 402c, and 402d arranged in a square configuration on its upper side. The first object 404 has IR detectors 406a and 406b on its nose and lower side, respectively. It should be understood that emitters 402a, 402b, 402c, and 402d appear differently at each sensor 406a, 406b, depending on the viewing angle and relative position. An example pattern corresponding to sensor 406a or 406b viewing emitters 402a-402d head-on (e.g., at a right angle) is shown in 408a. Other example patterns 408b and 408c correspond to a 45-degree angle, a 45-degree angle, and a 45-degree roll between the first and second objects, respectively.

[0109] In this example, the training data includes the following inputs: sensor reference number, relative position of the detected IR signal in each respective emitter's coordinate frame, and relative detected signal strength (compared to each other) of the detected IR signal detected by each emitter. The training data further includes ground truth values ​​for corresponding range, orientation, and roll, pitch, and yaw values ​​for each exemplary input. A complete training data set can be obtained, as described above, by using a CAD program or other simulation environment to systematically move a first object and a second object at various relative positions to build an unbiased training data set. A neural network (see below) can be trained to predict the ground truth values ​​in Appendix I from the ground truth input parameters. It should be understood that the example shown in FIG. 4 and Appendix I is only one example, and as described above, many variations are possible, such as having objects of different shapes, having different numbers and / or geometric configurations of emitters and receivers, emitting at different frequencies, and / or using pulsed radiation that may contribute to the pattern. Furthermore, many different combinations of neural network input and output parameters are possible as well, and the exemplary training data in Appendix I represents only one example.

[0110] There are various open-source neural network models suitable for use with the embodiments described herein, such as the neural network in scikit-learn described in Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011. Generally, the features herein can be obtained using the default neural network parameter settings described in the guidance.

[0111] Reference is now made to FIG. 5a, which illustrates one embodiment herein. In this embodiment, both the first object and the second object are autonomous aircraft or drones, although one or both may be semi-autonomous or manned. In the example of FIG. 5a, the autonomous aircraft has a standard fuselage-wing configuration. Here, it can be seen that IR emitters 10, 20 with rearward-facing fields are located at the wingtips and tail, and IR sensors 11, 21 with forward-facing fields of view 12 are located at the nose of the aircraft. This configuration enables aircraft or drone 1 to use its sensing equipment and implement method 300 to repeatedly (e.g., constantly or nearly constantly in near real time) calculate the direction to aircraft 2, the orientation (pitch, roll, and yaw) of aircraft 2, and the range to aircraft 2, and use this real-time information to augment its own flight control system to follow aircraft or drone 2 in a controlled and safe manner. Similarly, when both aircraft or drones are equipped with the same equipment, aircraft or drone 2 can follow aircraft or drone 1. With more assets, a sky train of multiple aircraft or drones may advantageously be operated autonomously, with only the lead aircraft or drone needing navigation capabilities, which, like any of the aircraft or drones described herein, may be autonomous, remotely piloted, semi-autonomous, or manned.

[0112] Thus, in this manner, the pattern can be used to very accurately determine or predict a continuous stream of position data of a second object in near real time.

[0113] FIG. 5b illustrates a method 500 implemented by the aircraft or drone shown in FIG. 5a. In this example, a first object, aircraft 1, repeats steps 302 and 304 to obtain a stream of (relative) position information for a second object, aircraft 2 (306). In this example, controller 100 may further cause the first object to follow the second object (502) by maintaining a fixed (or nearly fixed) spacing and / or heading between the first and second objects based on the stream of position data. In examples where first object, aircraft 1, is autonomous, this may include using the stream of relative position data to set a heading for autopilot control of aircraft 1 at a fixed offset from aircraft 2 according to the stream of position data.

[0114] Thus, method 300 can be used to obtain a real-time (or near-real-time) stream of position data that drones and other autonomous aerial vehicles can use to track (e.g., plot or monitor) another drone or aircraft. As described above, the first object may be stationary or moving. In embodiments in which both the first object and the second object are moving, method 300 facilitates high-precision (mutual) tracking of one moving object with another. Furthermore, method 300 can be applied to fleets of drones or other aerial vehicles. When each drone in a fleet or group performs method 300 on its leading drone, the fleet or group can perform coordinated operations based on the tracking.

[0115] In general, the controller 100 of the first object 1 may be configured to cause the first object 1 to follow or maintain a position relative to the second object 2 based on the stream of second object position data obtained using the method 300. The term "following" in this sense may mean maintaining a constant or fixed distance and / or a fixed orientation from the second object within a defined tolerance. As another example, the term "following" may mean maintaining a distance greater than a first threshold distance, e.g., maintaining at least a minimum distance from the second object. As another example, the term "tracking" may mean maintaining a distance less than a second threshold distance from the second object, e.g., staying within a fixed radius of the second object. As another example, the first drone may also maintain a distance from the second drone that is between a first threshold distance and a second threshold distance, e.g., between a minimum radius and a maximum radius.

[0116] Method 300 may be used by a first object to perform a coordinated operation with a second object. Those skilled in the art will appreciate that many operations are possible, such as programming a fleet of drones to perform dynamic light shows by coordinating operations based on the stream of position data output from method 300. Such operations may be performed with greater accuracy and speed using method 300 for determining position data compared to other methods, such as less accurate GPS.

[0117] Another example operation is a refueling operation, in which a first object is connected to a hose extending from a second object to receive fuel therefrom. In this example, one or more IR emitters can be positioned on the hose, allowing the first object to accurately determine the hose's location and move to the correct position for refueling, even if the second object is moving. Thus, using emitters and sensors, aerial refueling of an autonomous drone from an autonomous tanker may be possible. Again, either asset may be autonomous, remotely piloted, semi-autonomous, or manned. FIG. 6a shows an example in which an aircraft or drone 1 is refueling from a tanker 2 by attaching a probe 15 to a hose 24 via a basket 25. Positioning the IR sensor 11 near the probe and attaching the IR emitter 20 to the basket 25 allows the aircraft or drone 1 to autonomously approach the tanker hose and basket, with the emitter 20 within the field of view 13 of the sensor 11 so that the tracking information can be used by the aircraft or drone 1 flight control system to precisely steer the aircraft or drone 1 so that the probe 15 can engage the basket 20 and begin refueling.

[0118] FIG. 6b illustrates a method 600 that may be implemented by aircraft 1 and 2 as shown in FIG. 6a. In this example, a first object, aircraft 1, implements method 300 to obtain a stream of relative position data for a second object, aircraft 2. The first object then uses the stream of relative position data to cause the first object to perform a maneuver on aircraft 1 (602). In the example of FIG. 6a, as described above, the maneuver is a refueling operation. One or more IR emitters 20 are located on a hose protruding from second object, aircraft 2. During the refueling operation, first object, aircraft 1, connects to the hose of second object, aircraft 2, to receive fuel from aircraft 2. The stream of position data from the emitters on the hose is used to guide the first object to a position to connect with the hose.

[0119] As mentioned above, the hose of aircraft 1 and / or the inlet of aircraft 2 may be further equipped with a basket to connect aircraft 1 with aircraft 2 to facilitate the transfer of fuel. However, this is by way of example only, and aircraft 1 and aircraft 2 may include any other type of connection means or device to facilitate the transfer of fuel from aircraft 2 to aircraft 1.

[0120] It should further be appreciated that aircraft 1 may also be equipped with an IR emitter positioned, for example, in an inlet pipe or connection means of aircraft 1 for receiving fuel. In such an example, aircraft 2 may perform method 300 on aircraft 1, enabling mutual cooperation and operation between aircraft 1 and aircraft 2. Method 300 thus enables precise and efficient operation at close range.

[0121] In a further example of the present disclosure, as shown in FIG. 7a, aircraft may be enabled to track each other over a wide range of relative directions and orientations and relatively short ranges. Here, a first object, an aircraft or drone 1, is equipped with IR sensors above (upper) and below (lower) the fuselage, each IR sensor having a hemispherical field of view 12, possibly employing multiple IR cameras in each sensor assembly. Each aircraft is also equipped with at least four IR emitters 10 and 20 that act as reference markers at fixed, known locations, such as the nose, tail, and wingtips. In this case, it is advantageous for the at least four markers to be in different geometric planes.

[0122] The upper sensor 11 of the aircraft 1 is shown having a hemispherical field of view 12, with a corresponding field of view for the lower sensor. In this example, a second object, a cooperating aircraft 2, is shown as being within the field of view of the upper sensor of the aircraft 1, with the emitter 20 of the aircraft 2 being within a subset of that field of view 13. Distinguishing and specifically identifying each emitter may be beneficial to facilitate tracking, with each emitter being controlled to emit an identifying pulse code 14 that can be identified by the receiving sensor 11. Similarly, the aircraft 2 may also perform the method 300 reciprocally to track the aircraft 1.

[0123] FIG. 7b illustrates a method 700 that may be performed by one or both of the first object (aircraft or drone 1) and the second object (aircraft or drone 2) shown in FIG. 7b. In this embodiment, aircraft 1 performs method 300 to obtain a stream of position data for the second object. In this embodiment, in step 302, the first object receives infrared (IR) signals emitted by two or more IR emitters on the second object, as described above. In this embodiment, the first object performs step 304e to determine the position data for the second object using a pattern created by the received IR signals. In step 304e, the pattern relates to (e.g., consists of) the size or luminosity of the IR signals as seen by the first object, the spatial arrangement of the IR signals as seen by the first object, the frequency of the IR signals, and / or a code, such as pulse 14, embedded in the IR signals.

[0124] Then, in step 702, the method includes having the first object track the second object based on the stream of position data. The term "tracking" in this sense means creating or mapping a trajectory of movement made by the object 2. This tracking can be used by the first object, aircraft 1, to plan and coordinate its own flight path with the trajectory taken by the second object, aircraft 2. It should be understood that aircraft 2 may similarly be performing method 700 to obtain the trajectory of aircraft 1.

[0125] Generally, as noted above, the more markers at fixed, known locations and the more sensors in the complementary global sensing array, the more accurate or better the system can be in terms of both tracking accuracy and reliability, and the closer it can be to real-time. However, tracking can be achieved with fewer sensors and emitters as well, as in the example shown in FIG. 7a. Furthermore, the single sensor on the fuselage can be replaced with two sensors spaced far enough apart to provide the added tracking accuracy benefits of binocular sensing over monocular sensing.

[0126] Another example is mutual tracking and orientation measurement between two ships in a convoy at sea. When two ships operate in relatively close formation for the purpose of transferring goods or personnel, the range of relative movement is quite limited. In this case, mutual tracking and orientation measurement during coordinated operations can be achieved with one IR sensor 31, 41 on each ship and a small array of at least four IR emitters 30, 40, again assuming they are not all located in the same geometric plane. Figure 8a shows an example where the starboard sensor 31 of ship 3 has a required field of view 32 and the emitter 40 of ship 4 is within a subset 33 of that field of view.

[0127] In the embodiment shown in FIG. 8a, one or both of a first object, vessel 3, and a second object, vessel 4, perform method 300 for mutual tracking. FIG. 8b illustrates a method for mutual tracking and orientation measurement between two vessels in a convoy at sea, as shown in FIG. 8a. In this embodiment, the first vessel receives 302 infrared (IR) signals emitted by two or more IR emitters on the second vessel and determines 304 position data for the second vessel using a pattern created by the received IR signals. Steps 302 and 304 are performed repeatedly (e.g., in a loop) to obtain 306 a stream of position data for the second vessel. In step 802, a controller on the second vessel causes the first vessel to track the second vessel based on the stream of position data.

[0128] In this way, in these examples, and in the general case where there may be many cooperating objects or vehicles, each vehicle maintains constant (e.g., near real-time) situational awareness of the relative position and orientation of other objects or vehicles without requiring wireless RF communication between them and without the assistance of an external Positioning, Navigation, and Timing (PNT) system such as a Global Positioning System (GPS). In effect, the present disclosure adds a new, independent layer of PNT functionality to those existing systems, making the combination more robust, reliable, and accurate. It should be appreciated that the examples described above are much more accurate location systems than are possible using GPS.

[0129] As mentioned above, in some examples, the pattern detection process may also take weather and / or atmospheric conditions into account when determining second object position data from the pattern.

[0130] Another feature of the present disclosure is that the reliable detection range performance of the IR emitter-sensor combination is known in advance under various atmospheric conditions (e.g., weather, clouds, etc.) and lighting conditions (e.g., day or night, direct sunlight, etc.). Thus, degradation in detection performance (e.g., weaker or more erroneous received signals) observed as an object or vehicle moves to greater separation distances can be used in combination with object-to-object or vehicle-to-vehicle communication to notify and operate the object or vehicle to return to the desired operating range. For example, if a noise level exceeding a first threshold noise level is detected in the first plurality of IR signals, this may mean that the first object is moving toward the edge of the IR signal's range, and thus the controller may move or command the first object to move closer to the second object to maintain optimal separation and ensure optimal tracking. Another indication that the second object is moving out of the first object's range is that the IR signal's luminosity begins to flicker in an unexpected manner. This is another indicator that the second object is moving out of range or approaching the limits of the IR signal emitter and detector's functionality. In such a situation, in response to detecting a change in light intensity that exceeds a first threshold change level, the controller may move the first object closer to the second object.

[0131] As another example of a beneficial feature of the present disclosure, the highly accurate mutual tracking facilitated by the method 300 described herein allows each moving object or vehicle to communicate with each other using free-space laser optics methods, as each moving object or vehicle can maintain precise pointing of its partner's laser optical transceiver.

[0132] Those skilled in the art will be familiar with free-space optical communications, as described in Musa & Nelatury (2016): "Free Space Optical Communications: An Overview," European Scientific Journal 12(9):1857-7881. Briefly, free-space optical communications uses lasers to transfer data. It is a line-of-sight method and therefore requires accurate and reliable positioning of the emitter and receiver for successful data transfer. According to Musa & Nelatury (2016), cited above, optical transmission of data, voice, and video at up to 2.5 Gbps can be transmitted over air over long distances (4 km), enabling optical connections without the use of fiber optic cables or wireless network infrastructure.

[0133] According to the disclosure herein, free-space optical communications can be conducted between moving objects. This is achieved by having an optical communications transceiver at a fixed, known location and orientation on the object or vehicle within the same vehicle coordinate system as the fixed, known locations of the IR emitter and sensor. Thus, when a vehicle uses its IR emitter and sensor to track a precise direction and orientation to another vehicle, it can calculate in real time the precise bearing or direction to the other vehicle's laser optical transceiver. When the optical communications transceiver employs a pointing mechanism, such as, but not limited to, a gimbal and mirror combination, it can maintain its laser transceiver precisely pointed toward the other object's or vehicle's laser transceiver, and the other object or vehicle does the same. In effect, the mutual tracking functions are essentially independent, forming a dual system that realizes the benefits of increased reliability and availability, especially when tracking in one direction is aimed directly at the sun and can be subject to strong optical noise. In this situation, tracking in the opposite direction would be directly away from the sun.

[0134] This robust and accurate mutual tracking capability provides the significant benefits of free-space optical communications, such as high bandwidth, high security, and freedom from spectrum congestion and regulation, that can be realized for communications between moving objects or vehicles in relatively close proximity, typically up to 200-500 meters, over which range the well-known atmospheric attenuation problems associated with free-space optical communications are greatly mitigated.

[0135] Enabling free-space optical communication between moving vehicles may also extend to significant improvements to Applicant's motorsport and transportation system inventions described in WO 2021 / 051008 A, WO 2022 / 003343 A, and WO 2022 / 074406 A (the contents of which are incorporated herein by reference), whereby the addition of free-space optical communication between the vehicle and infrastructure can enhance IR tracking of moving vehicles using IR sensors mounted on infrastructure such as lampposts. In such an example, free-space optical communication between the vehicle and infrastructure can be used to communicate with the vehicle and provide instructions to the vehicle to perform maneuvers such as speed changes, lane shifts, etc. as part of an autonomous vehicle operation, or to provide warnings to the vehicle driver, e.g., road and / or traffic cameras, to alert them to potential problems that may follow. Such a system may be implemented by providing directional, steerable optical transceivers in both the vehicle and the infrastructure to provide the same capacity and security benefits as wireless RF communication technologies such as 4G / 5G and WiFi.

[0136] It should be understood that embodiments herein may generally be combined with one another. For example, the embodiments described above relating to a first object tracking and / or following a second object (as discussed with respect to FIGS. 5a and 5b, 7a and 7b, and 8a and 8b) may generally be implemented with the first object and second object further configured to perform a mutual operation (e.g., a refueling operation as discussed above with respect to FIGS. 6a and 6b).

[0137] It should further be appreciated that the embodiments described above with respect to Figures 5a, 5b, 6a, 6b, 7a, 7b and 8a, 8b may be realized in any of the ways described with respect to Figures 3b, 3c and 3d.

[0138] It should be further understood that the embodiments described above with respect to Figures 5a, 5b, 6a, 6b, 7a, 7b and 8a, 8b may be implemented using any of the various types of pattern data or combinations thereof referred to herein, such as, for example, geometric pattern data of the IR signal, relative signal strength, frequency and / or pulses or codes within the IR signal.

[0139] With reference now to other embodiments, it will be appreciated that method 300 may be embodied in a computer program. For example, a computer program product may include a computer-readable medium having computer-readable code embodied therein. The computer-readable code, when executed by a suitable computer or processor, may be configured to cause the computer or processor to perform a method (such as method 300) described herein.

[0140] A computer program may take various forms, such as, for example, source code, compiled code, executable code, or any other type of code. It should be understood that the source code of a computer program may be written in a variety of different programming languages ​​and may have different architectural designs. For example, the functionality described herein may be divided into various different subroutines. Furthermore, those skilled in the art will appreciate that many different ways of dividing functionality among different subroutines are possible. The subroutines may be stored together in a single executable file to form a self-contained program. Furthermore, the computer program may call external and / or standard libraries of computer code to perform specific subtasks related to the functionality described herein.

[0141] In another embodiment, there is a computer program product comprising a non-transitory computer readable medium having stored thereon a computer program as described above. Examples of the computer readable medium include, but are not limited to, a ROM, such as a CD-ROM, a semiconductor ROM, or a magnetic recording medium, such as a hard disk.

[0142] In another embodiment, there is a carrier that contains the computer program. Examples of carriers include, but are not limited to, electronic signals, optical signals, radio signals, computer storage media, etc. The carrier of a computer program may be any entity or device (e.g., hardware) capable of carrying a program. As an example, the carrier may be a computer-readable medium as described above. In another example, the carrier may be a transmissible carrier such as an electronic or optical signal, which may be conveyed via electrical or optical cable, or by radio or other means.

[0143] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these claims cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope. Furthermore, it should be understood that the features, advantages, and functions of the various embodiments described herein may be combined without departing from the spirit or scope of the disclosure herein.

[0144] [Table 1]

Claims

1. 1. A method for obtaining relative position data of a second object, performed by a controller of a first object, comprising: i) receiving infrared (IR) signals emitted by one or more IR emitters on the second object; ii) determining position data of the second object relative to the first object using the pattern created by the received IR signals; to obtain a stream of position data of the second object. A method comprising:

2. Both the first object and the second object are moving, and the method comprises: causing the first object to generate a trajectory of the second object based on the stream of position data of the second object; The method of claim 1 further comprising:

3. 3. The method of claim 1, further comprising causing the first object to follow the second object by maintaining a fixed spacing and / or orientation between the first object and the second object based on the stream of position data.

4. using the stream of position data of the second object to cause the first object to maintain a separation between the first object and the second object that is greater than a first threshold separation; and / or maintaining a spacing between the first object and the second object less than a second threshold spacing, the second threshold spacing being greater than the first threshold spacing; 10. The method of any one of the preceding claims, further comprising:

5. The method of claim 1 or 2, further comprising causing the first object to perform a coordinated operation with the second object using the stream of position data.

6. 6. The method of claim 5, wherein the one or more IR emitters are located on a hose protruding from the second object, the operation is a refueling operation, the first object connects to the hose to receive fuel from the hose, and the stream of position data of the emitters on the hose is used to guide the first object to a position for connecting with the hose.

7. 10. The method of any one of the preceding claims, further comprising classifying the second object according to type using the pattern of the sensed first plurality of IR signals.

8. determining a position of a laser optical communications transceiver on the second object based on the determined position data of the second object; using the determined position of the laser optical communications transceiver to initiate free space laser optical communications with the second object by commanding a pointing mechanism to align a transceiver on the first object with the transceiver on the second object; and 10. The method of any one of the preceding claims, further comprising:

9. The first object is stationary and forms part of a road infrastructure, and the second object is a vehicle on the road, and the free space laser optical communication transfers data to commanding said vehicle to perform a maneuver; or Providing traffic data to said vehicle The method of claim 8, used to perform the above.

10. iii) receiving infrared (IR) signals emitted by one or more IR emitters on a third object; iv) determining position data of the third object using the pattern produced by the received IR signals; to obtain a stream of position data of the third object; causing the first object to perform a coordinated operation with the second object and the third object using the streams of position data of the second and third objects; 10. The method of any one of the preceding claims, further comprising:

11. The pattern is the size or luminosity of the IR signal as seen by the first object; a spatial distribution of the IR signals as seen by the first object; the frequency of the IR signal, and A code embedded in the IR signal 10. A method according to any one of the preceding claims, relating to one or more of:

12. Step ii) is determining a transformation between coordinates of the IR signal and a known geometric configuration of the IR emitter on the second object; 10. A method according to any one of the preceding claims, comprising:

13. Step ii) is comparing the pattern to a plurality of stored patterns, each stored pattern in the plurality of stored patterns representing an example object at an example known location; determining the position data based on a closest matching stored pattern and the corresponding known location of the closest matching stored pattern; The method according to any one of claims 1 to 11, comprising:

14. Step ii) is predicting the position data of the second object using a neural network trained on previously sensed IR patterns and corresponding known position data of the previously sensed IR patterns. The method according to any one of claims 1 to 11, comprising:

15. 10. The method of any one of the preceding claims, further comprising: moving the first object closer to the second object in response to detecting a noise level in the first plurality of IR signals that exceeds a first threshold level or detecting a change in light intensity that exceeds a first threshold change level.

16. The location data is the orientation of said second object, the position of said second object, - a direction towards said second object, the distance to said second object, a direction to a point on the surface of said second object, and - pitch, roll and / or yaw of said second object 10. A method according to any one of the preceding claims, comprising one or more of:

17. 10. A controller comprising one or more processors collectively configured to carry out a method according to any one of the preceding claims.

18. at least one IR sensor for sensing IR signals; a controller according to claim 17; An object containing

19. 20. The object of claim 18, further comprising one or more IR emitters for emitting IR signals.

20. 20. An object according to claim 18 or 19, comprising at least four IR emitters offset in at least two different geometric planes.

21. An object according to any one of claims 18 to 20, which is an air, land or water vehicle and / or is unmanned.

22. a laser optical transceiver for free space laser optical communications; a pointing mechanism for changing the direction in which the laser optical transceiver is pointed; The object of any one of claims 18 to 21, further comprising:

23. a memory containing instruction data representing an instruction set; a processor in communication with the memory and configured to execute the set of instructions; wherein the set of instructions, when executed by the processor, causes the processor to: i) receiving infrared (IR) signals emitted by one or more IR emitters on a second object; ii) determining position data of the second object relative to the first object using the pattern created by the received IR signals; to obtain a stream of position data of the second object.

24. a first object; The second object and A system comprising: the first object is configured to perform a method according to any one of claims 1 to 16 to obtain a first stream of relative position data of the second object; A system wherein the second object is configured to perform a method according to any one of claims 1 to 16 to obtain a second stream of relative position data of the first object.

25. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 16.

26. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 16.

Images