Detecting objects in space

Abstract

A computer-implemented method for detecting a first object in a space (500) includes: i) receiving data regarding one or more detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a first marker positioned at a first point on the first object. The method then includes ii) determining the location of the first point in the space from the location of the detected signal in the received data. The space (500) may be a factory or an outdoor construction site where a large item (512) is being manufactured. The large item (512) has multiple markers on its surface that emit or reflect infrared (IR) radiation. IR sensors (204) are positioned at different points in the space and transmit data regarding the detection of the markers to an IR sensor control module (502) of the computer node (100). A positioning engine (504) determines the location of each point in the space corresponding to the marker from the location of the detected signal in the received data. The robot movement controller module 506 uses the determined locations to, for example, determine orientation and / or position / pose information for the robots 508a, 508b, 508c and determine actions to be performed by the robots. Markers may also be placed on people 510 working within the space 500 to track their movements in relation to the robots. Proximity alerts can be issued if someone gets too close to the automated machinery.

Description

[Technical Field]

[0001] The disclosure herein relates to object detection, monitoring, and tracking. The disclosure further relates to accurate object detection, monitoring, and tracking to improve efficiency, safety, and automation. The disclosure further relates to tracking and monitoring of robots, drones, other machinery, and workers, particularly in smart manufacturing, construction, and inspection environments. [Background technology]

[0002] The disclosure herein relates to tracking objects, such as robots, in factory environments. Various manufacturing environments use robotics. For example, the automotive industry, in particular, has applied robotics for decades to remove human operators from tedious, dirty, dangerous, and difficult tasks while reducing costs, improving quality, and speeding up production lines.

[0003] More recently, robots have been applied to work safely alongside humans as so-called "cobots." Factory robot tasks now extend beyond more traditional fabrication, assembly, and finishing lines to include material handling, internal logistics, inspection, packaging, shipping, and life-cycle maintenance.

[0004] A defining feature of robotics applications in manufacturing is the combination of mechanical grounding of the robot body, and therefore the robot arm, and fixation of the item being manufactured to the grounded robot body. Typically, a production line moves the manufactured item from one station to the next. This strategy allows both precise relative alignment of the manufactured item with the robot arm and precise control of the multiple degrees of freedom of the robot arm through pre-programmed motion instructions. Summary of the Invention [Problem to be solved by the invention]

[0005] As noted in the background section, robotic production lines currently tend to move articles, such as automobiles, from one manufacturing station to the next along a production line. However, as the size and complexity of the articles being manufactured increase, for products such as airplanes, trains, ships, submarines, and spacecraft, it becomes increasingly impractical to move articles along a production line frequently or quickly enough relative to mechanically grounded robots. Thus, it is becoming increasingly desirable to be able to deploy robots and "cobots" that are not fixed in location and can move efficiently around and within manufactured or constructed articles, whether in "positive" structures such as buildings or "negative" structures such as tunnels.

[0006] This logic naturally extends to many domains, including manufacturing, civil engineering, and agriculture, where the houses or goods to be made, bridges to be constructed, materials to be tunneled, land to be cultivated, etc., are virtually fixed.

[0007] Currently, the benefits of precision agriculture are widely enjoyed based on satellite imagery and the 10-cm accuracy that high-end GPS / GNSS navigation can provide for slow-moving machinery in relatively open agricultural spaces. However, this level of accuracy is simply not high enough because mobile plant, equipment, and machinery can operate at much higher speeds than agricultural machinery, often within factories or towns and cities where GPS is less reliable. For example, higher levels of positioning accuracy are needed in the robotic assembly of pre-fabricated building sections, where a local positioning system (LPS) is required that can provide accurate, near-real-time (low-latency) measurement of machine position in space, including precise measurement of major moving part / limb movements, in a cooperative, agile, and safe manner.

[0008] There are several existing local positioning system (LPS) technologies that are based on attaching radio frequency (RF) emitters to moving objects and placing RF receivers around a work site or factory that work together to calculate the emitter's location and trajectory. These have accuracy limitations when the movement of the object being measured involves high speeds or large accelerations. Thus, while they can track a slow-moving robot moving around a warehouse, they cannot track a fast-moving robot arm with the required accuracy and latency. There are also LPS technologies based on video camera image processing or photogrammetry that have latency and full computational complexity barriers, especially when the situation being measured is not purely repetitive.

[0009] Accuracy is not the only issue in these smart manufacturing and construction applications. To truly harness the potential of such systems, it is increasingly desirable for machines, robots, drones, etc. to be able to work independently, day and night, outdoors, and under a variety of weather conditions. It is also desirable for robots, drones, etc. to be able to work indoors and outdoors, above and below ground, and over long distances, such as those involved in large-scale construction, fabrication, inspection, and maintenance environments.

[0010] The embodiments herein aim to address, among other things, some of these issues. [Means for solving the problem]

[0011] According to a first aspect of the present specification, there is provided a computer-implemented method for detecting a first object in a space, the method including: i) receiving data regarding one or more detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from (e.g., reflected from or emitted from) a first marker disposed at a first point on the first object; and ii) determining a location of the first point in the space from a location of the detected signal in the received data.

[0012] In some embodiments, the narrowband signal is centered around about 800 nm and has a frequency range of about + / - 10 nm or a frequency range of about + / - 20 nm.

[0013] Thus, a method is provided for locating points on an object using infrared markers and narrowband infrared sensors. There are significant advantages associated with using narrowband detection in this manner. In particular, narrowband detection has a much higher signal-to-noise ratio than unfiltered wideband detection. This allows detection to be reliably performed over much longer distances (on the order of about 100 meters at the time of writing) with the high accuracy required for such applications (typically less than about 5 cm). These distances can be obtained during both day and night, even in adverse weather conditions. Measurements remain accurate even when the object is moving rapidly or accelerating. Furthermore, the markers (IR reflectors or emitters) in embodiments herein are generally at least as cost-effective as systems such as active RFID tags. Such systems can be used in a variety of scenarios, such as tracking humans or animals on stadiums or sports grounds.

[0014] These techniques can be further advantageously deployed in smart factory and smart manufacturing environments such as those mentioned above, where narrowband can be used to locate (with high accuracy) markers placed at specific points on people, machinery, robots, drones, etc., over long distances with high accuracy.

[0015] Thus, the systems and methods herein can be used to monitor the position of objects, such as manually operated machines, remotely controlled machines, autonomous machines and robots, cobots, drones, other mechanical systems, and human workers, with high accuracy, at any distance required, at night or day, and under a wide variety of weather conditions. High-accuracy position data over long distances can be used to send instructions to any of these object types for robotic collaboration with other robots, machinery, and / or people in industrial settings. Thus, described herein are LPS systems with sufficient accuracy at sufficiently long distances for use in the manufacturing of large or stationary objects, among other applications. The systems herein thus enable smart manufacturing of trains, airplanes, bridges, submarines, and the like, among other applications.

[0016] In some embodiments, steps i) and ii) are repeated iteratively to track the movement of the first point on the first object over time.

[0017] In some embodiments, the method further includes instructing an infrared emitter to emit pulses of infrared radiation into space, the pulses being emitted at a first pulsating frequency. Steps i) and ii) are then repeated for detection of reflections of each pulse from the first marker. By way of example, the pulses may be emitted at a pulsating frequency of about 60 to about 100 Hz. Using a pulsed IR emitter, as opposed to continuously turning on the IR emitter, is advantageous in that it reduces the energy used by the emitter while enabling highly accurate near-real-time tracking.

[0018] In some embodiments, steps i) and ii) are repeated for infrared radiation coming from two or more different markers located at two or more different points on the first object.

[0019] For example, in some embodiments, step i) includes receiving data regarding a point cloud of detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a plurality of markers disposed at a plurality of points on the first object, and step ii) may further include determining an orientation, position, or attitude of the first object in space from the point cloud.

[0020] In some embodiments, the method further includes using the stream of location, orientation, and position information for the first object to determine an action or operation to be performed by the first object, and transmitting a control signal to the first object to cause the first object to perform the action or operation. Accordingly, the methods herein can be used to automate the control of drones, machines, and the like in smart manufacturing and construction environments.

[0021] In some embodiments, the method further includes repeating steps i) and ii) to determine a location of the second point in space relative to a second marker placed on a second point on the second object, and determining a relative proximity of the first object and the second object using the determined locations of the first marker and the second marker. In some examples, the relative proximity is used to initiate a proximity alert or to send a command to stop or modify movement of the first object or the second object in response to determining that the relative proximity is below a first threshold proximity. Thus, the methods herein can be used to improve safety, such as in automated factory or manufacturing environments, especially when the second object is a person.

[0022] In some embodiments, the first object is a robot, a drone, a mechanical object, or a person. In some embodiments, the space is a construction site, a factory, or a field. In some embodiments, the first object is a building, a bridge, or a wind turbine. In some embodiments, the second object is a second robot, a second drone, or a person.

[0023] In some embodiments, the data in step i) is received from a drone, and the method further includes causing the drone to perform a measurement of the first object, using the determined location of the first point in space as a reference point for aligning the drone with the first object before the measurement is taken. Thus, in this way, the marker can be used by the drone to accurately and reliably locate a particular point or viewpoint relative to a large object, such as a wind turbine, from which the measurement of the first object will be taken. This can be used to ensure that measurements are taken in a repeatable and reliable manner so that the condition of the object can be monitored over time (e.g., for cracks, paint detection, structural changes, etc.).

[0024] In some embodiments herein, in step ii), triangulation is used to determine the location of the one or more markers.

[0025] In some embodiments, the method is carried out at night, outdoors, and / or under inclement weather conditions, and thus can be used to enable 24 / 7, all-weather manufacturing and construction.

[0026] According to a second aspect, there is a method for tracking a first object, the first object being a robot, drone, or other machinery in a manufacturing or construction space. The method includes: i) receiving data regarding detection, by at least one infrared sensor, of infrared radiation coming from a first infrared marker; and ii) determining a location of the robot, drone, or machinery in the space from a location of the detected signal in the received data. In some embodiments, the infrared sensor is a narrowband infrared sensor.

[0027] The use of infrared markers for use in manufacturing and construction has significant advantages. In particular, the use of infrared-emitting or -reflective markers to locate robots, drones, and other machinery allows the machinery to be used at night and in adverse weather conditions, thereby facilitating around-the-clock manufacturing processes. As previously mentioned, infrared light applied in this manner is accurate to a scale of less than 5 cm, even when the object is moving or accelerating. This is beneficial in many industrial sectors, such as automated factories, and civil engineering applications, such as the construction of buildings, bridges, and other infrastructure. Thus, systems and methods are provided for the accurate detection, monitoring, and tracking of objects in smart manufacturing, construction, and inspection environments.

[0028] According to a third aspect, there is a computer node comprising one or more processors configured to perform the method of the first aspect or the method of the second aspect.

[0029] According to a fourth aspect, there is a system for tracking a first object in space. The system includes a first marker disposed at a first point on the first object, at least one narrowband infrared sensor equipped with a narrowband infrared filter for detecting narrowband infrared radiation, and a computer node. The computer node includes one or more processors configured to: i) receive data regarding one or more detections made by the at least one narrowband infrared sensor of narrowband infrared radiation from the first marker; and ii) determine a location of the first point in space from a location of the detected signal in the received data.

[0030] In some embodiments, the narrowband infrared filter has a frequency range of about + / - 10 nm or a frequency range of about + / - 20 nm centered around about 800 nm.

[0031] In some embodiments, the marker comprises an infrared emitter, a reflective material, or a retroreflective material.

[0032] In some embodiments, the system further includes one or more infrared lamps configured to illuminate the space with infrared radiation such that the infrared radiation is reflected from the marker.

[0033] In some embodiments, the one or more infrared lamps are configured to emit pulses of infrared radiation at a first pulsating frequency, and the computer node is configured to repeat steps i) and ii) for each pulse.

[0034] In some embodiments, the computer node is further configured to perform the method of the first aspect.

[0035] According to a fifth aspect, there is a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of the first or second aspect.

[0036] According to a sixth aspect, there is a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of the first or second aspect.

[0037] Exemplary embodiments of the present specification are described with reference to the following drawings: [Brief explanation of the drawings]

[0038] [Figure 1a] 1 illustrates an exemplary computer node according to embodiments herein. [Figure 1b] 1 illustrates various components with which the computer node of FIG. 1a may interface. [Figure 2] 1a and 1b illustrate an example of how the computer nodes and components of FIGS. 1a and 1b can be used to track one or more objects in a factory environment. [Figure 3a] 1 illustrates an exemplary computer-implemented method for detecting a first object in a space, according to some embodiments herein. [Figure 3b] 3b illustrates an extension of the method of FIG. 3a to obtain a stream of position information for a first point on a first object, according to embodiments herein. [Figure 3c] 3b illustrates an extension of the method of FIG. 3a to a point cloud of detection of multiple markers placed on a first object, according to embodiments herein. [Figure 4a] 1 illustrates example markers on some example machinery and the resulting point cloud generated therefrom, according to an example herein. [Figure 4b] 4a in a variety of different locations and poses, and the resulting point clouds. [Figure 4c] 3c shows an extension of the method of FIG. 3c for use in controlling the manipulation of a first object. [Figure 5a] 1 illustrates an exemplary system in a manufacturing or construction environment, such as a factory, according to some embodiments herein. [Figure 5b] 5b illustrates an exemplary method performed by the computer nodes illustrated in FIG. 5a, according to some embodiments herein. [Figure 6a] 1 illustrates an exemplary application of the methods herein to monitoring large objects such as wind turbines. [Figure 6b] 3a illustrates an exemplary extension of the method shown in FIG. 3a for use in positioning a drone to take precise measurements of a first object. DETAILED DESCRIPTION OF THE INVENTION

[0039] Briefly, the techniques, systems, and methods described herein are particularly (but not exclusively) for use in tracking objects, e.g., mobile objects, in factory and / or construction environments. The objects may be in fixed environments and / or may be mechanically grounded objects. Some embodiments herein use infrared video imaging and near-infrared (NIR) cameras with NIR filters and / or narrowband IR filters and lenses at fixed locations around the site or factory, along with the attachment of matched NIR-emitting markers (LEDs) or retro-reflective markers to plant and / or machinery involved in manufacturing or construction, and / or to mechanically grounded products under construction. Such markers can also be attached to human operators working nearby.

[0040] In the case of NIR retroreflective markers, they can be effective under natural daylight conditions by providing an easily discernible marking that reflects and absorbs ambient light. They can also be effective outdoors at night and in closed or unlit factories in combination with NIR flood lamps installed near the detection camera. In the case of narrowband infrared filters, they are beneficial because they provide low signal-to-noise ratio detection, and therefore narrowband systems can be used to extend the detection range (even at night and / or in adverse weather conditions).

[0041] Embodiments herein use the detected reflections / emissions from such markers to determine the location, orientation, and position / pose of one or more objects to which they are attached. The methods herein use near-real-time computational methods (e.g., with the low latency necessary for closed-loop robotic control), such as triangulation or higher-order multilateration, that allow the reflections / emissions markers to be tracked continuously and with great accuracy (down to the <5 cm level). The systems and methods herein can be used to monitor movement and provide proximity warnings or stops, thereby increasing levels of safety in fields and factories, or they can be used directly in closed-loop robotic control applications to provide increased levels of automation and efficiency.

[0042] More particularly, with reference to FIG. 1a, in some embodiments, there is a computer node 100. The computer node 100 is for detecting (e.g., configured to detect) a first object in a space. The computer node 100 may generally be configured to perform (e.g., may function to perform) any of the methods and functions described herein, such as method 300 described below. The computer node 100 includes a processor 102, a memory 104, and an instruction set 106. The memory holds instruction data (e.g., compiled code, etc.) representing the instruction set 106. The processor may be in communication with the memory and configured to execute the instruction set. The instruction set, when executed by the processor, may cause the processor to perform any of the methods herein, such as method 300 described below.

[0043] Processor (e.g., processing circuitry or logic) 102 may be any type of processor, such as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or any other type of processing device. Processor 102 may include one or more sub-processors, processing units, multi-core processors, or modules configured to cooperate in a distributed manner to control a computer node in the manner described herein.

[0044] Computer node 100 may include memory 104. In some embodiments, memory 104 of computer node 100 may be configured to store program code or instructions executable by processor 102 of computer node 100 to perform the functions described herein. Memory 104 of computer node 100 may be configured to store any data or information referred to herein, such as, for example, requests, resources, information, data, signals, or the like, described herein. Processor 102 of computer node 100 may be configured to control memory 104 of computer node 100 to store such information.

[0045] In some embodiments, computer node 100 may be a virtual computer node, such as a virtual machine or any other containerized computer node. In such embodiments, processor 102 and memory 104 may be part of a larger processing resource and memory resource, respectively. Thus, in some examples, computer node 100 may be cloud-based.

[0046] It will be understood that computer node 100 may include components in addition to those shown in FIG. 1a. For example, computer node 100 may include a power source (e.g., mains power or battery power). Computer node 100 may further include a wireless transmitter and / or wireless receiver for wirelessly communicating with other computing nodes and / or sensors (e.g., IR sensors for detecting IR signals as described herein, etc.). In some embodiments, computer node 100 may have wired connections for communicating with other computer nodes, IR sensors, etc.

[0047] lb, in embodiments herein, computer node 100 may communicate (e.g., send and receive electronic messages) with one or more IR sensors 204 (sometimes referred to as IR receivers or IR detectors). In some embodiments herein, computer node 100 is further configured to send messages to one or more IR emitters 203, for example, to control when emissions occur.

[0048] Briefly, in embodiments herein, a computer node is configured to detect a first object in a space, the computer node is configured to receive data relating to one or more detections, by at least one infrared (IR) sensor, of infrared radiation from a first marker positioned at a first point on the first object, where the computer node is further configured to determine (information relating to) a location of the first point in space from a location of the detected signal in the received data.

[0049] As described in more detail below, in some embodiments herein, the first object is a robot, a drone, or any other machinery for use in a manufacturing or factory environment. Accordingly, the systems and methods herein can be used to track and subsequently send control commands to robots and the like in smart factory setups. Using infrared-based markers to track robots, machinery, drones, and people in manufacturing settings, both automated, semi-automated, and / or non-automated, offers various advantages. The determined location can be used to direct automated or semi-automated machinery and / or generate proximity alerts to reduce collisions. Infrared is particularly advantageous because it can be used at night (enabling "lights-out" manufacturing and around-the-clock construction) and in adverse weather conditions (enabling around-the-clock construction of bridges, buildings, and the like, regardless of the weather).

[0050] As described above, computer node 100 is for detecting (and monitoring) a first object in a space. The first object has a first marker attached to a first point on the first object. The marker can emit IR radiation (e.g., like a beacon) or can reflect IR radiation emitted by an IR emitter. In some embodiments in which the marker emits IR, the marker may include an IR bulb or IR radiation source. Thus, the marker may be an infrared beacon. As described below, in any of the embodiments herein, infrared may be replaced with near-infrared, and thus the marker may also be a near-infrared beacon.

[0051] In embodiments where the marker reflects IR radiation, the marker may include a reflective surface. In some embodiments, the marker includes a retroreflective material. The retroreflective material reflects the radiation at an angle of incidence θ i is the reflection angle θ r (e.g., θ i =θ r) which makes retroreflectors advantageous for object tracking. In embodiments in which the marker reflects IR radiation, there may be an infrared emitting source 203 (e.g., an IR bulb, lamp, flood lamp, or the like) that emits IR radiation into space (and reflects it back to the marker). In some embodiments, there may be more than one infrared emitting source. For example, there may be multiple infrared lamps that illuminate the space (in a manner similar to a flood lamp).

[0052] It will be appreciated that the first object may have more than one marker attached to it. In some embodiments, the first object has multiple markers attached to it at multiple different locations on the first object. Each marker may emit or reflect infrared radiation. In such an example, detection of the multiple markers forms a "point cloud" of different points, each representing a different point on the object.

[0053] The markers may be placed in different strategies, for example, they may be placed to form a triangular mesh configuration over the surface of the object, in which case the point cloud of the detections may be used to form a three-dimensional outline of the object's surface.

[0054] Markers may be placed on the mechanical joints of the first object, such that the markers can trace the position of the first object's movable limbs. In such an embodiment, drawing lines between such joints provides an outline of the first object's skeleton. Markers may also be placed on the object's limbs, such as the ends of mechanical limbs, to monitor the limbs and trigger a proximity alarm if the limbs move in a dangerous manner.

[0055] The number of markers on each object depends on the application. As an example, four markers in different planes may be used to track an object in three dimensions. In the general case, a mesh of at least four markers, not all in the same plane, may be used to determine the object's position and orientation in space, in addition to knowledge of the location of those points in the object's frame. If the object has moving parts, at least four markers may be placed on each moving component throughout the object. However, if the object's components have limited degrees of freedom (e.g., components that can only move up and down in a 2D plane), fewer markers may be required.

[0056] It will further be appreciated that different patterns or reflective properties may be used to indicate different locations on the first object or to identify particular markers. For example, the markers may be configured to emit or reflect different wavelengths of infrared radiation. As another example, where the markers include infrared emitters, they may be configured to emit different patterns of pulsed infrared radiation, such as a code.

[0057] In examples where there are multiple different objects each being tracked, different patterns or reflective characteristics may be used to distinguish between a first plurality of markers on a first object and a second plurality of markers on a second object. For example, the markers on the first object may be configured to reflect infrared radiation of a different wavelength (or a different narrowband) or with a different pattern than the markers on the second object.

[0058] In other embodiments, the pattern, the frequency of the emission / reflection, and / or the pulse code of the emission may further be used to identify the type of object being detected. For example, different patterns may be associated with different types of objects. As an example, patterns may be used to distinguish classes of vehicles (e.g., different domino patterns on the roofs of cars, HGVs, LGVs, buses, etc.). This may also be applied to excavators, dump trucks, scrapers, surfaces, etc. Furthermore, the emitters / reflectors may be configured to form barcodes, QR codes, or any other identifiable type of marker to identify individual machines.

[0059] In some embodiments in which the markers emit IR radiation, a first plurality of markers on a first object may be configured to emit pulses of IR radiation at a first pulsating frequency (e.g., having a first period between pulses). And a second plurality of markers on a second object may be configured to emit pulses of IR radiation at a second pulsating frequency. Pulsed emission has various advantages, including, but not limited to, the energy savings associated with periodically turning IR emitters on and off. Furthermore, timing may be adjusted so that the first plurality of markers emit (e.g., "on") while the second plurality of markers are "off," and vice versa. This can be used to group or distinguish markers within the first plurality of markers from markers within the second plurality of markers (e.g., to distinguish two point clouds).

[0060] In embodiments herein, the markers are detected by at least one IR sensor 204 (sometimes called an IR receiver or IR detector), which may include one or more IR cameras that may be used to capture the signal and generate a photograph or image of the pattern. IR cameras and / or video equipment may also be used to capture the IR signal patterns in a video stream at a defined frame update rate.

[0061] As described in more detail below, in any of the embodiments herein, the infrared sensor 204 may be a narrowband infrared sensor, and the received data may include one or more detections of a narrowband of infrared radiation from (e.g., emitted from or reflected from) a first marker. For example, the infrared sensor 204 may be fitted with a narrowband filter. In principle, any narrowband infrared filter may be used. In some examples, the narrowband signal is centered at about 850 nm and has a width (e.g., half-maximum frequency range) of about + / - 10 nm or about + / - 20 nm. The use of such a narrowband IR detector has several significant advantages, including an increased signal-to-noise ratio and streamlined detection of the emission or reflection from one or more markers. This increased signal-to-noise ratio enables detection of markers over much greater distances compared to broadband IR detectors. At current IR camera sensitivity levels (which will increase in the future), the detection range is on the order of 100 m. The use of narrowband infrared radiation can achieve a high signal-to-noise ratio under a variety of weather conditions, which is advantageous for outdoor construction, manufacturing, and agricultural applications, for example. It is also advantageous for construction scenarios, where the methods herein may be applied, for example, to the construction of homes, buildings, skyscrapers, bridges, etc. Narrowband infrared detection can occur at night, enabling around-the-clock construction projects. The narrowband may be centered around the peak transmittance of near-infrared radiation in air, further increasing the signal-to-noise ratio.

[0062] Additionally, it should be noted that in embodiments herein, there may be more than one IR sensor. For example, two or more sensors may be provided that are spatially offset from one another. The spatially offset sensors have different viewpoints, and the offset of the detected position of the first marker and the relative position of the IR sensor itself can be used to triangulate the relative and / or absolute position of the marker. For triangulation, at least two IR sensors may be used, although three or four may be used depending on the accuracy required for a particular application.

[0063] In embodiments in which different markers reflect at different wavelengths, the IR sensors may be configured with different filters to track different groups of markers (e.g., markers fixed to the first and second objects). As another example, different subsets of the IR sensors may be configured to detect at different wavelength bands.

[0064] In embodiments in which one or more markers include a reflective material for reflecting IR radiation, an IR emitter is also used to illuminate the space. Note that in embodiments in which the IR receiver is a narrowband receiver, any one or more IR emitters 203 in such embodiments may also include one or more filters for emitting in the same narrowband as the narrowband IR receiver or in a band that overlaps with the narrowband IR receiver.

[0065] In embodiments where one or more IR emitters 203 are used (e.g., the markers are reflective or retro-reflective markers), these may be co-located with the IR receiver. For example, a pair or IR emitter and IR receiver may be mounted on a lamppost or mast type arrangement. However, it will be understood that the emitter and receiver do not have to be co-located. It will also be understood that it is possible to have a movable emitter. As an example of a movable emitter, an IR emitter may be mounted on a drone for flexible illumination of a space.

[0066] In some embodiments where there is an IR emitter (and reflective marker) that emits into space, the IR emitter may be configured to emit pulses of IR radiation into space. For example, the pulses may be emitted at a frequency of about 60 Hz to about 100 Hz (e.g., about 10 ms). -1 ~approx. 16ms -1(Pulses emitted at intervals of . The receiver may be configured to receive the pulses (e.g., the shutter speed of the IR camera may be set to the same frequency as the IR emitter is emitting pulses and / or may be synchronized using either wired or wireless signaling). This may conserve energy while still allowing near real-time tracking of objects. In embodiments where the IR emitter emits into space like a floodlight, the energy savings are even more significant.

[0067] The term "space" is used herein to refer to the three-dimensional volume within which a first object is located or moves, or, in other words, the physical three-dimensional space within which a first object is located, used, or operates.

[0068] The first object can be any type of object. The first object can be stationary or moving. The first object can be automated, semi-automated, or fully controlled, for example, by a human operator or engineer.

[0069] In some embodiments, the space may be a manufacturing space, such as, for example, a factory, a warehouse, an outdoor construction site, or any other site where manufacturing takes place. In this sense, the space is the physical three-dimensional space or volume in which manufacturing takes place. In such embodiments, the first object may be a robot, a drone, machinery, or any other piece of equipment engaged in a manufacturing task. The first object (e.g., a robot or drone) may perform the manufacturing task in a fully automated, semi-automated, or manually controlled manner. The computer node 100 and / or method 300 may be used to transmit instructions, guidance, or more general information that may be used in the control process of the first object.

[0070] FIG. 2 illustrates an example in which space 200 is a construction site where a building 205 is being erected. The construction site includes drones and / or robots 201. The drones and robots are configured to move around the building as it is being constructed. In FIG. 2, the object under construction is a building 205, but it will be understood that this is merely an example and that object 205 may be any other object (e.g., a submarine, a bridge, a large vehicle, an airplane, etc.). In this example, drone 201 has marker 202 attached (e.g., drone 201 is an example of a first object as described herein). In this example, marker 202 includes a reflective material. IR emitter 203 emits IR radiation into space, which is reflected from the reflective material on marker 202 and detected by one or more IR radiation receivers 204. IR receiver 204 may be a wideband receiver or a narrowband receiver as described above. In this example, the IR emitter 203 and IR receiver 204 are mounted on a support (such as a lamppost or floodlight stand). In this manner, the IR emitter 203 can fill the space with IR radiation. It will be appreciated that additional markers may be positioned on the drone 201 to establish multiple point locations of the drone. It will further be appreciated that markers may be placed on the building under construction 205 and / or people 206 working in the space 200, as well as other drones and / or robots within the space 200. In this manner, the relative positions of one or more drones, people, and stationary objects within the space can be accurately tracked in real time.

[0071] In some embodiments, the space may be a construction site for a building or other civil engineering project, such as a bridge or other infrastructure. In such embodiments, method 300 may be used for automated or semi-automated construction. In such embodiments, the space is a construction site where a building, bridge, or other infrastructure is being constructed. In such embodiments, the first object may be a robot, a drone, a vehicle, or any other machinery or equipment engaged in a construction task. In embodiments where the first object is a structure, the structure may be "positive" (e.g., a building under construction or a road being laid) or "negative" (e.g., a foundation hole in the ground or a tunnel being dug).

[0072] It will be understood that the space may be a space within an object or a building. Thus, the space may be a three-dimensional space within a warehouse or factory. In other examples, the space may be a space within an object under construction, for example, the interior of a submarine, a train, or a building. In embodiments where embodiments herein are applied to the interior of an object under construction (e.g., the first object is a robot or drone working inside a large object under construction), a movable IR sensor and / or emitter, such as an emitter / receiver attached to a drone, may be deployed inside the object under construction.

[0073] In some embodiments, the space is a field or other agricultural space, and the first object is farm machinery, such as a tractor, combine, excavator, or any other agricultural machinery. Method 300, described below, can be used to detect the location of the machinery and provide instructions for automated movement of the machinery across the land. The high accuracy and long-range capabilities of the methods herein facilitate high-accuracy automated agricultural applications at night and in adverse weather conditions, thus supporting the provision of truly around-the-clock agricultural applications.

[0074] In some embodiments herein, a second object is also present in the space, which may be a second robot, a drone, or any other type of machinery or equipment.

[0075] In other embodiments, the second object may be a person operating in space (e.g., an engineer, builder, construction worker, etc.). In such embodiments, the method 300 described below may be used to coordinate safe operations between such a person and a first object operating in the space (e.g., machinery, a robot, a drone, etc.).

[0076] In some embodiments, the first object is a large manufacturing object (e.g., a submarine, a train, an aircraft, a spacecraft, or any other large manufacturing object, etc.) and the second object is a drone. In such embodiments, the space is a three-dimensional space in which the large object is being built.

[0077] In some embodiments, the first object is a structure, such as a building, a bridge, infrastructure, an onshore or offshore wind turbine, a power plant, a railroad track, or any other structure, and the second object is a drone or other robot. In such embodiments, the space is a three-dimensional space in which the structure and the drone are located and / or operate, respectively. The following method 300 can be used to enable a drone to locate a first point on a structure and perform precise measurements therefrom, for example, for purposes of building monitoring or external inspection of building quality. In this manner, the marker at the first point can serve as a reference point for remote, automated, and high-precision inspection of a structure, such as a building. Thus, IR markers, including but not limited to narrowband infrared markers, can be attached to completed construction for use as reference points for remote, automated, and high-precision inspection of the structure throughout its operational life. For example, the methods herein can be used in automated (e.g., operator-free) inspection of the external condition of offshore wind turbines using drones equipped with IR sensors, with the IR markers enabling the drone to return to the same inspection viewpoint with high accuracy, thus making it possible to compare the progression over time of, for example, cracks, paint deterioration, movement, etc.

[0078] Although examples are described herein using first and second objects in a space, it will be appreciated that the methods herein can be extended to locate, track, and provide real-time indication to any number of objects in a space.

[0079] Many embodiments herein relate to construction sites and manufacturing applications; however, it will be further understood that the techniques described herein are equally applicable to other scenarios in which objects are tracked. For example, in some embodiments, the first object is a person. Accordingly, embodiments herein can be used to track people or animals. As an example, the systems and methods herein facilitate accurate tracking of soccer players on a soccer field or horses on a racetrack. Thus, the above-referenced space may be a soccer field, a racetrack, or any other three-dimensional volume in which it is desirable to track people or animals.

[0080] It will be further understood that embodiments herein may be equally applicable to vehicles (e.g., the first object may be a race car). In such an example, the space is a race track. As another example, the first object may be a drone, and the space may be a flight area for the drone.

[0081] As another example, the first object may be a vehicle, and the space may be a portion of a road along which the vehicle travels. In such an embodiment, the vehicle may be fitted with a marker as described herein, and one or more IR sensors may be attached to roadside infrastructure, such as a lamppost or gantry. Method 300 may be used to track vehicles passing through a space within view of the roadside infrastructure. In such an example, the narrowband infrared filters described herein may be advantageously used to track vehicles over longer distances with greater accuracy compared to other methods of vehicle tracking. Thus, the methods herein may be used as part of a smart road infrastructure for monitoring and sending instructions to vehicles. Thus, the methods herein may be used in conjunction with the inventions described in International Publication Nos. WO 2021 / 051008 A, WO 2022 / 003343 A, and WO 2022 / 074406 A, the contents of which are incorporated herein by reference.

[0082] Embodiments herein may be applied to manned or unmanned, autonomous, semi-autonomous, or manual vehicles. Examples of vehicles include, but are not limited to, air vehicles (such as airplanes, drones, helicopters, airships, gliders, and / or any other airborne vehicles), land vehicles (such as manned or unmanned automobiles, large trucks, motorcycles, vans, and / or any other road-based vehicles), and water vehicles (such as boats, container ships, liners, yachts, and / or any other waterborne vehicles). Thus, the methods herein may be used to track and / or transmit command data to control vehicles in air, land, or underwater spaces.

[0083] Referring now to FIG. 3a, there is a computer-implemented method 300 for detecting a first object in a space, according to some embodiments herein. Method 300 may be executed by computer node 100 described above. Method 300, as described in the examples above, may be applied to the detection and monitoring of a wide variety of object types in a wide variety of spaces. Briefly, method 300 includes: i) receiving 302 data regarding one or more detections, performed by at least one infrared sensor, of infrared radiation coming from a first marker positioned at a first point on the first object. In a second step, the method includes ii) determining 304 a location of the first point in the space from a location of a detected signal in the received data. In some embodiments herein, in step 302, data is received from a narrowband infrared sensor, the data being indicative of one or more detections, performed by the narrowband sensor, of narrowband infrared radiation coming from the first marker.

[0084] The data received in step 302 indicates one or more detections of narrowband infrared radiation from (e.g., emitted from or reflected from) a first marker on a first object. The data may be obtained from two or more infrared sensors. In such an example, different infrared sensors may have different perspectives into space.

[0085] The data may be in the form of infrared images (such as near-infrared images) showing infrared photographs of the space. In examples where the data are infrared images, the method may include a pre-processing step, such as an image processing step (such as image segmentation) to identify a first marker in one or more images. In other examples, the data may include a file indicating the locations and / or intensities of infrared sources in the space. However, these are merely examples, and any other data regarding the relative positions of infrared sources in an image may be used as well.

[0086] In step 304, the method 300 includes ii) determining (304) a location of a first point in space from the location of the detected signal in the received data. The location of the first point in space can be determined using techniques such as triangulation or polylateration. Those skilled in the art are familiar with the principles of triangulation, which allows detection of a first marker made by two or more different IR sensors having different perspectives (or visual perspectives) into space to determine the location of the first marker based on the offset of the locations sensed by the different IR sensors. Thus, in this manner, the position of the first marker can be determined in an accurate and rapid manner, even over long distances, in poor weather conditions, or at night. It will be understood that while triangulation location measurements result in relative locations, the relative location data can be converted to absolute location data if the absolute locations of the IR sensors are known.

[0087] It will be appreciated that steps i) and ii) may be repeated iteratively to track the movement of the first point on the first object over time. For example, as shown in FIG. 3b, method 300 may be repeated to obtain a stream of position data for the first marker (306b). Iterative, as used herein, may mean continuously or nearly continuously (depending on computational power limitations). One advantage of the methods herein is the relatively low level of computational power required to determine location using IR. This is due to the high signal-to-noise ratio of the signals involved (especially when narrowband IR detection is used). Thus, the methods herein enable rapid and computationally inexpensive acquisition of a stream of position information.

[0088] Thus, a stream of location data can be obtained in real time (or near real time). Tracking the location of a single point in this manner can be useful in a variety of scenarios, including, but not limited to, tracking the location of a drone in a factory or construction environment, tracking the location of a person in a factory or construction environment, tracking soccer players on a soccer field, tracking cars on a road, or any other scenario where a single point can be used to track an object.

[0089] In embodiments where the first point is on a moving part of a machine, method 300 can be used to accurately, continuously, and with low latency measure the position of major moving components of plant and machinery (e.g., robotic arms, excavator limbs, bed height of a heavy dump truck, and / or crane hooks, etc.).

[0090] In some embodiments, as described above, there may be more than one marker attached to the first object, and therefore more than one marker may be visible to one or more infrared sensors at any one time. Accordingly, in some embodiments, method 300 may further include repeating steps i) and ii) for infrared radiation coming from two or more different markers positioned at two or more different points on the first object to determine two or more locations on the first object. It will be appreciated that steps i) and ii) need not be repeated sequentially (e.g., one after another) for each marker; for example, step i) may be performed for all markers at once, followed by step ii) for all markers at once. That is, steps i) and ii) may be performed as parallel computing processes.

[0091] In various embodiments, more than one marker may be used, for example, an IR marker may be attached to a component of an article under construction as a reference point for fabrication, assembly, construction, or manufacturing of the final article. As another example, markers may facilitate the precise fitting of major assemblies on a ship or aircraft production line, or the assembly of prefabricated building sections at a construction site.

[0092] In some embodiments, multiple markers can be applied to the first object, and when detected, these can form a point cloud of (e.g., narrowband) IR detections. A method for processing the point cloud is shown in FIG. 3c. In step 302c, method 300 can include receiving data regarding a point cloud of detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from multiple markers disposed at multiple points on the first object. In step 304c, location, orientation, and / or position information of the first object in space is determined from locations of the detected signals in the received data. Steps 302c and 304c are repeated to obtain (306c) a stream of such location, orientation, and position information for the first object. Herein, location may include x, y, and z coordinates of the object in space; orientation may relate, for example, to the pitch, yaw, and roll of the object; and position information may relate to the position or “pose” of any articulated portion on the first object.

[0093] Thus, the data may include the locations of multiple markers within the image, such as a "point cloud" of markers. Those skilled in the art will be familiar with point clouds and different methods that may be used to map location, orientation, and / or position information from a point cloud to a three-dimensional surface or structure.

[0094] As an example, points in the point cloud may be mapped (or fitted) to a model of the object to determine, for example, where a particular part of the first object (e.g., a lever, arm, mechanical connector, etc.) is located. One example of a suitable model is a three-dimensional deformable mesh structure, with each vertex of the mesh corresponding to a marker on the first object. Thus, the motion of the object can be represented by deformation of the three-dimensional mesh structure. Such a mesh has some static points and some dynamic points (to correspond to fixed and articulated parts of the first object, respectively), but with constrained and known degrees of freedom in the reference coordinate system of the machine body. The object model can be linked to a design model of a machine tool (e.g., in 3D CAD), and emitters / reflectors can be designed and analyzed to optimize tracking of the machine tool.

[0095] Additionally, specific markers can serve as “anchor” points (or boundary conditions) for the fitting process; for example, markers with unique patterns or frequencies of IR reflection or emission may be associated with specific points in the mesh. It will be appreciated that constraints on the allowable deformation of such a mesh may be applied depending on the range of allowable motion and articulation of the first object. Thus, embodiments herein use what may be thought of as a “customized point cloud” based on emitters / reflectors at a priori known points on the machine tool and structure. Customized point clouds have advantages over methods such as lidar scanning (which yields highly complex point clouds, e.g., 200 × 200 dots, each with a distance to a reflective surface) or photogrammetry, because the point cloud can be designed to maximize computational efficiency, allowing meshes to be fitted much faster and with near-zero latency, thereby enabling closed-loop machine control.

[0096] Additionally, as described above, triangulation may be used to determine one or more locations of the markers by comparing the detected locations of the markers in two or more different images of the markers taken from different viewpoints. In instances where multiple markers are present and problems arise with identifying individual markers within each image, signals from the markers may be distinguished from one another, for example, by frequency or emission pattern.

[0097] As another example, machine learning may be used to predict the location, orientation, and position information of an object from a point cloud of points.

[0098] Those skilled in the art will be familiar with machine learning and methods for training models using machine learning processes. Briefly, however, a model, sometimes referred to as a "machine learning model," includes a set of rules or (mathematical) functions that can be used to perform tasks related to data input into the model. Models may be taught to perform a wide variety of tasks on the input data; examples include, but are not limited to, determining a label for the input data, performing a transformation of the input data, predicting or estimating one or more output parameter values ​​based on the input data, or generating any other type of information that can be determined from the input data.

[0099] In supervised machine learning, a model learns from a training dataset that includes example inputs and corresponding ground truth (e.g., "correct") outputs for each example input. Typically, the training process involves learning weights or bias values ​​for the model to adjust the model to reproduce the ground truth outputs for the input data. Different machine learning processes are used to train different types of models; for example, machine learning processes such as backpropagation and gradient descent can be used to train neural network models.

[0100] A model herein may generally be any type of machine learning model that can be trained to take as input a point cloud of IR signals (or data indicative of such a point cloud) and output a prediction of the location, orientation, and / or position / pose information of the first object, as described above. Examples of suitable models include, but are not limited to, neural network models, linear regression models, and decision tree models.

[0101] In some examples, the model is a neural network. There are various open-source neural network models suitable for use in the embodiments described herein, such as the scikit-learn neural network described in the paper entitled "Scikit-learn: Machine Learning in Python" (Pedregosa et al., JMLR 12, pp. 2825-2830, 2011). In general, the features described herein can be obtained using the default neural network parameter settings described in the guidance.

[0102] There are many different possible combinations of input and output parameters. As one example, the input may be in the form of an image, e.g., an image showing a point cloud. Such an image may be supplemented with additional data, such as the location or other identification of the IR sensor that made the detection. As another example, the input may be a list of vectors corresponding to the center point of each detected IR signal in the point cloud. In yet another example, the input may be raw data from an IR sensor. Those skilled in the art will understand that these are merely examples and that other inputs may be provided in addition to or instead of those described above.

[0103] The neural network may also take as input data related to the first object's previous location, orientation, and / or position information, which may improve the neural network's predictions because this information has a strong causal relationship with the first object's current location, orientation, and / or position.

[0104] With respect to output parameters, the neural network may be trained to output any type of location, orientation, and / or position data, such as the relative location of the first object in space, the absolute location of the first object, the pitch, roll, and / or yaw of the first object, an indication of the location of a limb or articulated component (e.g., an arm, claw, shovel head, or the like) of the first object, or any other information related to the location, orientation, or mechanical attitude or state of the first object.

[0105] The neural network may also be trained to output other information about the first object that can be inferred from the point cloud, such as the type of the first object (e.g., type or make of drone, machinery, vehicle, etc.), its size, or range.

[0106] As described above, neural networks can be trained using training data that includes example inputs and "correct" or ground truth outputs for the example inputs. The training dataset can be constructed in various ways. For example, by simulating a first object in different positions, e.g., at different distances, angles, and orientations from a viewpoint, and simulating corresponding point clouds for each location and position. Such simulations may be performed using computer-aided design (CAD) tools. The training dataset may also be constructed systematically by sequentially sampling all possible "location / orientation / position" spaces available to the first object in physical space. In this way, a training set can be constructed that can train neural networks with high accuracy in a wide range of scenarios.

[0107] It will be appreciated that if further input parameters are provided, the training dataset can be expanded to sample possible positions and resulting patterns generated taking into account one or more additional parameters (and parameter spaces), such as, for example, point clouds generated using a combination of different IR markers (e.g., emitting with different frequencies, pulsed signals, coded signals, etc.).

[0108] It will be further appreciated that the training set may include point clouds representing two or more objects. For example, a neural network may be trained to predict different types of interactions (or interoperations) between a first and second object, and / or to output a proximity alert, etc., when a point cloud indicating that a first and second object are too close to each other is input.

[0109] It will be further appreciated that the training data set can be constructed from actual measurements, for example, by observing observable point clouds as a first object performs various maneuvers and recording the resulting point clouds and location / orientation / position data for use in constructing the training data set. It will be further appreciated that the training data set can also be comprised of a combination of actual and simulated data.

[0110] Some example pairs of locations and point clouds are shown in Figure 4b (also described below). In this example, in Figure 4b i), a first object in the form of a machine 400 is shown at a first location and position next to its point cloud 402. In Figure 4b ii), the same object detected at a more distant location with the same pose 400a is shown with its associated point cloud 402a. Figure 4b iii) shows the same machine with a different pose 400b next to its associated point cloud 402b. Figure 4b iv) shows a second object in the form of a truck 404 and its associated point cloud 406. Thus, in this example, a training dataset may consist of different objects in different poses at different locations.

[0111] Various further exemplary algorithms that may be modified for the purposes of this specification are described in the following papers: "LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds" (2019) by Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabas Poczos, and Yaser. This paper uses an autoencoder to fit deformable articulated meshes to point clouds. These types of techniques can be used in embodiments herein to obtain location data for moving parts, such as mechanical arms of robots or machinery. "Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image" by Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. In ECCV, 2016. This paper describes a method for determining human pose, which can be generalized to determining the pose of mechanical objects from point clouds. "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" by Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas (2017) describes an algorithm in which points in a point cloud are directly fed to a neural network that can be trained to disambiguate objects. Such a neural network can be trained to classify objects based on their location.

[0112] It will be understood that the above-cited papers are merely examples, and that other methods for identifying objects and their locations from point clouds of IR signals reflected and / or emitted from IR markers may be used as well.

[0113] In this manner, multiple markers can be used as described above to generate a point cloud of the first object from which the location, orientation, and position information of the first object can be determined.

[0114] Over time, method 300 can be repeated iteratively (e.g., frame by frame on the IR digital video data of the first object) to track and monitor location over time and determine motion parameters such as velocity, trajectory, and acceleration of the first object.

[0115] As mentioned above, the point cloud may also be processed to determine the orientation of the first object. For example, if the first object is a drone, the point cloud may be used to determine the pitch, yaw, and roll of the drone (e.g., this information may be output from the models listed above). Thus, the above method can be used to track an object and determine its position, orientation, and motion characteristics in real time.

[0116] In some embodiments, the location, orientation, and position information can be used to send instructions to the first object, such as to instruct the first object to perform an action or operation. Examples of instructions that may be sent to the first object to cause it to perform an operation include, but are not limited to, instructions to accelerate, stop, start, or pivot, or instructions to articulate an arm or moving part of the first object. Examples of instructions that may be sent to the first object to cause it to perform an action include, but are not limited to, an action associated with securing two parts together, an action associated with separating one part from another, an action associated with a mechanical arm (such as scooping, lifting, tilting, etc.), manipulating a tool, or any other action that can be performed by the first object.

[0117] Thus, in embodiments where the first object is a robot, drone, or mechanical object for use in a factory or construction environment, the method 300 described above may be used to monitor and provide instructions to the first object as part of an automated factory or automated construction program.

[0118] Turning now to other embodiments, as noted above, these principles can be applied to two or more objects, and the resulting data can be used to determine and coordinate interactions between the objects. For example, in some embodiments, method 300 can include repeating steps i) and ii) for a second marker placed on a second point on the second object to determine the location of the second point in space. The determined locations of the first and second markers can then be used to determine the relative proximity of the first and second objects.

[0119] The relative proximity can be used to initiate a proximity alert or to send a command (e.g., to an operator or other system) to stop or modify the movement of the first object or the second object in response to determining that the relative proximity is below a first threshold proximity. In embodiments where the first object is moving machinery and the second object is a person, method 300 can be used to accurately, continuously, and with low latency measure the location of a human worker working in close proximity to the moving machinery. Such information may be used to deliver proximity alerts to the human worker or adjacent persons, or to provide the basis for safety interlocks to reduce accident levels and / or reduce the economic impact of inadvertent collisions.

[0120] It will be appreciated that a first plurality of markers may be placed on a first object and a second plurality of markers may be placed on a second object, resulting in detection of a first point cloud and a second point cloud, respectively. Each point cloud may be processed using any of the techniques described above (e.g., using triangulation, by fitting a deformable mesh to each point cloud, and / or using machine learning) to determine relative location, orientation, and position / pose information for the first object and the second object, respectively.

[0121] The process can be made more efficient by, for example, distinguishing between a first and second plurality of markers, for example, using a different pattern or frequency for each marker. Additionally, markers within the first plurality of markers may be configured to emit or reflect IR radiation at alternating intervals relative to the second plurality of markers, such that only one marker is visible to an IR detector at any one time. Thus, the first plurality of markers may appear to "blink," followed by the second plurality. If the frequency of the blinking is high enough, each object can still be tracked in real time.

[0122] In other embodiments, the relative proximity and / or location, orientation, and position / pose information can be used to coordinate operations between a first object and a second object. For example, method 300 can be used to guide a mechanical arm of a first object (such as a robot or machine) toward a specific point on a second object under manufacture, to guide a first object in the form of a robot in attaching a first component to a specific portion of a second item under construction, to guide a pair of robots in a refueling operation, to guide an automated truck in a loading procedure, or to guide any other automated or semi-automated process between two objects. Thus, method 300 can be used to provide higher levels of automation in fabrication, assembly, construction, and manufacturing processes.

[0123] 4a shows a first object 400 in the form of machinery for use in a factory or construction site, bearing a plurality of markers 202. When detected by an infrared detector 204, the plurality of markers gives rise to a point cloud 402. As mentioned above, the appearance of the point cloud changes depending on the viewpoint of the IR detector 204, and therefore different viewpoints can be used to determine the location of the points using the principles of triangulation, or any of the other techniques mentioned above. In this embodiment, method 300 as shown in FIG. 3c can be applied to the point cloud to obtain a stream of location, orientation, and position information for the first object (e.g., machinery).

[0124] Figure 4b shows an exemplary point cloud observable for the machinery of Figure 4a, and how the point cloud changes with increasing distance (ii in Figure 4b) and articulation of the mechanical arm (iii in Figure 4b). Figure 4b iv shows a point cloud associated with a second object. Such an object-point cloud pair can be used as training data for training a machine learning model (such as a neural network) to take the point cloud as input and provide a prediction of the machine type and / or orientation as output, as described above.

[0125] In this embodiment, the method of FIG. 3c may further include steps 308c and 310c, as shown in FIG. 4c, whereby the stream of location, orientation, and position information output by the method steps shown in FIG. 3c is further used to determine an action or operation to be performed by the first object (e.g., using techniques such as reinforcement learning, pre-programmed rules, optimization, or any other suitable process), and in step 310c, the first object is instructed to perform the determined action or operation.

[0126] FIG. 5a illustrates an exemplary system for use in some embodiments herein. In this embodiment, a space or “sensing environment” 500 is a factory or construction site where a large item (e.g., a train, a submarine, a bridge component, etc.) 512 is being manufactured. The large item 512 has multiple markers on its surface (the markers may emit or reflect IR radiation). IR sensors 204 are positioned at different points in the space and transmit data regarding the detection of the markers on the large item 512 to the computer node 100. In this embodiment, the IR sensors may be broadband or narrowband IR sensors, as described above. In this embodiment, the computer node 100 includes three computing modules: an IR sensor control module 502 for performing step 302 and receiving data regarding one or more detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a first marker positioned at a first point on a first object; and an IR sensor module 502 for repeating step 302 for the data regarding the detection of each marker on the large item 512. The received data is sent to a positioning engine 504, which performs step 403 and determines the location of each point in space corresponding to a marker from the locations of detected signals in the received data. The positioning engine 504 may use any of the techniques described above in connection with step 504 for determining location information from one or more markers on an object (e.g., triangulation, mesh-based model fitting, and / or machine learning).

[0127] The IR sensor control module 502 and positioning engine 504 perform the same process for markers on the various robots 508a, 508b, and 508c engaged in the construction of the large item 512. The method 300 is used to determine location, orientation, and position information for each of the robots 508a, 508b, and 508c (e.g., the articulation of each of the above parts).

[0128] The robot movement controller module 506 uses the determined location, orientation, and posture information, for example, to determine orientation and / or position / posture information for the robot and to determine actions to be performed by the robot in erecting the large item 512.

[0129] Markers may also be placed on person 510 working within space 500 to track the person's 510 movements in relation to robots 508a, 508b, and 508c. In this way, proximity alerts can be issued if someone gets too close to the automated machinery.

[0130] A method for detecting a first object (the first object being a robot, drone, or other machinery in a manufacturing or construction space) is shown in Figure 5b. The method includes, in step 502, receiving data regarding one or more detections made by at least one infrared sensor of infrared radiation coming from a first marker positioned at a first point on the first object. Then, in step 504, the method 500 includes determining a location of the first point in space from a location of the detected signal in the received data.

[0131] Thus, a system for facilitating an automated factory or construction site is provided. It will be understood that the above details are merely examples, and that computer node 100 may include a different number or combination of modules than shown in Figure 5a. Furthermore, there may be a different number of robots and / or IR sensors than shown in the figures.

[0132] Referring now to FIG. 6, there is shown an embodiment of the present disclosure in which markers are used as reference points in the construction, monitoring, or inspection of large objects such as buildings, bridges, skyscrapers, or as shown in FIG. 6, wind turbines.

[0133] During construction, IR markers may be attached to components of the item under construction as reference points for remote, automated, and high-precision inspection of the final product, for example, inspection of the external condition of a high-rise building by a drone equipped with an IR sensor.

[0134] However, markers can also be used to monitor finished products throughout their service life. IR markers can be attached to objects as reference points for remote, automated, and highly accurate inspection of the object throughout its entire operational life, day and night, and under various weather conditions. For example, as shown in FIG. 6a, marker 602 may be placed on wind turbine 600 for automated (i.e., operator-free) inspection of the wind turbine's external condition by drone 604 equipped with an IR sensor, which may be configured to perform method 300 to locate marker 602 and enable drone 604 to return to the same inspection viewpoint with high accuracy, thus enabling, for example, comparison of the progression of cracks and paint deterioration over time. Thus, in this way, method 300 can be used to perform accurate comparative inspections of large structures over time, even under adverse conditions, such as on offshore wind turbines, regardless of whether the turbine blades are rotating or not. If the turbine blades are rotating (eg, the first object is moving), the location can be tracked using method 100 throughout the movement.

[0135] 6b shows steps that may be performed by computer node 100 in the embodiment shown in FIG. 6a. In this embodiment, the computer node performs steps 302 and 304 of method 300 as shown in FIG. 3. In this embodiment, data from step 302 is received from drone 604 and used to determine the location of a first point on the wind turbine in step 304. Computer node 300 then performs step 306d, as shown in FIG. 6b, to cause the drone to take a measurement of the first object using the determined location of the first point in space as a reference point for aligning the drone to the first object before the measurement is taken. Computer node 100 may cause the drone to take the measurement, for example, by sending instructions to the drone.

[0136] Turning now to other embodiments, it will be appreciated that method 300 may be embodied in a computer program. For example, a computer program product may include a computer-readable medium having computer-readable code embodied therein. The computer-readable code, when executed by a suitable computer or processor, may be configured to cause the computer or processor to perform one or more methods (such as method 300) described herein.

[0137] A computer program may take different forms, such as source code, compiled code, executable code, or any other type of code. It will be understood that the source code of a computer program may be written in a variety of different programming languages ​​and may employ different architectural designs. For example, the functionality described herein may be divided across a variety of different subroutines. Furthermore, those skilled in the art will appreciate that many different ways of dividing functionality among different subroutines are possible. The subroutines may be stored together in a single executable file to form a self-contained program. Furthermore, the computer program may call external and / or standard libraries of computer code to perform specific subtasks related to the functionality described herein.

[0138] In another embodiment, there is a computer program product comprising a non-transitory computer readable medium having stored thereon a computer program as described above. Examples of computer readable media include, but are not limited to, a ROM such as a CD ROM, a semiconductor ROM, or a magnetic recording medium such as a hard disk.

[0139] In another embodiment, there is a carrier that contains the computer program. Examples of the carrier include, but are not limited to, an electronic signal, an optical signal, a radio signal, a computer storage medium, or the like. The carrier of a computer program may be any entity or device (e.g., hardware) that can carry the program. As an example, the carrier may be a computer-readable medium as described above. In another example, the carrier may be a transmissible carrier such as an electronic or optical signal, which may be conveyed via electrical or optical cable, or by radio or other means.

[0140] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these claims cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope. Furthermore, it will be understood that the features, advantages, and functions of different embodiments described herein may be combined without departing from the spirit or scope of the disclosure herein.

Claims

1. 1. A computer-implemented method for detecting a first object in a space, the method comprising: i) receiving data relating to one or more detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a first marker located at a first point on the first object; ii) determining a location of the first point in space from a location of a detected signal in the received data; 11. A computer-implemented method comprising:

2. 2. The method of claim 1, wherein the narrowband signal is centered at about 800 nm and has a frequency range of about + / - 10 nm or a frequency range of about + / - 20 nm.

3. 3. The method of claim 1, further comprising iteratively repeating steps i) and ii) to track movement of the first point on the first object over time.

4. instructing an infrared emitter to emit pulses of infrared radiation into the space, the pulses being emitted at a first pulsating frequency; 10. A method according to any one of the preceding claims, wherein steps i) and ii) are repeated for the detection of a reflection of each pulse from the first marker.

5. 5. The method of claim 4, wherein the pulses are emitted at a pulsating frequency of about 60 to about 100 Hz.

6. 10. The method of any one of the preceding claims, further comprising repeating steps i) and ii) for infrared radiation coming from two or more different markers located at two or more different points on the first object.

7. 10. The method of any one of the preceding claims, wherein step i) comprises receiving data relating to a point cloud of detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a plurality of markers arranged at a plurality of points on the first object.

8. In step ii), The method of claim 7 , further comprising determining an orientation, position, or pose of the first object in the space from the point cloud.

9. determining an action or operation to be performed by the first object using the stream of location, orientation, and position information about the first object; transmitting a control signal to the first object to cause the first object to perform the action or operation; The method of claim 8 further comprising:

10. repeating steps i) and ii) for a second marker placed on a second point on a second object to determine the location of the second point in the space; determining a relative proximity of the first object and the second object using the determined locations of the first marker and the second marker; and 10. The method of any one of the preceding claims, further comprising:

11. Initiating a proximity warning; or in response to determining that the relative proximity is below a first threshold proximity, transmitting a command to stop or modify movement of the first object or the second object; The method of claim 10 further comprising:

12. 10. The method according to any one of the preceding claims, wherein the first object is a robot, a drone, a mechanical object, or a person.

13. 10. The method of any one of the preceding claims, wherein the space is a construction site, a factory, or a field.

14. 10. The method of any one of the preceding claims, wherein the first object is a building, a bridge, or a wind turbine.

15. 10. The method of any one of the preceding claims, wherein the second object is a second robot, a second drone, or a human.

16. The data in step i) is received from a drone, and the method further comprises:

9. The method of claim 1, comprising causing the drone to measure the first object using the determined location of the first point in space as a reference point for aligning the drone to the first object before the measurement is taken.

17. determining the location of the first marker; 10. A method according to any one of the preceding claims, comprising triangulating the location from the detected signals in the received data.

18. 10. The method according to any one of the preceding claims, wherein the method is carried out at night, outdoors or under adverse weather conditions.

19. 1. A computer node for tracking a first object in space, said node comprising: i) receiving data relating to one or more detections made by at least one narrowband infrared sensor of narrowband infrared radiation coming from a first marker located at a first point on the first object; ii) determining a location of the first point in space from a location of a detected signal in the received data; 1. A computer node comprising one or more computer processors configured to:

20. A computer node according to claim 19, further configured to perform the method according to any one of claims 2 to 18.

21. 1. A system for tracking a first object in space, the system comprising: a first marker located at a first point on the first object; at least one narrowband infrared sensor having a narrowband infrared filter for detecting a narrowband of infrared radiation; A computer node, i) receiving data regarding one or more detections made by the at least one narrowband infrared sensor of the narrowband infrared radiation coming from the first marker; ii) determining a location of the first point in space from a location of a detected signal in the received data; a computer node configured to perform Including, the system.

22. 22. The system of claim 21, wherein the narrowband infrared filter has a frequency range of about + / - 10 nm or a frequency range of about + / - 20 nm centered at about 800 nm.

23. 23. The system of claim 21 or 22, wherein the marker comprises an infrared emitter, a reflective material, or a retroreflective material.

24. The system comprises:

24. The system of claim 21, 22, or 23, further comprising one or more infrared lamps configured to illuminate the space with infrared radiation such that the infrared radiation is reflected from the marker.

25. 25. The system of claim 24, wherein the one or more infrared lamps are configured to emit pulses of infrared radiation at a first pulsating frequency, and the computer node is configured to repeat steps i) and ii) for each pulse.

26. The system of any one of claims 21 to 25, wherein the computer nodes are further configured to perform the method of any one of claims 2 to 18.

27. 1. A method for tracking a first object, wherein the first object is a robot, drone, or other machinery in a manufacturing or construction space, the method comprising: i) receiving data relating to one or more detections made by at least one infrared sensor of infrared radiation coming from a first marker located at a first point on the first object; ii) determining a location of the robot, drone, or other machinery in the space from locations of detected signals in the received data; and A method comprising:

28. 1. A computer node for detecting a first object, said first object being a robot, drone, or other machinery in a manufacturing or construction space, said node comprising: i) receiving data relating to one or more detections made by at least one infrared sensor of infrared radiation coming from a first marker located at a first point on the first object; ii) determining a location of the robot, drone, or other machinery in the space from locations of detected signals in the received data; and 1. A computer node comprising one or more processors configured to:

29. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 18 or claim 27.

30. A computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 18 or claim 27.

Images