Detecting objects in a space

Abstract

A computer-implemented method of detecting a first object in a space (500) includes i) receiving data relating to one or more detections by at least one narrowband infrared sensor of infrared radiation within a narrowband from a first marker placed at a first point on the first object. The method then comprises ii) determining the position of the first point in space as a function of the position of the detected signal in the received data. The space (500) may be a factory or an outdoor construction site where large articles (512) are being manufactured. The large article (512) has a plurality of indicia on a surface thereof that emit or reflect infrared (IR) radiation. The IR sensors (204) are located at different points in space and send marker detection data to an IR sensor control module (502) in the computer node (100). A position determination engine (504) determines the position of each point corresponding to the marker in the space from the position of the detected signal in the received data. The robot movement controller module (506) uses the determined position to determine orientation and / or location / attitude information, e.g., regarding the robot (508a, 508b, 508c), and to determine an action to be performed by the robot. The indicia may also be placed on a person (510) working in the space (500) to track their movement relative to the robot. If someone is too close to the automated machine, a proximity alarm can be issued.

Description

Technical Field

[0001] This disclosure relates to object detection, monitoring, and tracking. It further relates to precise detection, monitoring, and tracking of objects to improve efficiency, safety, and automation. It further relates to tracking and monitoring robots, drones, other machines, and operators, etc. in intelligent manufacturing, construction, and inspection environments. Background Art

[0002] This disclosure relates to tracking objects, such as robots in a factory environment. Various manufacturing environments use robots. For example, the automotive industry has been applying robotics for decades, especially to free human operators from dull, dirty, dangerous, and strenuous operations, while reducing costs, improving quality, and accelerating production lines.

[0003] Recently, robots have been applied to work safely in collaboration with humans, known as "cobots". Factory robot tasks now extend to material handling, internal logistics, inspection, packaging, distribution, and lifetime maintenance, as well as more traditional manufacturing, assembly, and finishing production lines.

[0004] A defining feature of applying robotics to manufacturing is the mechanical grounding of the robot body, thus mechanically grounding the robot arm, combined with fixing the item being manufactured relative to the grounded robot body. Typically, a production line moves the item being manufactured from one station to the next. This strategy enables precise relative alignment of the item being manufactured with the robot arm and precise control of multiple degrees of freedom of the robot arm through pre-programmed movement instructions. Summary of the Invention

[0005] As described in the background art section, robot production lines currently tend to move items such as cars from one manufacturing station to the next along the production line. However, as the size and complexity of items being manufactured, such as airplanes, trains, ships, submarines, and spacecraft, increase, it becomes increasingly impractical to move the items along the production line frequently enough or quickly enough relative to the mechanically grounded robots. Therefore, there is an increasing desire to be able to deploy robots and "cobots" that are not fixed in place but can move effectively around, adjacent to, or even inside the item being manufactured or constructed, whether the item is a "positive" structure (such as a building) or a "negative" structure (such as a tunnel).

[0006] This logic naturally extends to many fields, including manufacturing, civil engineering, and agriculture, etc., where the houses or items to be manufactured, the bridges to be built, the materials for tunneling, the land to be cultivated, etc. are actually immovable. Based on satellite imaging and high-end Global Positioning System (GPS) / Global Navigation Satellite System (GNSS) navigation, an accuracy of 10 centimeters can be achieved for slowly moving machines in relatively open agricultural spaces, and the benefits of precision agriculture are now being widely enjoyed. However, whether in factories or towns, this level of accuracy is simply not sufficient to enable mobile facilities, equipment, and machines to operate at much higher speeds than agricultural machines, and GPS is also less reliable in these places. For example, a higher level of positioning accuracy is required in the robotic assembly of prefabricated construction parts, where a Local Positioning System (LPS) is needed, which can provide accurate, near real-time (low latency) measurements of the position of machines in space in a collaborative, flexible, and safe manner, including precise measurements of the movement of their critical moving parts / limbs.

[0007] There are various existing Local Positioning System (LPS) technologies based on Radio Frequency (RF) transmitters and RF receivers. The RF transmitters are installed on mobile objects, and the RF receivers are placed around the operation site or factory to jointly calculate the position and trajectory of the transmitter. When the measured movement involves high speeds or significant accelerations, these technologies have accuracy limitations. Therefore, they can track robots moving slowly around a warehouse, but cannot track fast-moving robotic arms with the required accuracy and latency. There are also LPS technologies based on camera image processing or photogrammetry, which have obstacles of latency and extremely high computational complexity, especially when the measured situations are not exactly repetitive.

[0008] Accuracy is not the only issue in these intelligent manufacturing and construction applications. To truly realize the potential of such systems, there is an increasing desire for machines, robots, and drones, etc. to be able to work independently outdoors and in various weather conditions, whether during the day or at night. There is also a desire for robots and drones, etc. to be able to work over long distances, such as in large-scale construction, manufacturing, inspection, and maintenance environments over long distances, including indoor and outdoor, above and below ground.

[0009] The purpose of the embodiments of this article is to solve some of these problems.

[0010] According to a first aspect of the present disclosure, there is provided a computer-implemented method for detecting a first object in a space. The method includes: i) receiving data related to one or more detections of infrared (IR) radiation within a narrow band by at least one narrow-band IR sensor, the narrow-band IR radiation originating from (e.g., reflected from or emitted from) a first marker placed at a first point on the first object; and ii) determining the position of the first point in the space based on the position of the detected signal in the received data.

[0011] In some embodiments, the narrow-band signal is centered at approximately 800 nanometers and has a frequency range of approximately + / - 10 nanometers or approximately + / - 20 nanometers.

[0012] Thus, a method for locating a point on an object using an infrared marker and a narrow-band infrared sensor is provided. Using narrow-band detection in this way has significant advantages. In particular, narrow-band detection has a higher signal-to-noise ratio than unfiltered wide-band detection. This enables reliable detection at greater distances (on the order of ~100 m at the time of writing) with the high precision (typically less than ~5 cm) required for such applications. These distances can be achieved even under adverse weather conditions, day or night. The measurements remain accurate even if the object is moving rapidly or accelerating. Additionally, the markers (IR reflectors or emitters) in the embodiments herein are generally at least as cost-effective as systems such as active radio frequency identification (RFID) tags. Such systems can be used in a variety of scenarios, e.g., for tracking people or animals on a sports field or stadium.

[0013] The technology can be further advantageously deployed in smart factory and smart manufacturing environments as described above, where narrow bands can be used to locate (with high precision) markers at specific points on people, machines, robots, drones, etc. at long distances.

[0014] Thus, the systems and methods herein can be used to monitor the positions of objects (such as manually operated machines, remotely controlled machines, autonomous machines and robots, collaborative robots, drones, other mechanical systems, and human operators) with high precision within the required range, day or night, and under various weather conditions. Long-range high-precision position data can be used to send instructions to any of these object types to coordinate robots with other robots, machines, and / or people in an industrial setting. Thus, the LPS systems described herein are precise enough over a sufficient range to be used in manufacturing particularly large or stationary objects. Thus, the systems herein are capable of smart manufacturing of trains, airplanes, bridges, submarines, etc., as well as other applications.

[0015] In some embodiments, steps i) and ii) are repeated iteratively to track the movement of a first point on a first object over time.

[0016] In some embodiments, the method further includes instructing an infrared emitter to emit infrared radiation pulses into space, where the pulses are emitted at a first pulsation frequency. Then steps i) and ii) are repeated to detect the reflections of each pulse from the first marker. For example, the pulses can be emitted at a pulsation frequency between about 60 Hz and about 100 Hz. Using the IR emitter in a pulsed manner, rather than keeping the IR emitter on all the time, helps to reduce the energy used by the emitter while still enabling high-precision near-real-time tracking.

[0017] In some embodiments, steps i) and ii) are repeated for infrared radiation from two or more different markers at two or more different points on the first object.

[0018] For example, in some embodiments, step i) includes receiving data related to point cloud detection of infrared radiation within a narrowband by at least one narrowband infrared sensor, where the infrared radiation within the narrowband is from multiple markers placed at multiple points on the first object. Step ii) can further include determining the orientation, azimuth, or pose of the first object in space based on the point cloud.

[0019] In some embodiments, the method further includes using the position, orientation, and azimuth information stream of the first object to determine an action or maneuver to be performed by the first object; and sending a control signal to the first object to cause the first object to perform the action or maneuver. Thus, the methods herein can be used to automatically control drones, machines, etc. in intelligent manufacturing and construction environments.

[0020] In some embodiments, the method further includes repeating steps i) and ii) for a second marker placed at a second point on a second object to determine the position of the second point in space, and using the determined positions of the first marker and the second marker to determine the relative proximity of the first object and the second object. In some examples, the relative proximity is used to initiate a proximity warning or, in response to determining that the relative proximity is less than a first threshold proximity, send a command to stop or change the movement of the first object or the second object. Thus, the methods herein can be used to enhance safety in an automated factory or manufacturing environment, especially if the second object is a person.

[0021] In some embodiments, the first object is a robot, a drone, a mechanical object, or a person. In some embodiments, the space is a construction site, a factory, or a field. In some embodiments, the first object is a building, a bridge, or a wind turbine. In some embodiments, the second object is a second robot, a second drone, or a person.

[0022] In some embodiments, the data in step i) is received from a drone, and the method further includes: using the determined position of the first point in space as a reference point to cause the drone to measure the first object, and before performing the measurement, aligning the drone with the first object using the reference point. Thus, in this way, the drone can use the marker to accurately and reliably locate a specific point or a favorable point relative to a large object (such as a wind turbine, etc.) to measure the first object from the specific point or the favorable point. This can be used to ensure that the measurement is carried out in a repeatable and reliable manner, so that the condition of the object (such as cracks, paint defects, and structural changes, etc.) can be monitored over time.

[0023] In some embodiments herein, triangulation is used in step ii) to determine the position of the marker.

[0024] In some embodiments, the method is carried out at night, outdoors and / or under adverse weather conditions. Thus, the method can be used to allow 24-hour manufacturing and construction.

[0025] According to a second aspect, a method for tracking a first object, wherein the first object is a robot, a drone or other machine in a manufacturing or construction space. The method includes: i) receiving data related to the detection of infrared radiation from at least one infrared sensor from a first infrared marker; and ii) determining the position of the robot, drone or machine in the space according to the position of the detected signal in the received data. In some embodiments, the infrared sensor can be a narrowband infrared sensor.

[0026] Using infrared markers in the manufacturing and construction industries has significant advantages. In particular, using infrared-emitting or -reflecting markers to locate robots, drones and other machines enables the machines to be used at night and even under adverse weather conditions, thus facilitating a 24-hour manufacturing process. As described above, the infrared applied in this way is accurate on a scale of less than 5 centimeters and has this accuracy even when the object is moving or accelerating. This is valuable in many industrial fields (such as automated factories, etc.) and civil engineering applications (such as the construction of buildings, bridges and other infrastructure). Therefore, systems and methods are provided for accurately detecting, monitoring and tracking objects in intelligent manufacturing, construction and inspection environments.

[0027] According to a third aspect, a computer node including one or more processors, the processor being configured to execute the method of the first aspect or the method of the second aspect.

[0028] According to a fourth aspect, a system for tracking a first object in space. The system includes: a first marker placed at a first point on the first object, at least one narrowband infrared sensor including a narrowband infrared filter for detecting infrared radiation within a narrowband, and a computer node. The computer node includes one or more processors configured to: i) receive data related to one or more detections of infrared radiation in the narrowband from the first marker by the at least one narrowband infrared sensor; and ii) determine the position of the first point in space based on the position of the detected signal in the received data.

[0029] In some embodiments, the narrowband infrared filter is centered at approximately 800 nanometers and has a frequency range of approximately + / - 10 nanometers or approximately + / - 20 nanometers.

[0030] In some embodiments, the marker includes an infrared emitter, a reflective material, or a retroreflective material.

[0031] In some embodiments, the system further includes: one or more infrared lights configured to irradiate the space with infrared radiation so that the infrared radiation is reflected from the marker.

[0032] In some embodiments, the one or more infrared lights are configured to emit infrared radiation pulses having a first pulsation frequency, wherein the computer node is configured to repeat steps i) and ii) for each pulse.

[0033] In some embodiments, the computer node is further configured to perform the method of the first aspect.

[0034] According to a fifth aspect, a computer program containing instructions that, when executed by a computer, cause the computer to perform the method of the first aspect or the second aspect.

[0035] According to a sixth aspect, a computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the method of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Example embodiments herein will be described with reference to the following drawings, wherein:

[0037] Figure 1 a shows an example computer node according to an embodiment herein;

[0038] Figure 1 b shows Figure 1 the various components to which the computer node in a can be connected;

[0039] Figure 2 shows Figure 1 a andFigure 1 Examples of how computer nodes and components of b can be used to track one or more objects in a factory environment;

[0040] Figure 3a Shows an exemplary computer-implemented method for detecting a first object in a space according to some embodiments herein;

[0041] Figure 3b Shows according to embodiments herein Figure 3a An extension of the method in, which is used to obtain the position information flow of a first point on the first object;

[0042] Figure 3c Shows according to embodiments herein extending Figure 3a The method in to the detected point cloud of multiple markers placed on the first object;

[0043] Figure 4a Shows an example marker on an example item of a machine according to examples herein, and the resulting point cloud;

[0044] Figure 4b Shows Figure 4a The machine in various different positions and postures and the resulting point cloud;

[0045] Figure 4c Shows Figure 3c An extension of the method for controlling the manipulation of the first object in;

[0046] Figure 5a Shows an example system in a manufacturing or construction environment such as a factory according to some embodiments herein;

[0047] Figure 5b Shows according to some embodiments herein the example method performed by Figure 5a The computer node shown;

[0048] Figure 6a Shows an example application of the method herein in the monitoring of large objects such as wind turbines;

[0049] Figure 6b Illustrates Figure 3a An example extension of the method in for positioning a drone to perform precise measurements on the first object. Detailed Description

[0050] In short, the technologies, systems, and methods described herein are for tracking objects, particularly but not limited to factory and / or construction environments. For example, moving objects. The objects can be in a stationary environment and / or be mechanically grounded objects. Some embodiments herein use infrared video image processing and near-infrared (NIR) cameras, with NIR filters and / or narrowband IR filters and lenses used at fixed locations around the construction site or factory, and attaching matching NIR emission markers (LEDs) or retroreflective markers to the factories and / or machines involved in manufacturing or construction and / or the mechanically grounded products under construction. Such markers can also be installed on human operators working nearby.

[0051] In the case of NIR retroreflective markers, by providing markers that reflect and absorb ambient light and are easily recognizable, these markers can be effective under natural light conditions. They can also be used in combination with NIR floodlights located near the detection camera outdoors at night and in enclosed or even dark factories. In the case of narrowband infrared filters, these are advantageous because they result in low signal-to-noise ratio detection, so narrowband systems can be used to increase the detection range (even at night and / or under adverse weather conditions).

[0052] Embodiments herein use the detected reflections / emissions from such markers to determine the position, orientation, and pose of the objects to which they are attached. The methods herein use near-real-time computing methods (e.g., having a low latency time necessary for closed-loop robot control), for example, triangulation or higher-order multilateration, etc., which allow continuous and extremely precise (down to the <5 cm level) tracking of the reflection / emission markers. The systems and methods herein can be used to monitor movement and provide proximity warnings or stops, thereby improving the safety level of construction sites and factories, or they can be directly used in closed-loop robot control applications to provide a higher level of automation and efficiency.

[0053] More specifically, turning to Figure 1 a, in some embodiments, there is a computer node 100. The computer node 100 is used to (e.g., is configured to) detect a first object in space. The computer node 100 can generally be configured (e.g., operable) to perform any of the methods and functions described herein, such as method 300 described below, etc. The computer node 100 includes a processor 102, a memory 104, and an instruction set 106. The memory stores instruction data representing the instruction set 106 (e.g., such as compiled code, etc.). The processor can be configured to communicate with the memory and execute the instruction set. When executed by the processor, the set of instructions can cause the processor to perform any of the methods herein, such as method 300 described below, etc.

[0054] The processor (e.g., processing circuitry or logic) 102 can be any type of processor, such as a central processing unit (CPU), a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), or any other type of processing unit. The processor 102 can include one or more sub-processors, processing units, multi-core processors, or modules that are configured to work together in a distributed manner to control the computer node in the manner described herein.

[0055] The computer node 100 can include a memory 104. In some embodiments, the memory 104 of the computer node 100 can be configured to store program code or instructions that can be executed by the processor 102 of the computer node 100 to perform the functions described herein. The memory 104 of the computer node 100 can be configured to store any data or information mentioned herein, such as requests, resources, information, data, and signals described herein. The processor 102 of the computer node 100 can be configured to control the memory 104 of the computer node 100 to store such information.

[0056] In some embodiments, the computer node 100 can be a virtual computer node, such as a virtual machine or any other containerized computer node, etc. In such embodiments, the processor 102 and the memory 104 can be part of larger processing and storage resources, respectively. In some examples, the computer node 100 can thus be cloud-based.

[0057] It should be understood that the computer node 100 can include Figure 1 other components as shown in a. For example, the computer node 100 can include a power source (e.g., a main power source or a battery power source). The computer node 100 can also include a wireless transmitter and / or a wireless receiver to wirelessly communicate with other computing nodes and / or sensors (such as an IR sensor that detects the IR signals described herein, etc.). In some embodiments, the computer node 100 can have a wired connection and utilize this wired connection to communicate with other computer nodes and IR sensors, etc.

[0058] As Figure 1 shown in b, in the embodiments herein, the computer node 100 can communicate (e.g., send an electronic message to or receive an electronic message from) with one or more IR sensors 204 (also referred to as IR receivers or IR detectors). In some embodiments herein, the computer node 100 is further configured to send a message to one or more IR transmitters 203 to control, for example, when to transmit.

[0059] Briefly, in an embodiment of the present disclosure, a computer node is configured to detect a first object in space. The computer node is configured to receive data related to one or more detections of infrared (IR) radiation by at least one IR sensor, the IR radiation originating from a first marker located at a first point on the first object. The computer node is then further configured to determine the position of the first point in space (information regarding the position of the first point in space) based on the position of the detected signal in the received data.

[0060] As will be described in more detail below, in some embodiments of the present disclosure, the first object is a robot, a drone, or any other machine used in a manufacturing or factory environment. Thus, the systems and methods of the present disclosure can be used to track and subsequently send control instructions to robots and the like in an intelligent factory setup. In a manufacturing environment (automated, semi-automated, and / or non-automated), using IR-based markers to track robots, machines, drones, and people has several advantages. The determined position can be used to instruct automated or semi-automated machines and / or generate proximity alerts to reduce collisions. IR is particularly advantageous because it can be used at night (allowing "lights-out" manufacturing and round-the-clock construction) and in adverse weather conditions (allowing, for example, 24-hour construction on bridges and buildings regardless of the weather).

[0061] As described above, computer node 100 is used to detect (and monitor) a first object in space. The first object has a first marker mounted at a first point on the first object. The marker can emit IR radiation (e.g., in the manner of a beacon) or reflect IR radiation emitted by an IR emitter. In some embodiments, when the marker emits IR, the marker can include an IR bulb or an IR radiation source. Thus, the marker can be an infrared beacon. As described below, in any embodiment of the present disclosure, infrared can be replaced by near-infrared, and thus the marker can equally be a near-infrared beacon.

[0062] In an embodiment where the marker reflects IR radiation, the marker can include a reflective surface. In some embodiments, the marker includes retroreflective material. Retroreflective material reflects electromagnetic radiation such that the angle of incidence θ i is equal to the angle of reflection θ r , e.g., θ i = θ r . For this reason, retroreflectors are advantageous for object tracking. In an embodiment where the marker reflects IR radiation, there can be an IR emission source 203 (e.g., an IR bulb, a lamp, a floodlight, etc.) that emits IR radiation into the space (and is reflected from the marker). In some embodiments, there may be more than one IR radiation source. For example, there can be multiple IR lamps illuminating the space (in a manner similar to floodlights).

[0063] It should be understood that the first object may have more than one marker installed. In some embodiments, the first object has a plurality of markers attached thereto at multiple different locations on the first object. Each marker may emit or reflect infrared radiation. In such an example, the detection of the plurality of markers forms a "point cloud" of different points, with each point indicating a different point on the object.

[0064] The markers can be placed using different strategies. For example, they can be placed to form a triangular mesh structure on the surface of the object. In such an example, point cloud detection can be used to form a three-dimensional contour of the object's surface.

[0065] The markers can be placed at the mechanical joints of the first object. Thus, the markers can track the positions of the movable limbs of the first object. In such an embodiment, drawing lines between such joints gives the outline of the skeleton of the first object. The markers can be further placed at the ends of the object, such as at the ends of mechanical limbs, etc., so that the ends can be monitored and proximity alerts can be issued if the limbs move in a dangerous manner.

[0066] The number of markers on each object depends on the application. For example, four markers in different planes can be used for three-dimensional tracking of an object. In general, a grid of at least four markers that are not all in the same plane can be used to determine the position and orientation of an object in space, in addition to knowing the positions of those points in the object frame. If the object has moving parts, then at least four markers can be placed on each moving component of the entire object. However, if the components of the object have limited degrees of freedom (e.g., components that can only move up and down in a 2D plane, etc.), then fewer markers are required.

[0067] It should also be understood that different patterns or reflection characteristics can be used to indicate different positions on the first object or to identify specific markers. For example, the markers can be configured such that they emit or reflect infrared radiation of different wavelengths. As another example, in the case where the markers include infrared emitters, these can be configured to emit pulsed infrared radiation of different patterns in a coded manner.

[0068] In an example where there are multiple different objects and each object is being tracked, different patterns or reflection characteristics can be used to distinguish between the first plurality of markers on the first object and the second plurality of markers on the second object. For example, the markers on the first object can be configured such that they reflect infrared radiation of different wavelengths (or in different narrow bands) to the markers on the second object, or have different patterns.

[0069] In other embodiments, the emitted / reflected pattern, frequency, and / or emitted pulse code may further be used to identify the type of the detected object. For example, different patterns may be associated with different types of objects. For example, patterns may be used to distinguish vehicle classes (e.g., different domino patterns on the roofs of cars, heavy goods vehicles (hgv), large goods vehicles (lgv), buses, etc.). This also applies to excavators, dump trucks, scrapers, surface layers, etc. Additionally, the transmitter / reflector may also be configured to form barcodes, quick response (QR) codes, or any other recognizable type of marker to identify individual machines.

[0070] In some embodiments, when the markers emit IR radiation, the first plurality of markers on the first object may be configured to emit IR radiation pulses at a first pulsation frequency (e.g., with a first time period between pulses). And the second plurality of markers on the second object may be configured to emit IR radiation pulses at a second pulsation frequency. Pulse emission has various advantages, including but not limited to energy savings associated with periodically turning an IR emitter on and off. Additionally, the timing may be coordinated such that the first plurality of markers are emitting (e.g., "on"), while the second plurality of markers are "off", and vice versa. This can be used to group or distinguish the markers in the first plurality of markers from the markers in the second plurality of markers (e.g., distinguish two point clouds).

[0071] In the embodiments herein, the markers are detected by at least one IR sensor 204 (which may also be referred to as an IR receiver or detector), and the IR sensor 204 may include one or more IR cameras, which can be used to capture signals and generate a photo or image of the pattern. The IR camera and / or video device may also be used to capture the pattern of the IR signals in a video stream at a prescribed frame update rate.

[0072] As will be described in more detail below, in any embodiment herein, the infrared sensor 204 can be a narrowband infrared sensor, and the data received can include one or more detections of infrared radiation within the narrowband from a first marker (e.g., emitted or reflected from the first marker). For example, the infrared sensor 204 can be equipped with a narrowband filter. In principle, any narrowband infrared filter can be used. In some examples, the narrowband signal is centered at approximately 850 nanometers and has a width of approximately + / - 10 nanometers (e.g., half-height frequency range) or approximately + / - 20 nanometers. Using a narrowband IR detector in this way has many significant advantages, including increasing the signal-to-noise ratio and eliminating interference with the detection of emissions or reflections from the marker. This increase in signal-to-noise ratio enables the detection of the marker at a greater distance compared to a broadband IR detector. At the current IR camera sensitivity level, the detection range is on the order of 100 meters (which will increase in the future). By using narrowband infrared, a high signal-to-noise ratio can be achieved under various weather conditions, which is advantageous for applications such as outdoor construction, manufacturing, and agriculture. This is also advantageous for construction scenarios, where the methods herein can be applied to the construction of, for example, houses, buildings, skyscrapers, bridges, etc. Narrowband infrared detection can also be performed at night, allowing for 24-hour construction projects. The narrowband can be centered on the peak transmittance of near-infrared in air, further increasing the signal-to-noise ratio.

[0073] It should also be noted that in embodiments herein, there can be more than one IR sensor. For example, two or more sensors can be provided that are spatially offset from each other. Spatially offset sensors have different vantage points, and the offset of the detection location of the first marker and the relative positions of the IR sensors themselves can be used for triangulating the relative and / or absolute position of the marker. In the case of triangulation, at least 2 IR sensors can be used, however, depending on the accuracy required for a particular application, 3 or 4 can be used.

[0074] In embodiments where different markers reflect at different wavelengths, the IR sensors can be configured with different filters to track different sets of markers (such as markers fixed to a first object and a second object, etc.). As another example, different subsets of the IR sensors can be configured to detect in different wavelength bands.

[0075] In embodiments where the marker includes a reflective material that reflects IR radiation, an IR emitter is also used to illuminate the space. In embodiments where the IR receiver is a narrowband receiver, note that any IR emitter 203 in such embodiments can also include one or more filters to emit in the same narrowband as the narrowband IR receiver or in a frequency band that overlaps with the narrowband IR receiver.

[0076] In embodiments employing IR transmitter 203 (e.g., marked as reflective or retroreflective markers), these transmitters can be located at the same position as the IR receiver. For example, pairs of IR transmitters and receivers can be placed on lampposts or mast-type devices. However, it should be understood that the transmitter and receiver do not have to be located at the same position. It should also be understood that a movable transmitter can also be provided. As an example of a movable transmitter, the IR transmitter can be mounted on a drone for flexible illumination of the space.

[0077] In some embodiments, when there is an IR transmitter (and reflective marker) emitting into the space, the IR transmitter can be configured to emit IR radiation pulses into the space. For example, pulses can be emitted at a frequency between approximately 60 Hz and approximately 100 Hz (e.g., at intervals between approximately 10 ms -1 and approximately 16 ms -1 ). The receiver can be configured to receive the pulses (e.g., the shutter speed of the IR camera can be set to the same frequency as the frequency at which the IR transmitter emits pulses, and / or synchronized using a wired or wireless signal). This can save energy while still being able to track an object in near real-time. In embodiments where the IR transmitter emits into the space in a floodlight manner, the energy savings are even more significant.

[0078] As used herein, the term "space" is used to represent the three-dimensional volume in which a first object is located or moves. In other words, the term "space" is the physical three-dimensional space in which the first object is located, used, or operated.

[0079] The first object can be any type of object. The first object can be stationary or moving. The first object can be automatic, semi-automatic, or fully controlled, e.g., by an operator or engineer.

[0080] In some embodiments, the space can be a manufacturing space, such as a factory, warehouse, outdoor construction site, or any other location where manufacturing is performed, etc. In this sense, the space is the physical three-dimensional space or volume in which manufacturing is carried out. In such embodiments, the first object can be a robot, drone, machine, or any other mechanical device engaged in manufacturing tasks. The first object (e.g., a robot or drone) can perform manufacturing tasks in a fully automatic, semi-automatic, or manual control manner. The computer node 100 and / or method 300 can be used to send instructions, guidance, or more general information that can be used in the control process of the first object.

[0081] Figure 2 An example is shown where the space 200 is a construction site where a building 205 is being constructed. The construction site includes drones and / or robots 201. The drones and robots are configured to move around the building during construction. In Figure 2In this case, the object under construction is building 205. However, it should be understood that this is merely an example, and object 205 can be any other object (such as a submarine, a bridge, a large vehicle, an aircraft, etc.). In this example, the drone 201 is equipped with a marker 202 (for example, the drone 201 is an example of the first object as described herein). In this example, the marker 202 includes a reflective material. The IR emitter 203 emits IR radiation into the space, and this is reflected by the reflective material on the marker 202 and detected by the IR radiation receiver 204. The IR receiver 204 can be a broadband receiver or a narrowband receiver as described above. In this example, the IR emitter 203 and the receiver 204 are mounted on a pole (in the manner of a lamp post or a floodlight stand). In this way, the IR emitter 203 can fill the space with IR radiation. It should be understood that more markers can be placed on the drone 201 to establish multiple point positions of the drone. It should also be understood that the markers can be placed on the building 205 under construction and / or on the person 206 working in the space 200 and on any other drones and / or robots in the space 200. In this way, the relative positions of the drone, the person, and the stationary object in the space can be accurately tracked in real time.

[0082] In some embodiments, the space can be a construction site of a building or other civil engineering projects (such as a bridge or other infrastructure, etc.). In such an embodiment, the method 300 can be used for the purpose of automatic or semi-automatic construction. In such an embodiment, the space is a construction site of a building under construction, a bridge, or other infrastructure. In such an embodiment, the first object can be a robot, a drone, a vehicle, or any other mechanical equipment engaged in construction tasks. In the embodiment where the first object is a structure, the structure can be "positive" (such as a building under construction or a road under paving) or "negative" (such as an excavation pit on the ground or a tunnel under excavation).

[0083] It should be understood that the space can be the space inside an object or a building. Therefore, the space can be the three-dimensional space inside a warehouse or a factory. In other examples, the space can be the space inside an object under construction, such as the inside of a submarine, a train, or a building. In the embodiment where the embodiments herein are applied to the inside of an object under construction (for example, where the first object is a robot or a drone working inside a large object under construction), movable IR sensors and / or emitters, such as the emitter / receiver mounted on a drone, etc., can be deployed inside the object under construction.

[0084] In some embodiments, the space is a field or other cultivation space, and the first object is a farm machine, such as a tractor, a combine harvester, an excavator, or any other agricultural machine, etc. The method 300 described below can be used to detect the position of the machine and provide instructions for the automatic movement of the machine on land. The high-precision, long-distance potential of the methods herein contributes to high-precision automated agricultural applications at night and in adverse weather conditions, thus supporting applications that provide true 24-hour agriculture.

[0085] In some embodiments herein, there is also a second object in the space. In such embodiments, the second object can be a second robot, a drone, or any other type of mechanical equipment.

[0086] In other embodiments, the second object can be a person operating in the space (such as an engineer, a builder, or a construction worker, etc.). In such embodiments, the method 300 described below can be used to coordinate safe operations between the person and the first object operating on the construction site (e.g., machines, robots, and drones, etc.).

[0087] In some embodiments, the first object is a large manufacturing object (such as a submarine, a train, an airplane, a spacecraft, or any other large manufacturing object), and the second object is a drone. In such embodiments, the space is a three-dimensional space in which the large object is being built.

[0088] In some embodiments, the first object is a structure, such as a building, a bridge, an infrastructure, an onshore or offshore wind turbine, a power plant, a railway track, or any other structure, etc., and the second object is a drone or other robot. In such embodiments, the space is a three-dimensional space in which the structure and the drone are respectively located and / or operate. The method 300 below can be used to enable the drone to locate the first point on the structure and perform precise measurements, for example, for the purpose of building monitoring or external inspection of building quality. Thus, the marking at the first point can serve as a reference point for remote, automatic, and high-precision inspection of structures such as buildings. Therefore, IR markings (including but not limited to narrowband infrared markings) can be attached to the finished product under construction as reference points for remote, automatic, and high-precision inspection of it throughout the entire service life of the structure. For example, the methods herein can be used to automatically (e.g., without operator involvement) inspect the external condition of an offshore wind turbine by a drone equipped with an IR sensor, where the IR markings enable the drone to return to the same favorable inspection point with high precision and thus compare, for example, the progression of cracks, paint deterioration, movement, etc. over time.

[0089] Although examples of using a first object and a second object in a space are described herein, it should be understood that the methods herein can be extended to locate, track any number of objects in the space and provide real-time instructions to them.

[0090] While many embodiments of this disclosure relate to construction sites and manufacturing applications, it should be further understood that the techniques described herein are equally applicable to other situations of tracking objects. For example, in some embodiments, the first object is a person. Thus, the embodiments herein can be used to track people or animals. For example, the systems and methods herein facilitate precise tracking of football players on a football field or horses on a racetrack. Thus, the space mentioned above can be a football field, a racetrack, or any other three-dimensional volume where people or animals are desired to be tracked.

[0091] It should also be understood that the embodiments herein are equally applicable to vehicles. For example, the first object can be a racing car. In such an example, the space will be a racetrack. As another example, the first object can be a drone, and the space can be the flight area of the drone.

[0092] As another example, the first object can be a vehicle, and the space can be a portion of the road on which the vehicle travels. In such an embodiment, the vehicle can be equipped with the markers described herein, and the IR sensors can be mounted on roadside infrastructure such as lamp posts or pedestals. Method 300 can be used to track vehicles passing through the space within the field of view of the roadside infrastructure. In such an example, compared to other vehicle tracking methods, the narrowband infrared filters described herein can be advantageously used to track vehicles with high precision at longer distances. Thus, the methods herein can be used as part of an intelligent road infrastructure to monitor vehicles and send instructions to the vehicles. Thus, the methods herein can be used in combination with the inventions described in WO2021 / 051008A, WO2022 / 003343A, and WO2022 / 074406A (the contents of which are incorporated herein by reference).

[0093] The embodiments herein can be applied to manned or unmanned, automatic, semi-automatic, or manual vehicles. Examples of vehicles include but are not limited to aerial vehicles (such as airplanes, drones, helicopters, airships, gliders, and / or any other aerial vehicles, etc.), land vehicles (such as manned or unmanned cars, trucks, motorcycles, vans, and / or any other road-based vehicles, etc.), and water vehicles (such as boats, container ships, liners, sailboats, and / or any other water vehicles, etc.). Thus, the methods herein can be used to track and / or send command data to control vehicles in air, land, or water spaces.

[0094] Now turning to Figure 3a, according to some embodiments of the present disclosure, it is a computer-implemented method 300 for detecting a first object in a space. The method 300 can be executed by the aforementioned computer node 100. The method 300 can be applied to detect and monitor various object types in a variety of spaces, as described in the examples above. Briefly, the method 300 includes: i) receiving 302 data related to one or more detections of infrared radiation by at least one infrared sensor, the infrared radiation coming from a first marker placed at a first point on the first object. In the second step, the method includes ii) determining 304 the position of the first point in the space based on the position of the detected signal in the received data. In some embodiments of the present disclosure, in step 302, the data is received from a narrowband infrared sensor, and the data indicates one or more detections of narrowband infrared radiation from the first marker by the narrowband sensor.

[0095] The data received in step 302 indicates one or more detections of narrowband infrared radiation from a first marker on the first object (e.g., emitted or reflected from the first marker). The data can be obtained from more than one infrared sensor. In such an example, different infrared sensors can have different advantageous points of entry into the space.

[0096] The data can be in the form of an infrared image (such as a near-infrared image, etc.) showing an infrared photograph of the space. In an example where the data is an infrared image, the method can include a preprocessing step, such as an image processing step (such as image segmentation, etc.), to identify the first marker in the image. In other examples, the data can include a file indicating the position and / or intensity of infrared sources in the space. However, these are merely examples, and any other data of the relative position of infrared sources in the image can be used equally.

[0097] In step 304, the method 300 includes ii) determining 304 the position of the first point in the space based on the position of the detected signal in the received data. For example, techniques such as triangulation or multilateration can be used to determine the position of the first point in the space. Those skilled in the art are familiar with the principle of triangulation, whereby the detection of a first marker in the space by two or more different IR sensors at different advantageous points (or viewing perspectives) can be used to determine the position of the first marker based on the position offsets perceived by the different IR sensors. Thus, in this way, even at long distances, under adverse weather conditions, or at night, the position of the first marker can be determined accurately and quickly. It should be understood that the position measurement from triangulation will be a relative position. However, if the absolute position of the IR sensor is known, the relative position data can be converted into absolute position data.

[0098] It should be understood that steps i) and ii) can be repeated to track the movement of the first point on the first object over time. For example, as Figure 3b shown, method 300 can be repeated to obtain the position data stream of the first marker 306b. As used herein, iteration can mean continuously or near-continuously (subject to the limitations of computing power). One of the advantages of the method herein is that the computing power required to determine the position using IR is relatively low. This is due to the high signal-to-noise ratio of the signals involved (especially in the case of using narrowband infrared detection). Thus, the method herein allows for obtaining a fast and computationally inexpensive position information stream.

[0099] Thus, the position data stream can be obtained in real time (or near real time). Tracking the position of a single point in this way is useful in a range of scenarios, including but not limited to tracking the position of a drone in a factory or construction environment; tracking the position of a person in a factory or construction environment; tracking a football player on a football field, a car on the road, or in any other scenario where a single point can be used to track an object.

[0100] In an embodiment where the first point is located on a moving part of a machine, method 300 can be used to accurately, continuously, and with low latency measure the position of the main moving components of a factory and the machine, such as a robotic arm, an excavator limb, a dump truck lifting device, and / or a crane hook, etc.

[0101] In some embodiments, as described above, more than one marker is attached to the first object, so that the infrared sensor can see more than one marker at any time. Thus, in some embodiments, method 300 can also include repeating steps i) and ii) for infrared radiation from two or more different markers placed at two or more different points on the first object to determine two or more positions on the first object. It should be understood that steps i) and ii) do not have to be repeated in sequence (e.g., one after another) for each marker. For example, step i) can be performed on all markers at once, and subsequently step ii) can be performed on all markers at once. In other words, steps i) and ii) can be performed as a parallel computing process.

[0102] Two or more markers can be used in various embodiments. For example, IR markers can be attached to components of an item under construction to serve as reference points for the production, assembly, construction, or manufacture of the final item. As another example, the markers can facilitate the precise fitting of major components on a ship or aircraft production line, or the assembly of prefabricated building parts on a construction site.

[0103] In some embodiments, multiple markers can be applied to the first object, and upon detection, these markers can form a point cloud for (e.g., narrowband) IR detection. Figure 3cA method of processing point clouds is shown. In step 302c, method 300 may include receiving data related to point cloud detection of narrowband infrared radiation by at least one narrowband infrared sensor, the narrowband infrared radiation being from a plurality of markers placed at a plurality of points on a first object. In step 304c, the position, orientation, and / or pose information of the first object in space is determined based on the positions of the detected signals in the received data. Steps 302c and 304c are repeated to obtain 306c such a position, orientation, and pose information stream of the first object. As used herein, position may include the x, y, z coordinates of an object in space; orientation may relate to, for example, the pitch, yaw, and roll of an object, and position information may relate to the position or "pose" of any joint part on the first object.

[0104] Thus, the data may include the positions of the plurality of markers in the image, for example, in the form of a "point cloud" of the markers. Those skilled in the art will be familiar with point clouds and the different methods for mapping position, orientation, and / or pose information from point clouds to three-dimensional surfaces or structures.

[0105] For example, the points in the point cloud can be mapped (or fitted) to an object model to determine the positions of specific parts (levers, arms, mechanical connectors, etc.) of the first object. An example of a suitable model is a three-dimensional deformable mesh structure, where each vertex in the mesh corresponds to a marker on the first object. Thus, the movement of the object can be represented by the deformation of the three-dimensional mesh structure. Such a mesh will have some static points and some dynamic points (to accommodate the fixed and articulated parts of the first object respectively), but with constrained and known degrees of freedom in the machine body reference frame. The model of the object can be linked back to the design model of the machine tool (e.g., in three-dimensional (3D) computer-aided design (CAD)), and the emitter / reflector can be designed and analyzed to optimize machine tool tracking.

[0106] In addition, certain markers can be used as "anchor" points (or boundary conditions) for the fitting process. For example, markers with a unique pattern or frequency of IR reflection or emission can be associated with specific points in the mesh. It should be understood that constraints can be imposed on the allowed deformation of such a mesh according to the allowed movement and range of joint movement of the first object. In this way, the embodiments herein use a "customized point cloud" based on emitters / reflectors at priori known points on the machine tool and structure. The customized point cloud is superior to methods such as lidar scanning (which produces very complex point clouds, e.g., 200x200 points, each point having a distance to the reflecting surface) or photogrammetry, because the point cloud can be designed to maximize computational efficiency, enabling the mesh to be fitted faster and with near-zero latency, allowing closed-loop machine control.

[0107] In addition, as described above, by comparing the detected positions of the markers in two or more different marker images taken from different vantage points, triangulation can be used to determine the position of one or more markers. In examples where there are a large number of markers and there are problems in identifying individual markers in each image, the signals from the markers can be distinguished from each other, for example, in terms of the emitted frequency or pattern.

[0108] As another example, machine learning can be used to predict the position, orientation, and localization information of an object based on a point cloud of points.

[0109] Those skilled in the art will be familiar with machine learning and methods of training models using machine learning processes. But in brief, a model, also referred to as a "machine learning model", includes a set of rules or a group of (mathematical) functions that can be used to perform tasks related to the data input into the model. A model can be taught to perform a wide variety of tasks on input data, examples including but not limited to: determining the label of the input data, performing a transformation of the input data, predicting or estimating the values of one or more output parameters based on the input data, or generating any other type of information that can be determined from the input data.

[0110] In supervised machine learning, the model learns from a training dataset that includes example inputs and the corresponding ground truth (e.g., "correct") outputs for each example input. Typically, the training process involves learning the weight values or bias values of the model in order to adjust the model to reproduce the ground truth outputs of the input data. Different machine learning processes are used to train different types of models. For example, machine learning processes such as backpropagation and gradient descent can be used to train neural network models.

[0111] The models herein can generally be any type of machine learning model that can be trained to take as input a point cloud of IR signals (or data indicative of that point cloud), as described above, and output a prediction of the position, orientation, and / or position / pose information of the first object. Examples of suitable models include but are not limited to: neural network models, linear regression models, and decision tree models.

[0112] In some examples, the model is a neural network. There are various open-source neural network models applicable to the embodiments described herein, such as neural networks in free software machine learning libraries (scikit-learn) etc., which are described in the paper titled “Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011 (Free Software Machine Learning Library: Machine Learning in Python, Pedregosa et al., Journal of Machine Learning Research 12, pages 2825-2830, 2011)”. Generally, the features herein can be obtained using the default neural network parameter settings described in the guide.

[0113] There are different possible combinations of input parameters and output parameters. For example, the input can be in the form of an image, such as an image showing a point cloud. Such an image can be supplemented with additional data, for example, such as the position of an IR sensor performing detection or other identifications. As another example, the input can be a list of vectors corresponding to the center points of each detected IR signal in the point cloud. In other examples, the input can be raw data from an IR sensor. Those skilled in the art will understand that these are merely examples, and other inputs can be provided in addition to the above inputs.

[0114] The neural network can also take as input data related to the previous position, orientation, and / or pose information of the first object. Since this information has a strong causal connection with the current position, orientation, and / or pose of the first object, this can improve the prediction of the neural network.

[0115] Regarding the output parameters, the neural network can be trained to output any type of position, orientation, and / or pose data, such as the relative position of the first object in space, the absolute position of the first object, the pitch, roll, and / or yaw of the first object, the position indication of the end or articulated parts (e.g., arm, claw, shovel, or the like) of the first object, or any other information related to the position, orientation, or mechanical pose or state of the first object, etc.

[0116] The neural network can also be trained to output other information about the first object that can be inferred from the point cloud. For example, the type of the first object (e.g., the type or structure of a drone, machine, vehicle, etc.), its size or extent.

[0117] As described above, a neural network can be trained using training data that includes example inputs and the "correct" or ground truth outputs for the example inputs. The training dataset can be established in various ways. For example, by simulating the first object in different orientations, such as at different distances, angles, and orientations from the viewpoint, and simulating the corresponding point clouds for each position and orientation. Such simulations can be performed using computer-aided design (CAD) tools. The training dataset can be established in a systematic manner by sequentially sampling the entire possible "position / orientation / orientation" space available to the first object in physical space. In this way, a training set can be established that can train the neural network with high accuracy in a wide range of scenarios.

[0118] It should be understood that if more input parameters are provided, the training dataset can be extended to sample possible orientations and the resulting patterns given the additional parameters (and parameter space), for example, the point clouds generated by using different combinations of IR markers (e.g., emitting different frequencies, pulse signals, or coded signals, etc.).

[0119] It should also be understood that the training set can include point clouds representing more than one object. For example, a neural network can be trained to predict different types of interactions (or mutual manipulations) between a first object and a second object, and / or output a proximity alert if the point cloud in the input indicates that the first object and the second object have become too close, etc.

[0120] It should also be understood that the training dataset can be established through real measurements, for example, by observing the point clouds that can be observed when the first object performs different manipulations, and recording the resulting point clouds and position / orientation / orientation data for establishing the training dataset. It should also be understood that the training dataset can also be composed of a combination of real data and simulated data.

[0121] Figure 4b Some example pairs of positions and point clouds are shown (and will also be discussed below). In this example, the first object in the form of machine 400 is shown side by side with its point cloud 402 at a first position and orientation in Figure 4b i). The same object 400a with the same pose detected at a farther position is shown in Figure 4b ii) along with its associated point cloud 402a. Figure 4b iii) of shows the same machine 400b in a different pose along with its associated point cloud 402b. Figure 4b iv) of shows the second object in the form of a truck 404 and its associated point cloud 406. Thus, in this example, the training dataset can consist of different objects in different poses at different positions.

[0122] The following papers describe various other example algorithms that may be modified for the purposes of this article:

[0123] "LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds" (2019) Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabas Poczos, Yaser. This paper uses autoencoders to fit deformable, articulated mesh point clouds. These types of techniques can be used in the embodiments herein to obtain position data of moving parts, such as robotic arms of robots or machines.

[0124] "Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image" Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, Michael J. Black, 2016 European Conference On Computer Vision (ECCV). This paper describes methods for determining human poses. This can be generalized to determining the poses of mechanical objects from point clouds.

[0125] "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas (2017). This paper describes an algorithm by which points in a point cloud are directly fed into a neural network, which can be trained to elucidate objects. Such a neural network can be trained to classify based on the position of the object.

[0126] It should be understood that the papers cited above are only examples, and other methods of identifying objects and their orientations from point clouds of IR signals reflected and / or emitted from IR markers can also be used.

[0127] Thus, as described above, multiple markers can be used to generate a point cloud of the first object, from which the position, orientation, and pose information of the first object can be determined.

[0128] By repeating method 300 over time in an iterative manner (e.g., frame-by-frame processing of the IR digital video data of the first object), the position can be tracked and monitored over time to determine motion parameters such as the speed, trajectory, and acceleration of the first object.

[0129] As described above, the point cloud can also be processed to determine the orientation of the first object. For example, if the first object is a drone, the point cloud can be used to determine the pitch, yaw, and roll of the drone (e.g., this information can be output from the model described above). Thus, the above method can be used to track an object in real time and determine its pose, orientation, and motion attributes.

[0130] In some embodiments, the position, orientation, and pose information can be used to send instructions to the first object. For example, instructing the first object to perform an action or maneuver. Examples of instructions that can be sent to cause the first object to perform a maneuver include, but are not limited to: accelerating, stopping, starting, or turning, instructions to articulate the arm or moving parts of the first object. Examples of instructions that can be sent to cause the first object to perform an action include, but are not limited to: actions related to fixing two components together, detaching one component from another, actions associated with a robotic arm (scooping, lifting, tilting, etc.), waving, etc., or any other action that can be performed by the first object.

[0131] Thus, in embodiments where the first object is a robot, drone, or mechanical object for a factory or construction environment, the above method 300 can be used to monitor the first object and provide instructions to the first object as part of an automated factory or automated construction program.

[0132] Turning now to other embodiments, as described above, these principles can be applied to more than one object, and the resulting data can be used to determine and coordinate the interactions between the objects. For example, in some embodiments, method 300 can include repeating steps i) and ii) on a second marker at a second point placed on a second object to determine the position of the second point in space. Then, the determined positions of the first marker and the second marker can be used to determine the relative proximity of the first object and the second object.

[0133] In response to determining that the relative proximity is less than a first threshold proximity, the relative proximity can be used to initiate a proximity warning (e.g., to an operator or other system), or to send a command to stop or change the movement of the first object or the second object. In an embodiment where the first object is a moving machine and the second object is a person, method 300 can be used to accurately, continuously, and with low latency measure the orientation of an operator working near the moving machine. Such information can be used to issue a proximity warning to the operator or to a person nearby, or to provide a basis for a safety interlock, which will reduce the level of accidents and / or reduce the economic impact of an accidental collision.

[0134] It should be understood that a first plurality of markers can be placed on the first object and a second plurality of markers can be placed on the second object, such that a first point cloud and a second point cloud are respectively detected. Each point cloud can be processed using any of the above techniques (e.g., by triangulation, by fitting a deformable mesh to each point cloud, and / or using machine learning) to respectively determine the relative position, orientation, and azimuth / pose information of the first object and the second object.

[0135] For example, the first plurality of markers can be distinguished from the second plurality of markers by using different patterns or frequencies for each plurality of markers, making the process more efficient. Additionally, the markers in the first plurality of markers can be configured to emit or reflect IR radiation to the second plurality of markers at staggered intervals, such that at any given time the IR detector can only see one set of the plurality of markers. Thus, the first plurality of markers can appear to "flicker", followed by the second plurality of markers appearing to "flicker". If the frequency of the flashes is high enough, then each object can still be tracked in real time.

[0136] In other embodiments, the relative proximity and / or the position, orientation, and azimuth / pose information can be used to coordinate the manipulation between the first object and the second object. For example, method 300 can be used to guide the robotic arm of a first object (such as a robot or a machine) towards a specific point on a second object in manufacturing; to guide a first object in the form of a robot to mount a first component onto a specific part of a second object being manufactured; to guide a pair of robots in a refueling operation; to guide an automated truck during loading; or to guide any other automated or semi-automated process between two objects. Thus, method 300 can be used to achieve a higher level of automation in production, assembly, construction, and manufacturing processes.

[0137] Figure 4a There is shown a first object 400 in the form of a machine for a factory or a construction site, which has a plurality of markers 202. When detected by an infrared detector 204, the plurality of markers produce a point cloud 402. As described above, the appearance of the point cloud will change depending on the advantageous point of the IR detector 204, and thus, using the principle of triangulation or any of the other above techniques, different advantageous points can be used to determine the position of the points. In this embodiment, asFigure 3c The method 300 shown can be applied to a point cloud to obtain position, orientation, and pose information flows of a first object (e.g., a machine).

[0138] Figure 4b is shown Figure 4a an example point cloud observable by the machine in, and how the point cloud changes as a function of distance ( Figure 4b ii) of and the robotic arm joints ( Figure 4b iii) of increases. Figure 4b A point cloud associated with a second object is shown. As described above, such an object-point cloud pair can be used as training data to train a machine learning model (such as a neural network, etc.) to take the point cloud as input and provide a prediction of the machine type and / or orientation as output.

[0139] In this embodiment, Figure 3c the method in may further include Figure 4c steps 308c and 310c shown in, whereby Figure 3c the position, orientation, and pose information flows output by the method steps shown in are further used to determine an action or manipulation to be performed by the first object (such as using techniques such as reinforcement learning, pre-programmed rules, optimization, or any other suitable process, etc.), and in step 310c, the first object is instructed to perform the determined action or manipulation.

[0140] Figure 5aShows an example system used in some embodiments of this document. In this embodiment, the space or "sensing environment" 500 is a factory or a construction site where large items (such as trains, submarines, bridge components, etc.) 512 are being manufactured. The large item 512 has a plurality of markers on its surface (the markers can emit or reflect IR radiation). IR sensors 204 are located at different points in the space and send detection data regarding the markers on the large item 512 to the computer node 100. In this embodiment, the IR sensors can be broadband IR sensors or narrowband IR sensors, as described above. In this embodiment, the computer node 100 includes three computing modules. The IR sensor control module 502 is configured to perform step 302 and receive data related to one or more detections of infrared radiation within a narrowband by at least one narrowband infrared sensor, the infrared radiation within the narrowband coming from a first marker placed at a first point on a first object. The IR sensor module 502 repeats step 302 for the data related to the detection of each marker on the large item 512. The received data is sent to the orientation determination engine 504 to perform step 403 and determine the position of each point corresponding to a marker in the space based on the position of the detected signals in the received data. The orientation determination engine 504 can employ any of the techniques described above with respect to step 504 (e.g., triangulation, fitting of a grid-based model, and / or machine learning) to determine the position information based on one or more markers on the object.

[0141] The IR sensor control module 502 and the orientation determination engine 504 perform the same process on the markers on various robots 508a, 508b, and 508c that are constructing the large item 512. The method 300 is used to determine the position, orientation, and pose information (e.g., the articulation of its components) of each of the robots 508a, 508b, and 508c.

[0142] The robot motion controller module 506 uses the determined position, orientation, and pose information to determine, for example, the orientation and / or positioning / pose information regarding the robot and determine the actions to be performed by the robot during the construction of the large item 512.

[0143] Markers can also be placed on a person 510 working in the space 500 to track the movement of the person 510 relative to the robots 508a, 508b, and 508c. In this way, a proximity alert can be issued if someone gets too close to the automated machines.

[0144] Figure 5bA method for detecting a first object is shown, where the first object is a robot, a drone, or other machine in a manufacturing or construction space. The method includes, in step 502, receiving data related to one or more detections of infrared radiation by at least one infrared sensor, the infrared radiation coming from a first marker placed at a first point on the first object. In step 504, the method 500 then includes determining the position of the first point in space based on the position of the detected signal in the received data.

[0145] Accordingly, a system facilitating an automated factory or construction site is provided. It should be understood that the above details are merely examples, and the computer node 100 may include a different number of modules or different combinations of modules than those Figure 5a shown. Additionally, the number of robots and / or IR sensors may be different from those shown in the figure.

[0146] Now turning to FIG. 6, FIG. 6 shows an embodiment herein where markers are used as reference points in the construction, monitoring, or inspection of large objects such as buildings, bridges, skyscrapers, or a wind turbine as shown in FIG. 6.

[0147] During the construction process, IR markers can be attached to components of the item being constructed as reference points for remote, automated, and high-precision inspection of the final product. For example, the external condition of high-rise construction is inspected by a drone equipped with an IR sensor.

[0148] However, markers can also be used to monitor a finished product during its life cycle. IR markers can be attached to an object as reference points for remote, automatic, and high-precision inspection of the object during its entire working life, during both day and night and under various weather conditions. For example, as Figure 6a shown, the marker 602 can be placed on the wind turbine 600 for automatic (i.e., without operator involvement) inspection of the external condition of the wind turbine 600 by a drone 604 equipped with an IR sensor, and is configured to execute the method 300 to locate the marker 602 and enable the drone 604 to return to the same inspection advantageous point with high precision, and thus be able to compare, for example, the progression of cracks and paint deterioration over time. Thus, in this way, the method 300 can be used for precise comparative inspection of large structures over time, even under adverse conditions such as an offshore wind turbine, regardless of whether the turbine blades are rotating. If the turbine blades are rotating (e.g., the first object is moving), the method 100 can be used to track the position throughout the movement.

[0149] Figure 6b is shown to be available by Figure 6aSteps performed by computer node 100 in the illustrated embodiment. In this embodiment, computer node executes steps 302 and 304 of method 300 as shown in FIG. 3. In this embodiment, the data in step 302 is received from unmanned aerial vehicle 604 and is used in step 304 to determine the position of a first point on the wind turbine. Then, computer node 300 executes step 306d as shown in Figure 6b and uses the determined position of the first point in space as a reference point to cause the unmanned aerial vehicle to perform measurements on a first object. Before performing the measurements, the unmanned aerial vehicle is aligned with the first object using the reference point. Computer node 100 can cause the unmanned aerial vehicle to perform measurements by, for example, sending instructions to the unmanned aerial vehicle.

[0150] Turning now to other embodiments, it should be understood that method 300 can be implemented in a computer program. For example, a computer program product can include a computer-readable medium having computer-readable code embodied thereon. The computer-readable code can be configured such that when executed by a suitable computer or processor, the computer or processor performs one or more methods described herein (such as method 300, etc.).

[0151] Computer programs can take different forms, for example, source code, compiled code, executable code, or any other type of code. It should be understood that the source code of a computer program can be written in many different programming languages and can adopt different architectural designs. For example, the functions described herein can be divided into various different subroutines. In addition, those skilled in the art should understand that many different ways of dividing the functions among different subroutines will be possible. The subroutines can be stored together in an executable file to form a self-contained program. In addition, a computer program can call external and / or standard computer code libraries to perform certain subtasks associated with the functions described herein.

[0152] In another embodiment, a computer program product includes a non-transitory computer-readable medium having stored thereon a computer program as described above. Examples of computer-readable media include, but are not limited to, read-only memory (ROM) (such as compact disc read-only memory (CDROM), semiconductor ROM, etc.) or magnetic recording media (such as a hard disk, etc.).

[0153] In another embodiment, a carrier containing a computer program. Examples of carriers include, but are not limited to, electronic signals, optical signals, radio signals, computer storage media or the like. A carrier of a computer program can be any entity or device (such as hardware) that is capable of carrying the program. For example, the carrier can be the computer-readable medium as described above. In other examples, the carrier can be a transmissible carrier, such as an electronic or optical signal, which can be transmitted via a cable or an optical fiber cable or by radio or other means.

[0154] From the study of the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these claims cannot achieve advantages. Any reference signs in the claims should not be construed as limiting the scope. Furthermore, it should be understood that the features, advantages, and functions of the different embodiments described herein can be combined without departing from the spirit or scope of the disclosure herein.

Claims

1. A computer-implemented method for detecting a first object in a space, the method comprising: i) receiving data related to one or more detections of infrared radiation within a narrow band by at least one narrow-band infrared sensor, the infrared radiation within the narrow band being from a first marker placed at a first point on the first object; and ii) determining the position of the first point in the space based on the position of the detected signal in the received data.

2. The method according to claim 1, wherein The narrow-band signal is centered at approximately 800 nanometers and has a frequency range of approximately + / - 10 nanometers or approximately + / - 20 nanometers.

3. The method according to claim 1 or 2, further comprising: repeating steps i) and ii) in an iterative manner to track the movement of the first point on the first object over time.

4. The method according to any one of the preceding claims, further comprising: instructing an infrared emitter to emit infrared radiation pulses into the space, wherein the pulses are emitted at a first pulsation frequency; and wherein steps i) and ii) are repeated to detect the reflection of each pulse from the first marker.

5. The method according to claim 4, wherein, The pulses are emitted at a pulsation frequency between approximately 60 Hz and 100 Hz.

6. The method according to any one of the preceding claims, further comprising: repeating steps i) and ii) for infrared radiation from two or more different markers placed at two or more different points on the first object.

7. The method according to any one of the preceding claims, wherein, Step i) includes: receiving data related to point cloud detection of infrared radiation within a narrow band by at least one narrow-band infrared sensor, the infrared radiation within the narrow band being from a plurality of markers placed at a plurality of points on the first object.

8. The method according to claim 7, further comprising: In step ii), determining the orientation, azimuth, or pose of the first object in the space based on the point cloud.

9. The method according to claim 8, further comprising: using the position, orientation, and azimuth information stream of the first object to determine an action or maneuver to be performed by the first object; and and sending a control signal to the first object to cause the first object to perform the action or the maneuver.

10. The method according to any one of the preceding claims, further comprising: repeating steps i) and ii) for a second marker placed at a second point on a second object to determine the position of the second point in the space; and and using the determined positions of the first marker and the second marker to determine the relative proximity of the first object and the second object.

11. The method according to claim 10, further comprising: initiating a proximity warning; or responding to determining that the relative proximity is less than a first threshold proximity by sending a command to stop or change the movement of the first object or the second object.

12. The method according to any one of the preceding claims, wherein, The first object is a robot, a drone, a mechanical object, or a person.

13. The method according to any one of the preceding claims, wherein The space is a construction site, a factory, or a field.

14. The method according to any one of the preceding claims, wherein, The first object is a building, a bridge, or a wind turbine.

15. The method according to any one of the preceding claims, wherein, The second object is a second robot, a second drone, or a person.

16. The method according to any one of claims 1 to 8, wherein, The data in step i) is received from a drone, and wherein the method further comprises: Using the determined position of the first point in the space as a reference point, the drone measures the first object. Before the measurement, the drone is aligned with the first object using the reference point.

17. The method according to any one of the preceding claims, wherein, The step of determining the position of the first marker includes: Triangulating the position based on the detected signals in the received data.

18. The method according to any one of the preceding claims, wherein, The method is carried out at night, outdoors or under adverse weather conditions.

19. A computer node for tracking a first object in space, the node comprising: One or more computer processors configured to: i) Receive data related to one or more detections of infrared radiation within a narrow band by at least one narrow-band infrared sensor, the infrared radiation within the narrow band being from a first marker placed at a first point on the first object; And ii) Determine the position of the first point in the space based on the position of the detected signals in the received data.

20. The computer node according to claim 19, further configured to perform the method according to any one of claims 2 to 18.

21. A system for tracking a first object in space, the system comprising: A first marker placed at a first point on the first object; At least one narrow-band infrared sensor including a narrow-band infrared filter for detecting infrared radiation within a narrow band; And A computer node configured to: i) Receive data related to one or more detections of the infrared radiation within the narrow band by the at least one narrow-band infrared sensor, the infrared radiation within the narrow band being from the first marker; And ii) Determine the position of the first point in the space based on the position of the detected signals in the received data.

22. The system according to claim 21, wherein, The narrow-band infrared filter is centered at about 800 nanometers and has a frequency range of about + / - 10 nanometers or about + / - 20 nanometers.

23. The system according to claim 21 or 22, wherein The marker includes an infrared emitter, a reflective material or a retroreflective material.

24. The system according to claim 21, 22 or 23, wherein, The system further includes: One or more infrared lamps configured to irradiate the space with infrared radiation so that the infrared radiation is reflected from the marker.

25. The system according to claim 24, wherein, The one or more infrared lamps are configured to emit infrared radiation pulses having a first pulsation frequency, and wherein the computer node is configured to repeat steps i) and ii) for each pulse.

26. The system according to any one of claims 21 to 25, wherein, The computer node is further configured to perform the method according to any one of claims 2 to 18.

27. A method for tracking a first object, wherein, The first object is a robot, a drone or other machine in a manufacturing or construction space, and the method includes: i) Receive data related to one or more detections of infrared radiation by at least one infrared sensor, the infrared radiation being from a first marker placed at a first point on the first object; and ii) Determine the position of the robot, drone or other machine in the space based on the position of the detected signals in the received data.

28. A computer node for detecting a first object, wherein, The first object is a robot, a drone or other machine in a manufacturing or construction space, and the node includes: One or more processors configured to: i) Receive data related to one or more detections of infrared radiation by at least one infrared sensor, the infrared radiation being from a first marker placed at a first point on the first object; and ii) Determine the position of the robot, drone or other machine in the space based on the position of the detected signal in the received data.

29. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 18 or 27.

30. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 18 or 27.

Images