Autonomous radiography inspection system

Abstract

Method and systems for autonomous radiography inspection of a physical object are described wherein the method comprises: controlling a first drone comprising a first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object; positioning the first drone at a target position associated with one of the one or more radiography waypoints, the positioning including: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position; and, executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source.

Description

[0001] Autonomous radiography inspection system

[0002] Technical field

[0003] The embodiments relate to autonomous radiography inspection, and, in particular, though not exclusively, to autonomous radiography inspection systems, inspection methods using an autonomous radiography system and a computer program product for executing such methods.

[0004] Background

[0005] Large structural objects such as wind turbines in a windfarm, power plants or factory sites require inspection to identify defects and monitor structural health of these objects in order to maintain reliability and prevent costly catastrophic failures. Internal defects of structural objects, such as wind turbine blades, piping and electrical cables for example, can be detrimental to their structural integrity and may lead to premature failure. For example, wind turbine blades can be considered the most critical part of a wind turbine because of their size, exposure to weather conditions and mechanical stress when converting the wind energy into mechanical kinetic energy. Since the process to replace a wind turbine blade is a very expensive, time-consuming repair process. This is mostly considered as an obstruction factor to the expansion of global wind turbine installation. It is therefore essential to prevent damage to the wind turbine blades or at least detect possible defects that may evolve in structural damage as early as possible.

[0006] LIS2019 / 0041856 describes an aerial radiography inspection system comprising a source and detector drone. The drones are controlled based on GPS or satellite navigation and a flight plan that includes information about the timing and location of capturing X-ray images of an object. Alignment of the source and detector UAV is achieved by markers that are attached to the drones. In operation, the markers are detected using a camera and used to align the radiography source with the radiography detector relative to the location of the object that needs to be inspected. Positioning of the drone and alignment of the source and detector based on visual markers attached to a drone is often not possible or at least challenging as the dimensions of the object that needs to be imaged is much larger than the dimensions of the drones. Hence, often the object blocks visual detection of the markers, even if the markers are attached to arms so that the markers extent from the main body of the UAVs. This way, accurate positioning and alignment of the drones and the source and detector, which are essential for high quality radiography imaging, cannot be achieved. A model-based localization process by registering a measured three dimensional (3D) view to a stored 3D target model may improve UAV positioning. However, coordinate- based or registration-based navigation alone does not ensure stabilization of radiographic projection geometry during exposure under wind, vibration, and actuation transients.

[0007] Further challenges for radiography inspection system relate to power consumption of the drones. The large structural objects are often located in remote areas, wherein one inspection mission preferably should cover inspection of an entire object or group of objects. Power consumption of the drones not only includes power consumption of the motors of the drones but also power consumption of the data processing modules of the drones for navigation and inspection, which may require real-time processing of multimodal 3D image data using GPUs.

[0008] Hence, from the above it follows that there is a need in the art for improved radiography-based inspection of objects. In particular, there is a need in the art for methods and systems that allow accurate autonomous radiography-based inspection of large structural objects at sites that are not easy to access such as windfarms, power plants and factory sites.

[0009] Summary

[0010] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, a method or a computer program product.

[0011] Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

[0012] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

[0013] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0014] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Python, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the person's computer, partly on the person's computer, as a stand-alone software package, partly on the person's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the person's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0015] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0016] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0017] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0018] In an aspect, the embodiments may relate to a computer-implemented method for autonomous positioning a first drone relative to a physical object, wherein the method may comprise: controlling a first drone device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more waypoints, each waypoint being associated with a target position relative to a region of interest (ROI) of the physical object; and, positioning the first drone at a target position associated with one of the one or more waypoints, the positioning including: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; and, registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position.

[0019] In a further aspect, the embodiments may relate to a computer-implemented method for autonomous radiography inspection of a physical object, the method comprising: controlling a first drone comprising a first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the first radiography device being connected to the first drone using a controllable pivoting structure, preferably a gimbal structure, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object and a target angular orientation, preferably a 3D alignment vector, for aligning the radiography device with the ROI; positioning the first drone at a target position associated with one of the one or more radiography waypoints; and, executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source; wherein the positioning of the first drone includes:

[0020] - determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone;

[0021] - registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position;

[0022] - controlling the pivoting structure to align an orientation of the first radiography device with the target angular orientation based on angular information generated by a first inertial measurement unit (I MU) connected to the first drone and / or a second IMU connected to the first radiography device.

[0023] Hence, the drone continuously scans the point clouds generated by its perception sensors and matches these scans with the 3D model of the object, e.g. a wind turbine. This involves matching the scanned point cloud to the appropriate portion of the 3D map using a registration algorithm. The registration algorithm may output a transformation matrix, e.g. a pose transformation in a world fixed frame. In an embodiment, the algorithm may also output a registration error residual (R) which is indicative of an error associated with the registration process (e.g., mean or robust mean of point-to-plane errors after outlier rejection). The transformation matrix aligns the two coordinate frames (i.e. the coordinate frame of the drone and the coordinate system of the 3D object), which allows precise determination of the waypoints associated with the 3D object that is associated with the mission map. The residual metric may be monitored in real-time and used for exposure gating and / or post-acquisition quality assessment.

[0024] In an embodiment, the 3D alignment vector may define a line-of-sight (LOS) through the ROI. In another embodiment, each or at least part of the radiography waypoints may further include a registration error threshold associated with the position (registration) of the first drone and / or an angular error threshold associated with the orientation of the radiography device.

[0025] In an embodiment, each UAV, e.g. the first and / or second drone, may be operated based on a nested alignment control process using a UAV-body IMU and a radiography device gimbal IMU, wherein an outer loop of the nested alignment control process is configured to minimize a LOS angular error relative to the alignment vector and an inner loop of the nested alignment control process is configured to stabilize the gimbal using rate feedback.

[0026] In an embodiment, radiography exposure, i.e. execution of the radiography imaging process, may be permitted only if the computed errors, e.g. the registration error < R and / or the LOS angular error < 0 meet a predetermined condition for a predetermined time. For example, in an embodiment, the registration error may be smaller than the registration error threshold and the LOS angular error may be smaller than the an angular error threshold for a predetermined time, the so-called dwell time T.

[0027] Hence, the embodiments provide autonomous radiography methods and systems, which, in addition to accurate positioning, provide a stable projection geometry at the moment of exposure. The system combines registration-grade pose estimation with an alignment control process that generates an error value. The radiography imaging process may be executed based on the error value. For example, the radiography imaging process may be executed when the error value is below a predetermined threshold.

[0028] In an embodiment, the alignment control process may include a dual IMU nested alignment control, wherein the radiography imaging process may be executed when an error associated with the registration process and an error associated with a registration residual threshold and a line-of-sight (LOS) angular error threshold are met and held for a dwell time, and trigger exposures with deterministic inter UAV jitter sufficiently small to maintain projection consistency, thereby yielding radiographs suitable for pose constrained 3D fusion with quality based rejection. In some embodiments, auxiliary sensing platforms as nodes in specific formations and with coordinated mission, can refresh the model.

[0029] In an aspect, the embodiments may relate to a computer-implemented method for autonomous radiography inspection of a physical object, wherein the method may comprise: controlling a first drone comprising a first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object; each radiography waypoint may be associated with a 3D alignment vector defining a desired LOS angular orientation through a ROI, and associated positional and angular tolerances used for alignment and exposure gating; positioning the first drone at a target position associated with one of the one or more radiography waypoints, the positioning including: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position; and, executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source.

[0030] In an embodiment, the execution of the radiography imaging process by the first and second radiography device may be synchronized via a direct radio communication channel between the first drone and second drone. In another embodiment, the execution of the radiography imaging process by the first and second radiography device may be synchronized via first radio communication channel between the first drone and a base station and a second radio communication channel between the second drone and the base station.

[0031] The first and second (source and detector) drones may be configured to execute peer-to-peer time synchronization with deadline-based triggering to bound interdrone jitter to J. A base-station relay may be used only as a fallback.

[0032] In an embodiment, each of the one or more waypoints may be further associated with a target angular orientation in a 3D coordinate system, preferably the target angular orientation being represented by an alignment vector, and wherein the first radiography device is connected to the first drone based on a controllable pivoting structure, preferably a gimbal structure.

[0033] In an embodiment, the method may further comprise: controlling the pivoting structure to align the first radiography device with the target angular orientation based on angular information, preferably an angular orientation and an angular rate, generated by a first inertial measurement unit (IMU) connected to the first drone and / or a second IMU connected to the first radiography device.

[0034] In an embodiment, a nested alignment controller may be configured to use a UAV-body IMU and a gimbal IMU to executed a nested alignment control process, wherein an outer loop of the nested alignment control process is configured to minimize LOS angular error relative to the alignment vector and an inner loop of the nested alignment control process is configured to stabilize the gimbal using rate feedback. Execution of radiography exposure may be enabled if R < Rmax and |LOS error| < 0maxare both maintained for at least a dwell time T.

[0035] In an embodiment, the measured 3D view and the constructed 3D view may be point cloud models and a point cloud registration algorithm may be used to register the measured 3D view with the constructed 3D view.

[0036] In an embodiment, the mission map further may includes information about structural features of the target 3D model and wherein registering at least part of the measured 3D view with at least part of the constructed 3D view further comprises: determining first features associated with the measured 3D view; determining second features associated with the constructed 3D view; and, registering at least part of the first structural features with at least part of the second structural features to compute the transformation matrix.

[0037] Each UAV, the first and / or second drone, may be configured to register a measured 3D view to the 3D model to compute a pose transformation in a world fixed frame and positions according to the transformation. In an embodiment, the correspondence set may be pruned by a graph theoretic method retaining a k-core or densest subgraph of mutually consistent matches prior to pose estimation.

[0038] In an embodiment, the execution of the radiography imaging process may be started if both the first radiography device and the second radiography device is aligned with the target angular orientation, the execution of the radiography imaging process including determining one or more radiography images of the ROI.

[0039] In an embodiment, navigating the first drone further may include: if the physical object is outside the visual range of a visual sensor, e.g. a camera, of the first drone, navigating the first drone towards the physical object based a geo-location of the first inspection drone and the geo-location of the physical object, a GPS module of the first inspection drone generating the geo-location of the first drone.

[0040] In an embodiment, navigating the first drone may further include: if the physical object is within the visual range of a visual sensor, e.g. a camera, of the first drone, navigating the first drone towards the physical object based image processing of video frames generated by a camera of the first drone, the image processing including object detection of the physical object and, optionally, pose estimation of the first drone relative to the physical object.

[0041] In an embodiment, the one or more waypoints may further include radiography settings to configure the first radiography device to capture the one or more radiography images when the second radiography device of the second drone exposes the ROI to radiation.

[0042] In an embodiment, the method may further include: evaluating the quality of the one or more radiography images of the ROI and if the quality of the one or more radiography images is below a threshold value, repeating the execution of the radiography imaging process. Quality evaluation may include contrast-to-noise ratio and edge sharpness scores; thresholds are deployment specific.

[0043] In an embodiment, the method may further include: determining a 3D radiography model of the physical object based on the determined radiography images and providing the 3D radiography model to a deep neural network model that is trained to detect defects in the 3D radiography model and to classify and / or segment the detected defects.

[0044] In an embodiment, the deep neural network model may include a 3D convolutional neural network model, for example a 3D recurrent convolutional neural network R-CNN model,

[0045] In a further aspect, the embodiments may relate to a device for autonomous radiography inspection of a physical object, wherein the device may comprise; a first drone connected to a first radiography device; a 3D sensor for generating point cloud data; a position sensor for generating position data associated with the first drone; and, a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably an ASIC or FPGA, coupled to the computer readable storage medium responsive to executing the computer readable program code.

[0046] In an embodiment, the processor may be configured to perform executable operations comprising: controlling a first drone comprising a first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object; positioning the first drone at a target position associated with one of the one or more radiography waypoints, the positioning including: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position.

[0047] In an embodiment, the method may include executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source.

[0048] In an embodiment, each of the one or more waypoints is further associated with a target angular orientation in a 3D coordinate system, preferably the target angular orientation being represented by an alignment vector, and wherein the first radiography device is connected to the first drone based on a controllable pivoting structure, preferably a gimbal structure, the executable operations comprising: controlling the pivoting structure to align the first radiography device with the target angular orientation based on angular information, preferably an angular orientation and an angular rate, generated by a first inertial measurement unit (I MU) connected to the first drone and / or a second IMU connected to the first radiography device.

[0049] In an embodiment, the processor of the device may be configured to: (i) compute a pose transformation by registering a measured 3D view to the 3D model; (ii) operate an alignment controller using at least one IMU associated with the UAV body and at least one IMU associated with the gimbal to minimize the angular error (e.g. the LOS angular error) and stabilize gimbal rate; (iii) execute radiography exposure which is controlled based on the one or more error thresholds R, 0, and a dwell time T; (iv) time synchronize with a peer UAV over a direct link to issue a trigger for executing a radiography imaging process with a reduced jitter < J; and (v) provide pose constrained acquisition metadata for fusion and quality evaluation.

[0050] In an embodiment, the first drone (an UAV) may be operated based on a nested alignment control process using at least one IMU associated with the UAV body and at least one IMU associated with the gimbal, wherein an outer loop of the nested alignment control process is configured to minimize LOS angular error relative to the alignment vector and an inner loop of the nested alignment control process is configured to stabilize the gimbal using rate feedback. In an embodiment, radiography exposure may be permitted only if a registration error < R and a LOS angular error < 0 meet a predetermined condition (i.e. the registration error is smaller than the registration error threshold and the LOS angular error is smaller than the an angular error threshold) for a predetermined time, the so-called dwell time T. In an embodiment, the measured 3D view and the constructed 3D view may be point cloud models and a point cloud registration algorithm may be used to register the measured 3D view with the constructed 3D view.

[0051] In an embodiment, the mission map may further include information about structural features of the target 3D model.

[0052] In an embodiment, registering at least part of the measured 3D view with at least part of the constructed 3D view may further comprise: determining first features associated with the measured 3D view; determining second features associated with the constructed 3D view; and, registering at least part of the first structural features with at least part of the second structural features to compute the transformation matrix.

[0053] In and embodiment, registration of a measured 3D view of the object with a 3D model of the object (either pre-installed in the memory of the drone or dynamically constructed based on image data generated by the sensor data of the drone) may be achieved by registering features of a measured 3D view with features of the 3D model of the object. Point cloud registration is very accurate but very resource intensive, while registration based on prominent features is less accurate but much more efficient in terms of computational resources.

[0054] Hence, in an embodiment, a first registration process may be based on a point cloud registration to obtain an accurate initial position and orientation of the drone and to reduce accumulated errors. A second registration process may be executed based on features of the constructed and measured 3D view for at least a predetermined period and / or a predetermined number captured image data frames. This process (a point cloud registration followed by a number of feature-based registrations) may be repeated throughout the flight so obtain accurate positioning, while limiting the computational resource usage. This approach is specifically useful to optimize onboard resource usage and allows for utilization of 3D overlays in the perception view of the drone. This way it can approach target points defined in an overlay.

[0055] In yet a further aspect, the embodiments may relate to a system for autonomous radiography inspection of a physical object, the system comprising a first drone comprising a first radiography device and a second drone comprising a second radiography device, wherein each of the first and second radiography device may be configured to: navigate towards a physical object based on a mission map stored in a memory of the first and second drone respectively, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geolocation and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object; position itself at a target position associated with one of the one or more radiography waypoints, the positioning including: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters which are used to move to the target position; and, execute a radiography imaging process for capturing one or more radiography images of the ROI, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source.

[0056] In an embodiment, the system further comprising an auxiliary map-update sensing node configured to supply refreshed 3D views to a centralised mission director for model update, the auxiliary node being decoupled from the exposure process, omitting any X-ray generation or detection, and not required during exposure.

[0057] In an embodiment, the mission map may store, for each radiography exposure, one or more waypoints, wherein a waypoint may include a 3D alignment vector and one or more error thresholds, e.g. tolerances. The system may be configured to execute the radiography imaging process when constraints regarding position (or registration) and orientation are met. The system may fuse exposures using poses and alignment vectors as constraints while rejecting frames exceeding the one or more error thresholds (e.g. pose / quality thresholds). In an embodiment, pose estimation may employ a graduated robust estimator configured to return R, t and an inlier set, and the post-fit residual is used as a criterion for exposure.

[0058] The embodiments may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.

[0059] The embodiments will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments.

[0060] Brief description of the drawings

[0061] Fig. 1 depicts part of a radiography system according to an embodiment, according to an embodiment.

[0062] Fig. 2 depicts a general overview of a radiography system according to an embodiment;

[0063] Fig. 3A and 3B depict a positioning system for a radiography system according to an embodiment; Fig. 4 depicts illustrate a method of processing radiography images according to an embodiment;

[0064] Fig. 5 depicts a motion control and alignment system for a drone according to an embodiment

[0065] Fig. 6 depicts a method of radiography inspection of an object according to an embodiment.

[0066] Fig. 7 depicts a method of determining a mission map for a drone that is part of a radiography system according to an embodiment.

[0067] Fig. 8 illustrates an example of part of a mission map comprising a 3D point cloud model of an object a radiography waypoints according to an embodiment.

[0068] Fig. 9 depicts a method of a 3D radiography model using radiography images taken by a drone that is part of a radiography system according to an embodiment.

[0069] Fig. 10 is a block diagram illustrating an exemplary data processing system that may be used in as described in this disclosure.

[0070] Description of the embodiments

[0071] Fig. 1 depicts part of a radiography system according to an embodiment. In particular, the figure depicts a radiography system comprising a first drone 102i comprising an X-ray source 104 (a source drone) and a second drone 1022 comprising a radiography detector 106 (a detector drone). The source and the detector may form a radiography (X-ray) imaging system wherein the source may be configured to produce a directional X-ray beam to expose part of an object and wherein the detector may be configured to detect X-ray radiation that is transmitted through the exposed part of the object. The direction of the X-ray beam can be controlled by controlling the position of the drone and / or the position of the X-ray source relative to the drone. The source and detector may be connected to the drone via a 3-axis pivoting structure, e.g. a 3D gimbal, which can rotate over a yaw, pitch and roll axis. The pivoting structure may include actuators to control the 3D orientation of the source or detector relative to the orientation of the drone.

[0072] As shown in the figure, the source drone and the detector drone may be controlled to capture X-ray images of different parts of a physical object 110. These parts may be referred to as region-of-interests ROIs 120I,2. To that end, the source drone and the detector drone may be autonomously controlled to position themselves relative to the physical object so that the object is positioned between the source drone and the detector drone. Each drone may be configured to move around the object based on a certain predetermined trajectory 112I,2, i.e. a first trajectory 112i associated with the first drone comprising first waypoints 114I,2 and a second trajectory 1122 associated with the second drone comprising second waypoints 1161,2.

[0073] A waypoint on the first trajectory and a waypoint on the second trajectory (for example a first set of waypoints 114i and 1161) may define positions of the source drone and the detector drone at which X-ray images of part of the object may be taken. At these way points, the X-ray source of the source drone and the X-ray detector of the detector drone need to be positioned and aligned in 3D space relative to the object such that the X-ray beam generated by the X-ray source exposes a predetermined part, i.e. a region-of-interest ROI, of the object and exits the object in a direction where the detector drone is positioned. This way, the detector is exposed to the X-ray beam exiting the object so that one or more first X-ray images of a first ROI 120i of the object can be captured.

[0074] The positioning of the drones relative to the object may be based on an alignment vector 102i which defines a direction in the 3D space connecting the position of the X-ray source at waypoint 114i with the position of the X-ray detector at waypoint 1161. When the X-ray source and the X-ray detector are located at the waypoints and aligned based on a first alignment vector 1081 , the X-ray source may expose a first ROI 120i of the object with an X-ray beam and the detector drone may detect X-ray radiation that passes through the object. After, the images have been taken, each drone may move along its trajectory to a next waypoint (e.g. a second set of waypoints 1142 and 1162) to align the source and detector based on a further second alignment vector IO81 to capture one or more second X-ray images.

[0075] To enable accurate capturing of X-ray images of predetermined ROIs of the object, the trajectories and the waypoints of the drones relative to the object need to be computed accurately in real-time. Further, the positioning of the drones at the waypoints and the execution of the radiography imaging process, i.e. the X-ray source generating an X-ray beam for exposing a ROI of the object and the X-ray detector capturing an X-ray image of the exposed ROI, needs to be accurately synchronized. To that end, the drones may include a radio communication unit 1181,2 so that they can communicate with each other via a base station or directly.

[0076] Fig. 2 depicts a general overview of a radiography system according to an embodiment. The figure illustrates the source drone 202i and the detector drone 2022 which are communicatively connected via a wireless connection 228i,2to a base station 222. Each of the drones may include a processor 232I,2 , e.g. a GPU and / or CPU, a radio-interface 2181,2, a positioning module 234I,2, an alignment and motion compensation module 236I,2 and a memory in which a mission map 238i,2may be stored.

[0077] The positioning module 234I,2 of the drone may include a plurality of sensors, i.e. position sensors such as GPS and IMUs for continuously determining position data, i.e. position, velocity, acceleration and orientation of a done. The positioning module may further comprise perception sensors, such as visible light 3D camera sensors (e.g. a stereovision camera) and / or LiDAR camera sensors that can generate 3D images, e.g. point cloud scans, of part of an object that needs inspection.

[0078] In an embodiment, the positioning module may be configured to execute a localization process to accurately position and move the drone to a waypoint relative to the object based on the mission map and the data generated by the position sensors and the perception sensors. The positioning module may control the position of the drone based on a pre-built mission map that includes a 3D model of an object that needs inspection.

[0079] In another embodiment, the positioning module may be configured to execute a simultaneous localization and mapping (SLAM) process to accurately position and move the drone to a waypoint relative to the object based on the mission map and based on a 3D model of an object that is determined based on sensor data generated by the position sensors and the perception sensors. When moving relative to a target object, the drone may use the perception sensors to capture image data, typically point cloud data, of a 3D view of the target object and generate a 3D model of the physical object based on the generated 3D image data.

[0080] Hence, a drone may position itself relative to the object based on a 3D model that is either pre-built or determined while the drone is moving around the object. When approaching the object from a certain position it may determine a 3D view of the object using sensor data from the perception sensors. The 3D view of the object as perceived by the drone may be referred to as the measured 3D view of the object, i.e. a 3D model of the physical object “seen” from the perspective of the drone.

[0081] The alignment and motion compensation module 236I,2 of a drone may be configured to control pivoting structure 205i, 2 connecting the source or detector to the body of the drone. In particular, the alignment and motion compensation module may be configured to control actuators of a pivoting structure to control the 3D orientation of a radiography device, e.g. a radiography source or detector, relative to the orientation of the drone. The orientation of the drone body may be determined by sensors of the positioning module using a first IMU associated with the positioning module. The orientation of the source or detector may be determined by a second IMU connected to the radiography devices mounted to the drone. The system may leverage fusion of the sensor data generated by the drone IMU and sensor data generated by the IMU of the radiography device to provide effective motion compensation.

[0082] The LOS angular error is computed as the angle between the measured device boresight (from the gimbal IMU / encoders) and the alignment vector of the current waypoint pair. Pose constrained fusion uses the alignment vectors and registered poses as geometric constraints and rejects frames exceeding thresholds for R, LOS error, or pose covariance (Z) prior to Al analysis. Fusion uses the poses {Ti} and alignment vectors as constraints; frames with |LOS error| > Qmax, R > Rmax, or Z > Zmax are discarded.

[0083] The mission map 238I,2 may further comprise data that a drone needs for executing a radiography mission for inspecting a physical object located at a certain geolocation. The data may include a geographical map in which positions may be defined based on coordinates of a suitable geo-coordinate system, such as latitude, longitude and, optionally, altitude. The geographical map may further include one or more 3D models (either pre-installed or determined on the fly) of one or more physical objects that may be inspected. Each 3D model may be mapped into a set of geo-coordinates on the geographical map. The geo-coordinates of a 3D model, which may be referred to as a target model, may match the geo-coordinates of the associated real-world physical object. A suitable geographic information system (GIS) data model such as vector data model may be used to construct the geographical map that includes the target model and intended waypoints.

[0084] In an embodiment, a target model may be generated upfront (i.e. before the actual radiography imaging process) by one or more drones scanning a physical object using visible light and / or lidar cameras of the drones. This way, a target model may be formed based on the 3D image data, typically point cloud data. The target model may be formed by registering partly overlapping 3D images of parts of the physical object. Based on the target model, object features, e.g. edges, curvatures, surfaces, etc. of the object may be determined. In an embodiment, the object features may be determined using a trained deep learning model. In some embodiments, the target model may include information regarding material properties and / or mechanical properties of different parts of the object. In an embodiment, the mission map may include information about the object material, internal structure, etc. into consideration. In a further embodiment, the mission map may include a defect probability map identifying regions in which the probability of detects are high. These regions may be part of the regions of interest (ROIs) of an object.

[0085] The positioning and movement of the drone relative to the target object may be based on the data generated by the position and perception sensors of the drone. In particular, the positioning module 234I,2 of a drone may determine based on sensor data a measured 3D view of the physical object model. Then it may map (register) the measured 3D view to (part of) the 3D model stored in its memory. Based on the registration, it can determine it position relative to the target object and move itself relative to the target object. Here, the 3D model stored in the memory of the drone can be pre-installed or determined on the fly using the simultaneous localization and mapping (SLAM) process.

[0086] As will be described hereunder in greater detail, the positioning process may include the steps of: generating a measured 3D view, e.g. a point cloud model, of part of the physical object based on the 3D image data generated by the perception sensors of the drone; mapping the measured 3D view onto a part of the 3D model (the target model) to compute a transformation matrix; and, updating the position of the drone based on the transformation matrix. Here, the mapping of the measured 3D view onto the target model may include a registration step wherein the measured 3D view is registered onto the target model. The registration step provides information regarding the position and orientation of the drone relative to the target object.

[0087] This way, a first drone may use this information to autonomously navigate itself to a desired position and orientation relative to the physical object so that it can execute a radiography imaging process with a further second drone that positioned and oriented relative to the object, in alignment with the first drone.

[0088] The mission map may include trajectory information comprising trajectories as described above. The trajectories may include an initial trajectory to move the drone from a base station (which may be the start of the mission) towards an object. It further may include a radiography trajectory to move the drone in a predetermined way around the object to capture radiography images of ROIs. The trajectories may be defined based on absolute geo-coordinates of the mission map or local map frame coordinates. A radiography trajectory may include waypoints which define 3D (geo) positions for a drone relative to the target at which the drone may perform a specific task, e.g. capturing an X-ay image of part of the object. For each waypoint, the mission map may include settings for the X-ray imaging system, i.e. settings for the X-ray source to control the intensity (dose) and / or dimensions of the X-ray beam and settings for the X-ray detector, e.g. exposure time, current and voltage to control the imaging of the X-ray exposed part of the object.

[0089] A mission map may be configured according to the tasks a drone has to perform. For example, a mission map be a source mission map for a source drone executing a first trajectory around a physical object at a certain geo-position and a detector mission map for a detector drone for executing a second trajectory around the object as e.g. depicted in Fig. 1. To capture a radiography image of an ROI, the source of the source drone needs to be accurately aligned with the detector of the detector drone. To that end, pairs of waypoints on the first and second trajectories may include alignment information which the drones may be used to align the X-ray source with the X-ray detector. In an embodiment, the alignment information associated with a waypoint may include an alignment vector for aligning between the X-ray source and the X-ray detector of the drones. The alignment vectors may be used to position and align drones so that X-ray images of one or more ROIs can be captured.

[0090] The alignment and motion compensation module 234I,2 may be configured to control actuators to control the position and / or orientation of a drone during capturing of radiography images. In particular, it may control the pivoting structure 205I,2, e.g. a gimbal structure, of the drone connecting a radiography device (e.g. a radiography source or detector) to the body of the drone to control the 3D orientation of the radiography device relative to the orientation of the drone.

[0091] As will be explained hereunder in greater detail, in an embodiment, the actuators may be further controlled to dynamically respond to vibrations and disturbances, which are measured by the I Mils of the drone and the actuator during the radiographic imaging process. Real-time processing of sensor data ensures accurate detection of movements and real-time adjustments to detected movements to mitigate undesired movements, e.g. vibrations or drone movements due to the wind. The actuators may adjust the projection angles of the radiography device based on the computed alignment vectors associated with waypoints of trajectories. This way precise alignment of the drone with a further drone over designated ROIs for accurate radiography imaging can be ensured.

[0092] Each drone of the radiography inspection system may be configured to autonomously execute a radiography mission based on the mission map that includes a 3D model of the object and based on synchronized actions with other drones. These synchronized actions may be communicated over a direct wireless connection between drones or via the base control station. These synchronized actions may include: navigate through the waypoints following a calculated trajectory, adjust projection angles and radiography settings of the radiography devices (source and detector) and capture radiographic images of ROIs. In an embodiment, the quality of captured radiography images may be evaluated by the detector drone on the flight and if unacceptable, the detector drone may communicate with the source drone that execution of a further imaging process is needed. Quality evaluation may include signal-to-noise ratio, contrast-to-noise ratio and edge sharpness scores; thresholds are deployment-specific.

[0093] When starting a radiography mission, the drones may independently move towards initial waypoints as defined in the mission maps. Once arrived at initial waypoints, the drones may establish a communication connection to synchronize the imaging and the movement of the drones to the waypoints on the trajectories using registrations of 3D view with prebuilt 3D model or a dynamically built 3D model using the SLAM process.

[0094] Alternatively, it may use target waypoint overlays (as described below).

[0095] In an embodiment, drones may be configured to establish a fast direct wireless connection 230 between each other, which may be used to exchange information about the position and / or orientation of the drone and to synchronize the actions that are needed in order to capture X-ray images. This information may include exchange of information that the X-ray source and detector are correctly positioned and oriented and that they are configured according to the settings as indicated in the mission map. Further, the fast direct wireless connection may be used to trigger the X-ray source to expose a ROI with a certain X-ray dose for a certain exposure time and to trigger the X-ray detector to determine a radiography image by capturing X-ray radiation exiting the object during the exposure.

[0096] Furter, the drones may establish a data connection with the base station 122. This way, after the imaging process, a detector drone may transmit the radiography data to the base station for data processing, analysis, and storage and defect detection. In some embodiments, the base station may be connected to a server system 226 (e.g. a cloud platform). The base station and / or server or cloud system may comprise a suite of data processing algorithms to process the radiography data to perform defect detection, classification, segmentation and reporting. This way, these intensive computational tasks are offloaded from the drones' onboard processors to the ground base station.

[0097] The base station may be configured to run a central mission control system 240 for controlling tasks that are executed by drones and tasks executed by the mission control system itself. The division of the tasks are such that resource and battery power utilization is enhanced, so that the overall performance and scalability of the radiography system is optimized.

[0098] The radiography images of a ROI captured from different angles may be transmitted to the base station for reconstruction of a 3D radiographic representation 242, e.g. a voxelized 3D representation of the target object. The 3D radiographic representation may be generated by processing the radiography images captured by the drone using one or more machine learning models, e.g. one or more deep neural networks, which are trained for defect detection, classification, and segmentation. The model may be trained to identify defect types and evaluate the severity of the defects and affected regions. The results may be published as an electronic inspection report 244 that may be accessible through a dashboard or the like. The captured X-rays images of predetermined ROIs of the object may have a level of overlap so that they can be registered with the target model. This way, based on the captured X-ray images a 3D radiographic model 242 of the object, e.g. a turbine blade, can be determined.

[0099] Fig. 3A and 3B depict a positioning system for a drone according to an embodiment. Here, the drone may be part of a radiography system as described with reference to the embodiments in this disclosure. Fig. 3A depicts a diagram illustrating the workflow and components of a position system, referred to as a simultaneous localization and mapping (SLAM) system, which may be executed by a drone. The drone may use the positioning system to accurately position itself relative to an object that needs inspection based on a mission map that includes a 3D model that is determined using image data generated by the perception sensors of the drone and position information generated by the position sensors of the drone. As shown in Fig. 3A, a first part of the positioning system 302, the drone positioning system may include one or more schemes that are executed by the drone and a second part of the positioning system 304, the base positioning system, may include one or more schemes that are executed by the base station of the radiography system. The drone positioning system 304 may be configured to process data 306 generated by a plurality of sensors of the drone. The sensors may include perception sensors 308 and position sensors 310. The perception sensors may include 2D and / or 3D cameras that can capture visible and non-visible light (e.g. near IR) and lidar sensors and / or cameras for generating image data of the surroundings of the drone, including objects that need inspection. Here, the image data may include video frames generated by a conventional camera and point cloud data generated by a lidar camera.

[0100] The position sensors may include GPS and one or more inertial measurement units (IM Us) that are configured to continuously generate position and motion information, e.g. position coordinates and the velocity and acceleration as a function of the position. In some embodiments, the IMUs may also be configured to determine the orientation (pose) of the drone. An IMU may include one or more accelerometers and / or gyroscopes, which may be used as position sensors that can measure a position relative to a specific location. The IMU may be used for example when the GPS sensor does not have a satellite signal.

[0101] Based on the position information generated by the position sensors and the image data generated by perception sensors, the module may compute (e.g. construct) a 3D model 31 of the physical object. As will be explained hereunder in greater detail, the 3D model computed by the drone based on the generated sensor data may be used to determine the drone's position and orientation relative to the physical object. In particular, it may be used to refine the localization of the drone by computing predictions regarding position, movement, acceleration and orientation based on currently measured image data. Based on the image data generated by the perception sensors of the drone a second model, a 3D view of at least part of the object can be determined. This 3D view may be referred to as the measured 3D view, i.e. the 3D view of the object as perceived by the drone camera. The measured 3D view may be registered with the computed 3D model to accurately determine the position of the drone relative to the physical object.

[0102] Further, based on the generated image data of the object, a feature extraction module 314 may be configured to determine features. The feature detection may include detecting characteristic points, edges, surfaces, etc. of part of the imaged object. During the flight of the drone, the drone may continuously generate image data of the object it is approaching. Based on the image data, a model of the object (as perceived by the drone) is computed. The computed 3D model and associated features may then be used by a localization and mapping module 316 to determine an accurate position of the drone relative to the object In particular, the localization and mapping module may use sensor data and feature predictions to simultaneously update the drone's position and the map.

[0103] Resource expensive computations and tasks may be outsourced to the base localization system 308. To enable this, at least part of the data, which may be preprocessed by the drone, may be transmitted to the base station. For example, more complex tasks like closed-loop detection 318 which identifies so-called loop closures, wherein a drone returns to a previously mapped area. Loop detection helps in correcting drift and aligning the map accurately. Further, a global optimization module 320 may be configured to optimize the map globally based on the image data, ensuring consistency and accuracy of the mapped object and mapped area around the object. This optimization process may be executed using one or more Al algorithms. For example, a sequential deep learning model based on a Long Short Term Memory (LSTM) network or Transformer deep neural network architecture, which are capable of learning long-term dependencies in sequential data and which can be used to determine optimizations and adjustments based on sequences of image data as generated by the perception sensors of a drone. Thus, during a mission the mission map and / or a radiography trajectory may be updated in real-time based on the sensor data, e.g. the image data generated by the perceptual sensors, in response to unexpected interruptions, changes in environmental conditions or detected features of the physical object that are not present in the target model that is stored in the memory of the drone.

[0104] Fig. 3B provides a more detailed scheme of the localization and mapping module 316 of a drone according to an embodiment. As shown in this figure, it includes a localization module 322 and a mapping module 324. The localization module may be configured to determine a precise position and orientation of the drone relative to the object in real-time. The mapping module may construct and update the mission map, including the 3D model (the target model), using the data received from the localization module. The mapping module ensures dynamic updating the mission map and the 3D target object so that the mission map and the 3D target object reflect any changes in the object and / or its surroundings and corrects any inaccuracies.

[0105] As shown in the figure, the localization module may be configured to receive position data from the position sensors 310 for localization of the drone. The position data may be (pre)processed by a data processor 316 of the localization module. The processing of the position data may include filtering, noise reduction, and data synchronization before the data may be provided to the input of a position estimation module 318, which may include an algorithm for estimating the drone's current position and its orientation (pose). Based on the current position and its orientation and the 3D model associated with the mission map, the localization module may predict a 3D model 312 of object as perceived by the drone. In parallel to the processing of the position data and the determination of a 3D model of the physical object, the localization module may process image data received from perception sensors 308 of the drone, i.e. one or more cameras that generate 3D image data of the surroundings of the drone. Hence, during the flight of the drone, the perception sensors generate 3D image data of the surroundings and the physical object so that at least part of these image data represent a partial 3D model of the object as perceived by the drone. During the flight, the drone continuously determines measured 3D views of the physical object (based on the captured 3D image data) as truly perceived by cameras and registers these 3D views with the 3D model of the object. In an embodiment, the registration may include the registration of two point clouds. In another embodiment, the registration may include extracting structural features from the measured 3D views and using these structural features with features of the 3D model. Registration of these features may be used to obtain accurate position information about the position and orientation of the drone relative to the object.

[0106] A feature extraction algorithm 314 may be used to determine features from the image data representing the measured 3D view of the object. The feature detection may include detecting characteristic points, edges, surfaces, etc. of part of the imaged object. A matching module 322 then tries to register (match) the constructed 3D view with the measured 3D view based on detected features.

[0107] The registration of a 3D view with an 3D model of an object may be achieved in different ways. In an embodiment, a point cloud registration method may be used to register a point cloud representation of a 3D view of the physical object with a point cloud representation of the 3D model of the physical object. For example, in an embodiment, a first rough registration step of the measured 3D view of the object with the 3D model of the object may be performed using a registration method such as RANSAC to determine an initial transformation matrix. This initial transformation matrix may be used to perform a fine-tuned registration scheme using ICP or GICP method to map the coordinate system of the mission map frame into the real-world frame coordinate system.

[0108] Registration yields a pose transformation in a world-fixed frame; its residual is tracked and used as a criterion for exposure gating. To increase robustness on symmetric or repetitive structures, candidate correspondences may undergo geometric suppression, whereby spatially clustered or mutually ambiguous matches are down weighted or removed to preserve geometric dispersion (e.g., non-maximum suppression over local neighborhoods and minimum-distance separation of selected keypoints). This improves correspondence diversity, stabilizes the registration residual (R), and, consequently, the exposure gate. In an implementations, graph-theoretic pruning might be applied to the correspondence set. A correspondence consistency graph is constructed and a k-core subgraph (or similar densest subgraph) is retained, removing weakly supported outliers. The pruned set is then passed to the pose solver; the resulting reduction in outliers further stabilizes R for the exposure gate.

[0109] The transformation matrix may be used to determine the real-world coordinates of the waypoints in the mission map (either local north-east-down (NED) coordinates to approach the waypoints via a feedback control system as e.g. described with reference to Fig. 5. Alternatively, the local coordinate system may be transformed into global GPS coordinates (by having a reference point local coordinates and its respective GPS coordinate) and then either use the feedback control or RTK GPS positioning to navigate and position the drone on the waypoint based on the selected approach.

[0110] In a further embodiment, registration of a measured 3D view of the object with a 3D model of the object (either pre-installed in the memory of the drone or dynamically constructed based on image data generated by the sensor data of the drone) may be achieved by registering features of a measured 3D view with features of the 3D model of the object. Point cloud registration is very accurate but very resource intensive, while registration based on prominent features is less accurate but much more efficient in terms of computational resources. Hence, in an embodiment, a first registration process may be based on a point cloud registration to obtain an accurate initial position and orientation of the drone and to reduce accumulated errors. Thereafter, a second registration process may be executed based on features of the constructed and measured 3D view for at least a predetermined period and / or a predetermined number captured image data frames. This process (a point cloud registration followed by a number of feature-based registrations) may be repeated throughout the flight so obtain accurate positioning, while limiting the computational resource usage. This approach is specifically useful to optimize onboard resource usage and allows for utilization of 3D overlays in the perception view of the drone. This way it can approach target points defined in an overlay.

[0111] For feature-based registration, correspondences may be established based on keypoints. In particular, multi-resolution keypoint extraction and bidirectional match consistency with descriptor-space distance ratios may be used. To improve robustness regarding symmetric or repetitive structures, correspondences may be based on surface¬ normal compatibility, planar / edge saliency, and local geometric signatures, and are re¬ weighted by a bounded robust kernel. Temporal priors (e.g., predicted pose from the previous frame) restrict the search region. The resulting correspondence set is optimized to estimate the pose transformation. A post-fit residual error value (R) may be used for determining when exposure is possible (exposure gating) and quality control.

[0112] Pose estimation may employ a graduated robust estimator (e.g., graduated non-convex weighting) to jointly estimate R, t and an inlier set under high outlier ratios. The final post-fit residual (R) is recorded for exposure gating and fusion acceptance. In an embodiment, the controller may inhibits exposure unless (i) R < Rmax, (ii) | LOS error| < 0max, and (iii) these conditions persist for T. If any condition fails, alignment continues until the criteria are satisfied or a timeout occurs.

[0113] In an embodiment, RANSAC initialization with outlier rejection (e.g., point-to-plane residual thresholding) may be used to seed the ICP / GICP optimizer. The final mean / robust mean residual may be reported as R and included in the acquisition metadata.

[0114] To increase robustness on symmetric or repetitive structures, candidate correspondences may undergo geometric suppression, whereby spatially clustered or mutually ambiguous matches are down weighted or removed to preserve geometric dispersion (e.g., non-maximum suppression over local neighborhoods and minimum-distance separation of selected keypoints). This improves correspondence diversity, stabilizes the registration residual (R), and, consequently, the exposure gate.

[0115] Local 3D descriptors may include histogram-based features (e.g., FPFH-family descriptors) with efficient matching schemes to reduce runtime. The descriptor / matcher selection is implementation dependent and does not limit the method.

[0116] The result of the registration is a transformation matrix comprising spatial information (translation, rotation and / or scaling) for mapping a constructed 3D view of the object on a measured 3D view of the object. This information may be used to determine coordinates of waypoints which may be used by the drone to control the motors to maneuver the drone towards a desired position and / or into a desired orientation.

[0117] Successful matching confirms the position estimate, while mismatches may indicate errors or changes in the environment and new features detection. A data fusion module 326 may fuse multiple measured 3D views that have been successfully matched with a 3D model into a fused 3D model the object. This way, the 3D model that the drone uses for localization and navigation can be refined with respect to the position and orientation of the drone relative to the physical object. The fused 3D model of the object may be determined using processing techniques such as Kalman filtering and / or particle filtering to combine different sensor data.

[0118] Hence, the processing of the measured 3D view and the 3D model may be used to improve and correct the 3D model of the object that is associated with the mission map and that stored in the memory of the drone. During the matching process 322 one or more new features 334 may be detected. In that case, the new feature may be validated and added to the 3D object associated with the mission map. This step involves checking the significance and accuracy of the newly detected feature using the functions of the mapping module 330. Thus, newly detected features may be validated and imported into the current mission. This way, new features may be added to the mission map. Further, discrepancies between newly measured features and already stored features are corrected 328 to ensure that the mission map remains accurate and up-to-date. New observed features are integrated into the mission map, expanding its coverage and detail. Further, redundant and / or inaccurate features may be removed from the mission map 336 to maintain its quality and efficiency. This process prevents clutter and ensures the mission map's integrity. This continuous process of updating the mission map with new features allows the map to grow and adapt as the drone explores new areas.

[0119] Hence, when navigating towards an object, the SLAM system of the drone uses the sensors (location and perception sensors) and the mission map to determine 3D views which are registered with a 3D model of the object. The resulting transformation matrix is used by the drone to autonomously navigate itself towards a physical object and to accurately position and orient itself relative to the physical object. Further, the measured 3D views of the object may be used by the drone to update and improve the mission map, including the 3D model of the object and features associated with the 3D model.

[0120] The navigation of a drone towards the object may be divided in three modes of operation of the drone. In a first mode of operation, a drone is outside a first zone of a first perimeter around a target object. Outside of the first zone, the image data of the object generated by the visible camera sensors (3D or 2D camera sensors) are not suitable for reliable navigation. In that case, the image data generated by the perception sensors does not comprise sufficient information about the object. Hence, in the first mode of operation, the operating system of the drone is configured to navigate towards the target object based on the geo-coordinates, typically GPS coordinates, of the drone and the geo-coordinates of the target object. Based on the geo-coordinates, the distance between the drone and the target object can be estimated.

[0121] Upon reaching the first zone, the operating system of the drone may start a second phase of operation, wherein navigation of the drone may be based on processing image data (in particular visual image data of a 2D or 3D camera) of the target object. The operating system may include one or more machine learning algorithms for vision-based navigation of the drone. To that end, the operation system may include one or more deep neural networks, which are trained to detect (real-time) the target object in the image data and to determine features, e.g. features associated with the target object. Based on measured features, the drone may estimate its orientation and distance relative to the target object and controls its approach.

[0122] In an embodiment, a regression model may be used to determine the relationship between one or more independent variables that are used to navigate the drone. In particular, a real-time feature detection algorithm may be used to detect object features, e.g. features of a wind turbine, in one or more captured video frames and a deep learning regression model may be trained to determine a (relative) angle based on the position of features detected in the captured images. The (relative) angle defines an angle between the orientation of the drone and the orientation of the object as detected in the video frames of the drone camera. The result of the regression interference may be a value of the angle (in radians). A feedback control system may use the computed angle to adjust the angle of the drone's approach until it receives a further computed angle associated with one or more further captured video frames and corrects its approach based on this newly computed angle. Based on a GPS signal (with enough satellite coverage and RTK signal corrections), the drone may compute absolute GPS coordinates using the relative orientation computed by regression model and position itself accordingly.

[0123] Hence, in this mode of operation, the operating system of the drone may navigate the drone based on GPS information and visual image data towards a waypoint relative to the target object that is needed for executing a radiography imaging processes of a predetermined portion (a region of interest) of the target object. Real-time feature detection and a trained deep learning regression model is used in a feedback control system to assist a drone to move towards the target area.

[0124] When the drone is within the range of the lidar camera, the drone may switch to a third mode of operation in which navigation is based on image data, including 3D point cloud data of the target object. These images data may be used by the SLAM system (e.g. point cloud scan matching with a prebuilt 3D model or a 3D model that is built by the drone based on the perception sensors of the drone) in order for the drone to accurately position and orient itself relative to the object as e.g. described with reference to Fig. 3A and 3B. This 3D model is stored in the memory of the drone.

[0125] The positioning system for a drone as described with reference to Fig. 3 may be used to execute a radiography imaging process together with a further drone. Fig. 4 depicts a method for autonomous radiography inspection of a physical object according to an embodiment. In a first step 402 a first drone comprising a first radiography device navigates itself to towards a physical object based on a mission map stored in a memory of the first drone. The mission map may include a 3D target model, e.g. a point cloud model, of the physical object. Further, the 3D target model may be associated with a geo-location and one or more radiography waypoints, wherein each radiography waypoint may be associated with a target position relative to a region of interest (ROI) of the physical object.

[0126] The first drone may be positioned at a target position associated with a radiography waypoint (step 404), wherein the positioning may include: determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone; registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters which are used to move first drone to the target position. Then, the first drone may execute the radiography imaging process together with a second drone comprising a second radiography device (step 406), wherein the first radiography device is a radiography source and the second radiography device is a radiography detector or vice versa. In an embodiment, the execution of the radiography imaging process by the first and second radiography device may be synchronized via a communication channel between the first and second drone. In another embodiment, synchronization may be achieved via first radio communication channel between the first drone and a base station and a second radio communication channel between the second drone and a base station.

[0127] Fig. 5 depicts a motion control and alignment system used by the drone according to an embodiment. In particular, the figure depicts a control block diagram including a positioning control block 502 for controlling the positioning of the drone based on a waypoint 510 and an alignment control block 504 for controlling the alignment of a radiography device, e.g. e.g. an X-ray source or X-ray detector to an alignment angle 522 that is associated with the waypoint. As explained with reference to Fig. 1 , a waypoint may be associated with an alignment vector representing an orientation in the 3D space of the drone that is used by a drone to align its radiography device, e.g. a radiography source or detector, relative to a ROI of the object and a radiography device of a further drone.

[0128] The positioning control block may include a position controller 512, which manages the drone's position in the universal world fixed frame of a geocentric coordinate system (e.g. a cartesian spatial reference system that represents locations in the vicinity of the Earth). The position controller includes an input for receiving a waypoint position set point 510 (Xs,Ys,Zs) in the universal world fixed frame and for receiving measured positioning information (X,Y) as determined by the drone sensors and an output for outputting adjustments to the coordinates to achieve and maintain the desired position (XS,YS,ZS) as defined by the waypoint position set points.

[0129] A set of feedback controllers 514 may be used to continuously compute an error value between a desired set point (as defined by a waypoint) and a measured positioning variable. For example, the controllers may include an altitude controller to control the drone's vertical position (Z). The signal of the altitude controller may control the thrust to maintain or change altitude. The controllers may further include a roll controller to control the roll angle ( ) of the drone to stabilize the drone around the longitudinal axis providing lateral stability and balance, a pitch controller to control the pitch angle (0) of the drone to stabilize the drone around the lateral axis and a heading (yaw) controller to control the yaw angle (MJ) of the drone associated with the drone's directional heading. In an embodiment, each of the controllers may be implemented as an active proportional-integral-derivative (PID) controller. A speed controller 516 (mixer) may be configured to receive control signals from altitude, roll, pitch, and yaw controllers and to output motor speed commands (Qi, Q2, Q3, Q4) for the motors and propellers of the drone 518. A motors and propeller function 528 may be configured to receive speed controller commands and to generate thrust in accordance with the speed controller commands to move the drone towards the waypoint As described in detail with reference to Fig. 3, the drone sensors 506 may include position sensors and perceptional sensors to accurately determine position and orientation of the drone as a function of time. The position sensors may include a first drone IMU, which is configured to measure the drone's position, orientation and motion. In an embodiment, the drone IMU may also include a rate gyro to measure angular rates, i.e. change of the angular orientation as a function of time and / or an accelerometer to measure changes in the motion of the drone. For autonomy, positional and other set points may be issued by modules that execute the SLAM scheme depicted in Fig. 3 or a machine vision and pose estimation system that uses the perceptional sensors of the drone. This way, accurate information about the desired position (X,Y,Z), angular orientation (0, ,0) , and angular rates (0’,0’,0’) is continuously measured and used to control the motors and actuators of the drone.

[0130] As shown in the figure, the measured X,Y coordinates may be provided in a feedback loop to the position controller 51 to control the drone towards the desired position. Similarly, the measured altitude Z, the drone orientation (0, 0, 0) and angular rates may be provided in a feedback loop to the feedback controllers 514 to keep the drone at a desired altitude and orientation, while navigating towards the coordinates of the waypoint. The feedback loop of the positioning control block thus controls navigation of the drone towards a waypoint and positioning of the drone at a waypoint.

[0131] The position data of the drone sensors 506 (drone motion feed 520) are also provided as input data to the alignment control block, which controls the alignment and the orientation of the radiography device (radiography source or detector) as measured by the gimbal IMU with a desired orientation of the radiography device relative as defined by a predetermined alignment angle 522 associated with the waypoint in the mission map. Hence, in the alignment and stabilization control block the system manages alignment of the radiography device to a predetermined vector in the universal world fixed frame in the tracking loop and the motion compensation and stabilization of radiography device is carried out in the stabilization loop. This ensures that the radiography device, such as an x-ray source or detector, mounted on an actuator-controllable gimbals is correctly oriented and stabilized for the radiography task.

[0132] To achieve the alignment and stabilization, the control block includes two feedback loops, a tracking loop 540 and a stabilization loop 542. The tracking loop tracks the desired alignment angle for the gimbal based on the alignment angle 522 associated with the waypoint. The tracking feedback loop ensures that the radiography device mounted to the gimbal remains correctly oriented when it is positioned at a waypoint The stabilization loop 542 compensates for undesired changes in the orientation and / or motion of the drone due to inertia forces exerted onto the drone, e.g. motor torque and external influences such as the wind.

[0133] The tracking loop computes 540 an error, e.g. a line-of-sight (LOS) error, based on a desired alignment angle 522 (i.e. the angle associated with the alignment vector of the waypoint) and based on a measured angle 538, which defines the measured orientation of the radiography device that is mounted to gimbal. The orientation of the radiography device may for example be defined as the orientation of the normal vector of the detector plane of the radiography detector. The measured angle and motion may be determined based on sensor fusion of angular data and angular rate data generated by the first drone IMU that is part of the set of drone sensors 506 and angular and angular rate data generated by the second gimbal IMU. To that end, the gimbal IMU may include a rate gyro 530 for measuring the angular rate (change in orientation) of the radiography device as a function time. The gimbal IMU may also include an accelerometer which is configured to generate accelerometer data, which may be used to reduce or eliminate effects of drift. An angle controller 524 receives the (LOS) error and outputs an angular rate to minimize the error so that the gimbal is aligned correctly with the desired vector.

[0134] Changes (fluctuations) in the orientation of the radiography device may be caused by inertial forces 508 that are exerted onto the drone. The stabilization loop 542 is configured to compensate these changes based on the measured angular rate generated by the rate gyro 530. To that end, angular rate data measured by the rate gyro of the gimbal IMU may be combined with angular rate data measured by the rate gyro of the drone IMU using a sensor fusion function 532. This function merges data from the drone and gimbal IMU to provide more accurate and stable angle and angular rate data.

[0135] The angular rate data as determined by the sensor fusion function may be combined with the angular rate set point as computed by the angle controller 524 to determine an angular rate error. A rate controller 526 is configured to generate motor voltages that control an actuator motor to adjust the orientation of the gimbal. This way, discrepancies between the desired angular rates and the measured angular rates (4>’, 0’, ’+’’) caused by inertia forces exerted on the drone during the flight can be compensated resulting in an extremely stable orientation of the radiography device at the waypoint.

[0136] The angular rate data determined by the sensor fusion function is also used for the tracking feedback loop. The angular rate data may be combined with accelerometer data generated by an accelerometer of the gimbal IMU using accelerometer data function 534. A kinematic integration module 536 may subsequently integrate the measured angular rate data into a measured angle 538 which is needed to maintain the alignment angle, wherein the accelerometer data are used to mitigate effects of drift.

[0137] Hence, the embodiments provide autonomous radiography methods and systems, which, in addition to accurate positioning, provide a stable projection geometry at the moment of exposure. The system combines registration-grade pose estimation with an alignment control process that generates an error value. The radiography imaging process may be executed based on the error value. For example, the radiography imaging process may be executed when the error value is below a predetermined threshold.

[0138] In an embodiment, the alignment control process may include a dual IMU nested alignment control, wherein the radiography imaging process may be executed when an error associated with the registration process and an error associated with a registration residual threshold and a line-of-sight (LOS) angular error threshold are met and held for a dwell time, and trigger exposures with deterministic inter UAV jitter sufficiently small to maintain projection consistency, thereby yielding radiographs suitable for pose constrained 3D fusion with quality based rejection. In some embodiments, auxiliary sensing platforms as nodes in specific formations and with coordinated mission, can refresh the model.

[0139] After positioning of the drone and orientation of the radiographic device mounted to the drone, the radiographic imaging process of a ROI by a source drone and an associated detector drone can be started. Fig. 6 depicts a method of radiography inspection of an object according to an embodiment. As shown in the figure, the method may start with loading a mission map into the memory of a first drone (step 602). The mission map may comprise a 3D target model of a physical object that needs inspection. The 3D target model includes radiography waypoints associated with regions of interests (ROIs) of the physical object and alignment vectors for aligning a first radiography device with a ROI of the object wherein the first radiography device is pivotable mounted to the drone.

[0140] The drone may position itself at a waypoint based on the mission map (step 604). The positioning process based on the mission map may include determining a measured 3D view of the object based on image data, in particular 3D point cloud data, generated by perception sensors of the drone. Then, a registering algorithm may be used to register (map) the measured 3D view with the 3D model to determine a transformation matrix. Based on the (inverse) transformation matrix real world coordinates of the waypoint may be determined. Then, the drone may position itself at the waypoint based on the determined coordinates.

[0141] During the positioning at the waypoint, then angular data angular rate data generated by a drone IMU and a gimbal IMU may be used to control the gimbal to align the first radiography device with an alignment vector (step 608) associated with the waypoint. Once the first drone is in position, it may inform a second drone that is positioned at a second waypoint that is associated with the first waypoint to start execution of a radiography imaging process. The first and second drone a second radiography device of the second drone is aligned with the ROI and with the first radiography device of the first drone

[0142] In case of a disturbance, the first drone and the second drone may agree to repeat the positioning and orienting at the waypoint and to execute the radiography imaging process (step 610). After the execution of the radiography imaging process, the detector drone may execute an initial image validation process to accept or reject the captured image. In case the captured image is rejected, the drones may agree to repeat the positioning and orienting at the waypoint and execution the radiography imaging process (step 612).

[0143] If the image is accepted, the captured image may be tagged with position information and timestamp data and send to the basis system or the cloud for further processing (step 614). Thereafter, the process of positioning and orienting the drone and executing the radiography imaging process may be executed for a further waypoint associated with a further ROI.

[0144] Fig. 7 depicts a method of determining a mission map for a drone that is part of a radiography system according to an embodiment. As shown in the figure, first a 3D model, preferably a 3D point cloud model, of an object to be inspected may be determined. This model may be generated by a drone scanning the object using a point cloud sensor, e.g. a lidar sensor or a lidar camera (step 700). The 3D model may be associated with a geolocation on a geographical map. Further, regions of interests (ROIs) on the object may be determined that need inspection by radiography imaging. A region of object may be determined based on information about the mechanical and material structure of the object and a defect probability map of the object (step 702). For each ROI radiography waypoints may be computed (step 704) wherein each waypoint may be associated with a position relative to the object and an orientation (an alignment vector) for a radiography device mounted on the drone. For each radiography waypoint radiography settings for the radiography device may be computed based on the information about the mechanical and material structure of the object at the ROI that is associated with the waypoint (step 706). Then, an optimal navigation trajectory for the drone may be computed based on the radiography waypoints (step 708). Finally, a mission map be determined including the geo¬ located 3D point cloud model, the navigation trajectory comprising the radiography waypoints and an alignment vector and radiography settings for each waypoint.

[0145] Hence, the radiography mission map is built upon a 3D point cloud reconstruction of the object, e.g. a wind turbine including the blades. A map generator is configured to receive parameters such as internal structure, material, defect occurrence probability map and determines the regions of interest (ROI), i.e. the areas of the object that require inspection. It then computes the locations, i.e. the waypoints, for the source and detector drones, and X-ray settings (kVp, mA, exposure time) per radiography waypoint. A path planner then may dynamically compute the most optimum path to traverse between waypoints to conduct the radiography mission. The centralized mission director takes the waypoints and computes control points defining the validated navigation path and send the control commands travers the path between the waypoints for each drone.

[0146] Fig. 8 illustrates an example of part of a mission map comprising a 3D point cloud model of an object including radiography waypoints according to an embodiment. Alignment vectors 804i, 2 associated with a ROI 806i,2Of an object are visualized in the figure. The alignment vectors are 3D vectors are used to align the source and detectors of the drones so that each vector passes through the X-ray source and detector device. The alignment vectors are defined in the mission map so the alignment and compensator mechanism as described with reference to Fig. 5 can align the source and the detector radiography devices and supress the vibration in a collaborative approach between two drones so that radiography images of acceptable quality can be determined. As shown in the figure, many different alignment vectors may be computed which are used for imaging many different ROIs of the object. These different radiography images taken at different waypoints and associated with different alignment vectors may be combined and registered to form a 3D radiography model of the object.

[0147] Fig. 9 depicts a method of forming a 3D radiography model using radiography images taken by a drone that is part of a radiography system according to the embodiments described in this application. As shown in the figure, the method may include the detector drone sending validated and geo-location tagged radiography images for a plurality of overlapping ROIs of an object via a base station to a server system (step 902), which may comprise one or more server applications to process the radiography images.

[0148] The radiography images of the overlapping ROIs of the object may be preprocessing to remove noise and enhance defect visibility and the pre-processed radiography images may be registered and combined (fused) to form a 3D radiography model of at least part of the object (steps 904 and 906).

[0149] The 3D radiography model may be provided to the input of a neural network, which is trained to detect defects and to classify and segment detected defects (step 908). In an embodiment, the neural network may be a mask R-CNN based neural network that is trained for detecting and classifying defects. A further neural network, preferably a deep neural network having a U-Net based encoder-decoder architecture, may be trained to refine the candidate regions passed from Mask R-CNN to accurately localize and segment the defect regions and to improve the Mask R-CNN segmentation (step 910). In some embodiment, a deep convolutional generative adversarial network (DCGAN), StyleGANv2 or other network may be used to simulate defects that are used as data augmentation for training CNNs to improve the generalization of models and reduce overfitting.

[0150] The detected defects and classification and segmentation information associated with the detected defects are stored in database (step 912). Further, the criticality of the defects may be determined and a report maybe generated including the classified detected defects (step 914).

[0151] Fig. 10 is a block diagram illustrating an exemplary data processing system that may be used in as described in this disclosure. Data processing system 1000 may include at least one processor 1002 coupled to memory elements 1004 through a system bus 1006. As such, the data processing system may store program code within memory elements 1004. Further, processor 1002 may execute the program code accessed from memory elements 1004 via system bus 1006. In one aspect, data processing system may be implemented as a computer that is suitable for storing and / or executing program code. It should be appreciated, however, that data processing system 1000 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.

[0152] Memory elements 1004 may include one or more physical memory devices such as, for example, local memory 1008 and one or more bulk storage devices 1010. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 1010 during execution.

[0153] Input / output (I / O) devices depicted as input device 1012 and output device 1014 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and / or output device may be coupled to data processing system either directly or through intervening I / O controllers. A network adapter 1016 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and / or networks to said data and a data transmitter for transmitting data to said systems, devices and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1000. As pictured in Fig. 10, memory elements 1004 may store an application 1018. It should be appreciated that data processing system 1000 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 1000, e.g., by processor 1002. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.

[0154] In one aspect, for example, data processing system 1000 may represent a client data processing system. In that case, application 1018 may represent a client application that, when executed, configures data processing system 1000 to perform the various functions described herein with reference to a "client". Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like.

[0155] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0156] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0157] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. CLAIMS1. A computer-implemented method for autonomous radiography inspection of a physical object, the method comprising:3.controlling a first drone comprising a first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the first radiography device being connected to the first drone using a controllable pivoting structure, preferably a gimbal structure, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object and a target angular orientation, preferably a 3D alignment vector, for aligning the radiography device with the ROI;4.positioning the first drone at a target position associated with one of the one or more radiography waypoints; and,5.executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source;6.wherein the positioning of the first drone includes:7.- determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone;8.- registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position;9.- controlling the pivoting structure to align an orientation of the first radiography device with the target angular orientation based on angular information generated by a first inertial measurement unit (I MU) connected to the first drone and / or a second IMU connected to the first radiography device.

2. Method according to claim 1, wherein the execution of the radiography imaging process by the first and second radiography device is synchronized via a direct radio communication channel between the first drone and second drone or wherein the execution is synchronized via first radio communication channel between the first drone and a base station and a second radio communication channel between the second drone and the base station.

3. Method according to claim 1 or 2 wherein the 3D alignment vector defines a line-of-sight (LOS) through the ROI, and wherein each radiography waypoint further includes a registration error threshold associated with the position (registration) of the first drone and an angular error threshold associated with the orientation of the radiography device.

4. Method according to any of claims 1-3 wherein the measured 3D view and the 3D target model are point cloud models and wherein a point cloud registration algorithm is used to register to compute the transformation matrix.

5. Method according to any of claims 1-4 wherein the mission map further includes information about structural features of the target 3D model and wherein registering at least part of the measured 3D view with at least part of the 3D model further comprises:13.determining first structural features associated with the measured 3D view; determining second structural features associated with the 3D target model; registering at least part of the first structural features with at least part of the second structural features to compute the transformation matrix.

6. Method according to any of claims 1-5 wherein15.the execution of the radiography imaging process is started if both the first radiography device and the second radiography device is aligned with the target angular orientation, the execution of the radiography imaging process including determining one or more radiography images of the ROI; and / or,16.wherein navigating the first drone further includes:17.if the physical object is outside the visual range of a visual sensor, e.g. a camera, of the first drone, navigating the first drone towards the physical object based a geolocation of the first inspection drone and the geo-location of the physical object, a GPS module of the first inspection drone determining the geo-location of the first drone.

7. Method according to any of claims 1-6 wherein navigating the first drone further includes:19.if the physical object is within the visual range of a visual sensor, e.g. a camera, of the first drone, navigating the first drone towards the physical object based image processing of video frames generated by a camera of the first drone, the image processing including object detection of the physical object in the video frames and, optionally, pose estimation of the first drone relative to the physical object.

8. Method according to any of claims 1-7 wherein the one or more waypoints further include radiography settings for configuring the first radiography device to capture the one or more radiography images when the second radiography device of the second drone exposes the ROI to radiation.

9. Method according to any of claims 1-8 wherein the method further includes: evaluating the quality of the one or more radiography images of the ROI and if the quality of the one or more radiography images is below a threshold value, repeating the execution of the radiography imaging process to determine one or more further radiography images.

10. Method according to any of claims 1-9 wherein the method further includes:22.determining a 3D radiography model of the physical object based on the determined radiography images, preferably the determining including fusing the radiography images into a 3D radiography model using poses derived from the registration transforms and the alignment vectors as geometric constraints, wherein, optionally, radiography images exceeding thresholds for LOS angular error, registration error residual, and / or pose covariance are rejected, and providing the 3D radiography model to a deep neural network model, preferably a 3D convolutional neural network model, for example a 3D region-based convolutional neural network (R-CNN) based model, that is trained to detect defects in the 3D radiography model and to classify and / or segment the detected defects and / or to infer the criticality of the defected region in order to adjust maintenance priorities associated with the target object.

11. A device for autonomous radiography inspection of a physical object, the device comprising;24.a first drone connected to a first radiography device;25.3D perception sensor for generating point cloud data;26.a position sensor for generating position data associated with the first drone; a computer readable storage medium having at least part of a program embodied therewith; and a processor, preferably an ASIC or FPGA, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising:27.controlling the first drone comprising the first radiography device to navigate towards a physical object based on a mission map stored in a memory of the first drone, the first radiography device being connected to the first drone using a controllable pivoting structure, preferably a gimbal structure, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object and a target angular orientation, preferably a 3D alignment vector, for aligning the radiography device with the ROI;28.positioning the first drone at a target position associated with one of the one or more radiography waypoints; and,29.executing by the first drone, a radiography imaging process for capturing one or more radiography images of the ROI together with a second drone comprising a second radiography device, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source;30.wherein the positioning of the first drone includes:31.- determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone;32.- registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position;33.- controlling the pivoting structure to align the first radiography device with the target angular orientation based on angular information generated by a first inertial measurement unit (I MU) connected to the first drone and / or a second IMU connected to the first radiography device.

12. Device according to claim 11 wherein the 3D alignment vector defines a line-of-sight (LOS) through the ROI, and wherein each radiography waypoint further includes a positional error threshold associated with a position of the first drone and an angular error threshold associated with the orientation of the radiography device.

13. Device according to claims 11 or 12 wherein36.wherein the measured 3D view and the 3D target model are point cloud models and wherein a point cloud registration algorithm is used to register the compute the transformation matrix.; and / or,37.wherein the mission map further includes information about structural features of the target 3D model and wherein registering at least part of the measured 3D view with at least part of the 3D model further comprises:38.determining first structural features associated with the measured 3D view; determining second structural features associated with the constructed 3D view;39.registering at least part of the first structural features with at least part of the second structural features to compute the transformation matrix; optionally, the featurebased registration employing at least one of: multi-resolution keypoints, bidirectional consistency, descriptor-ratio gating, surface-normal compatibility, robust-kernel re-weighting, and histogram based local descriptors with efficient approximate matching.

14. A system for autonomous radiography inspection of a physical object, the system comprising a first drone comprising a first radiography device and a second drone comprising a second radiography device, wherein each of the first and second radiography device is configured to:41.navigate towards a physical object based on a mission map stored in a memory of the first and second drone respectively, the first radiography device being connected to the first drone using a controllable pivoting structure, preferably a gimbal structure, the mission map including a 3D target model, preferably a point cloud model, of the physical object, the 3D target model being associated with a geo-location and one or more radiography waypoints, each radiography waypoint being associated with a target position relative to a region of interest (ROI) of the physical object and a target angular orientation, preferably a 3D alignment vector, for aligning the radiography device with the ROI;42.position itself at a target position associated with one of the one or more radiography waypoints; and,43.execute a radiography imaging process for capturing one or more radiography images of the ROI, wherein the first radiography device is a radiography detector and the second radiography device is a radiography source; or, wherein the second radiography device is a radiography detector and the first radiography device is a radiography source;44.wherein the positioning of the first drone includes:45.- determining a measured 3D view of the physical object based on point cloud images generated by a 3D sensor of the first drone;46.- registering at least part of the measured 3D view with at least part of the 3D target model to determine transformation parameters for moving the first drone to the target position;47.- controlling the pivoting structure to align the first radiography device with the target angular orientation based on angular information generated by a first inertial measurement unit (I MU) connected to the first drone and / or a second IMU connected to the first radiography device.

15. Computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of claims 1-10.

Images