Fiducial object and method and system for obtaining sensor data and position

Abstract

The invention relates to fiducial object for determining the mutual position of a camera system on the one hand and the fiducial object on the other hand, by processing images taken using the camera system. The fiducial object comprises protrusions at its edges and visual markers. The protrusions and visual markers have different colours. A method is also described which uses a fiducial object to create a map of sensor data of a part using an inspection device, for inspection devices with a relatively small range as compared to the size of the part. Further, a sensor package is described usable in this method. In both cases, an IMU is optionally integrated with a camera system and a processing system to allow accurate and fast positioning.

Description

[0001] FIDUCIAL OBJECT AND METHOD AND SYSTEM FOR OBTAINING SENSOR DATA AND POSITION

[0002] The invention relates to a fiducial object for determining the mutual position of a camera system on the one hand and the fiducial object on the other hand by processing images taken of the fiducial object using the camera system. The invention further relates to a method of, and a set for, obtaining sensor data relating to a part using an inspection device having a range that corresponds to only a portion of the part.

[0003] Fiducial objects as such are known. As an example, reference is made to US 9,448,758 B2 which uses 2D markers (e.g. stickers) presenting a visible and recognisable pattern on a 3D object, to facilitate orientation with respect to the 3D object. It is well documented that these 2D markers are usable, but they have limitations. As an example, 2D markers can only be used accurately from a limited set of viewing angles around the normal position.

[0004] An improvement has been described for instance in Fiducial objects: Custom Design and Evaluation by Garcia-Ruiz et al. in Sensors 2023, 23, 9649 (DOI: 10.3390 / s23249649). Garcia- Ruiz et al. propose amongst others a dodecahedron shaped fiducial object with 2D patterns on each face. Similarly, a dodecahedron shaped fiducial object is used in DodecaPen: Accurate 6DoF Tracking of a Passive Stylus by Wu et al. in UIST 2017, October 22-25, 2017, Quebec City, QC, Canada (DOI: 10.1145 / 3126594.3126664). A further improvement was presented in ChromaTag: A Colored Marker and Fast Detection Algorithm by DeGol et al. in 2017 IEEE ICCV, pp. 1481- 1490, (DOI: 10.1109 / ICCV.2017.164). DeGol et al. introduce a fiducial marker and detection algorithm designed to use opponent colours to limit and quickly reject initial false detections and greyscale for precise localisation.

[0005] While decidedly more usable in complex applications than more traditional 2D marker stickers also known in the art, the fiducial objects presented in these papers are not yet usable for applications that are demanding, for instance in terms of accuracy and / or speed.

[0006] One such application is the creation of maps of sensor data by combining local sensor data into a map based on the position where the local sensor data was taken. Of course, the accuracy of the resulting map depends on the accuracy of knowledge about the position of the local sensor data. This application arises for instance in inspection of parts (including objects, systems, devices), which are larger than the range of the inspection device used. In such a case, sensor data taken from multiple mutual positions of the part and the inspection device needs to be accurately combined into a map. Maps like these are also useable for instance in quality assurance.

[0007] The same application also brings about a high demand in speed, especially for the calculation of the position. Preferably, the resulting map can be built up and seen while the inspection is taking place. Finally, a relatively small marker, especially with small details, may prove less effective for inspection of relatively large parts, because an image taken of the marker may not include sufficient detail for accurate position determination.

[0008] The invention has as its object to at least in part alleviate at least some of these problems.

[0009] The object is achieved using a fiducial object according to the independent device claim.

[0010] As recited in the independent device claim, the fiducial object comprises a body of a three- dimensional shape, the body having an outer surface presenting a plurality of faces and a plurality of edges along which said faces meet, wherein a series of protrusions is arranged on the edges.

[0011] The three-dimensional shape of the body allows the fiducial object to be seen from different positions around the fiducial object. Accordingly, a single fiducial object can be used for positioning from a relatively large number of locations. The body having faces and edges along which said faces meet may aid in accurate positioning, since the edges may be detectable relatively easily.

[0012] The series of protrusions being arranged on the edges allows detection of the protrusions from different viewpoints around the fiducial object. The protrusions may allow a relatively quick assessment of where approximately in an image the fiducial object is, as they may be detectable relatively easily. From the protrusions, it is also possible to obtain a first estimate of the mutual position between fiducial object and camera system. From this estimate, the 3D fiducial object can be matched to a 2D representation and vice versa.

[0013] As recited also in the independent device claim, each of the faces includes a two- dimensional pattern acting as a visual marker. The pattern is preferably a visible pattern.

[0014] The two-dimensional pattern may aid in detection and identification, and may especially be detected with a relatively high accuracy, depending on the type of marker used.

[0015] The protrusions and the visual markers together thus allow a two-step approach which is both fast due to the fast detection of the protrusions by their colour and relatively accurate due to the visual markers. The presence of the protrusions allows efficient use of the 2D markers on a 3D fiducial.

[0016] Finally, the visual markers are of one or more first colours, and the protrusions are of one or more second colours, wherein the first colours and second colours are mutually exclusive. That is, the second colours differ from the first colours.

[0017] The use of different colours for markers and protrusions allows to distinguish between the two relatively easily. This facilitates not only the detection of both the protrusions and visual markers per se, but further facilitates the two-step approach described above.

[0018] It is possible the edges meet at vertices. Placing the protrusions at vertices may be particularly advantageous, because they may be more visible at vertices. In principle it is sufficient to place protrusions at some but not all vertices, although the best detection may be obtained if all vertices are provided with a protrusion. The faces may be planar, i.e., flat. The edges may be sharp. The outer surface may be comprised of faces entirely. Preferably, the body is a convex shape.

[0019] The camera system may comprise one or more cameras. The one or more cameras may be directional cameras, wide-angle cameras, or both. A wide-angle camera may be a camera provided with a wide-angle lens, whereas a directional camera may be a camera with a regular, i.e. non- wide-angle, lens.

[0020] Preferably, the protrusions are spherical. Spherical protrusions look the same from each viewpoint around the protrusions, thereby facilitating detection.

[0021] The protrusions may be or may all be of the same size, i.e. they may be equisized. Accordingly, using the apparent size of the protrusions in an image, the distance between the camera system and the fiducial object can be determined. This is especially easy when the protrusions are spheres, because in that case the apparent size in an image does not change depending on the perspective of the camera system. The apparent size is reflected in the image by the silhouette or shape / outline of the protrusion.

[0022] The edges may be of one or more third colours, wherein the third colours are mutually exclusive with the first colours and with the second colours.

[0023] Using yet another (set of) colour(s) for the edges allows detecting the edges more easily. Accordingly, the detection process may be sped up even more and / or become more accurate.

[0024] It is advantageous if all similar elements of the fiducial object are of the same colour or colours, that is to say, any of the following:

[0025] - the protrusions are identical to each other in spectral appearance; and / or

[0026] - the edges are identical to each other in spectral appearance; and / or

[0027] - the patterns are identical to each other in spectral appearance.

[0028] When these elements are identical in spectral appearance, such as chromatic appearance, they may be detected without having to make an estimate of the viewpoint of the camera system with respect to the fiducial object. That is to say, the colours perceived at the camera system are not changed simply by rotation of the fiducial object.

[0029] It is generally practical if each of the elements has the least number of possible colours, for instance:

[0030] - each of the protrusions has a single colour; and / or

[0031] - each of the edges has a single colour; and / or

[0032] - each of the patterns has exactly two colours.

[0033] In this way, colours can be chosen for each of the elements that differ strongly in appearance from each other. As an example, contrastive colours may be chosen for the protrusions, edges and patterns, optionally each being of a different primary colour, optionally one of the additive primary colours red, green and blue.

[0034] By selecting contrastive colours, it may be possible to distinguish between the features corresponding to these elements of the fiducial object in an image taken using the camera system even under bad lighting conditions. Additionally, or alternatively, the amount of processing needed to distinguish may be reduced, or the distinguishing may be more accurately performed.

[0035] It is noted that the pattern in general requires the use of at least two colours to allow the pattern to be created. It is proposed to use a background colour and a foreground colour contrastive with the colours of the protrusions and the edges for the patterns. Preferably, the background and foreground colour are relatively distant in the colour space, to allow easily distinguishing them visually. The background colour preferably is a tint of grey, preferably a relatively light or relatively dark tint of grey, i.e. white and / or black. This allows interpretation of the pattern relatively easily, as these colours can be distinguished relatively easily from the foreground colour, and colours contrastive therewith.

[0036] It is principally possible that the surface of the fiducial object comprises parts which are not covered by a visual marker, are not an edge, and are not covered by a protrusion. These parts may be of a or the background colour, such as white, black, or anther tint of grey.

[0037] At this time, it is noted that the colours, although preferably in the visible spectrum, are not limited thereto. In fact, it is very well possible to use a multi-spectral camera system that captures, e.g., UV and / or IR radiation. The normal concepts of colour and chromatic appearance can in that case be extended over the UV and / or IR range, making it possible to pick colours or appearances (that is, multi-spectral colours) that are more contrastive because the spectrum employed is larger. Therefore, colour as used in this application also extends to the same concept in the UV and / or IR range. Said appearance may thus also be a spectral or electromagnetic appearance and not be limited to appearance in terms of (visible) colour. Nevertheless, it is considered optional in some cases to use only the visible spectrum. This may offer several advantages, one thereof being that an operator can distinguish relatively easily between the colours used. Moreover, some camera systems use chips sensitive specifically to primary colours in the visible range, so that maximal use can be made of the camera’s sensitivity by selecting corresponding colours.

[0038] It is advantageous if each pattern is different from the others, so that each visual marker is unique.

[0039] When each visual marker is unique, identification of a marker visible in an image taken of the fiducial object allows determining the orientation of the fiducial object with respect to the camera system. The visual marker may additionally be rotationally non-symmetric, so that a rotation of the visual marker around the view axis of the camera can also be registered.

[0040] In particular, the visual markers may be multi-scale markers. Multi-scale markers allow detection from relatively far away and from relatively close by due to a particular arrangement of the features in the marker. In the relevant field, multi-scale markers are a well-defined concept, and has for instance been described in Fourier tags: Smoothly degradable fiducial markers for use in human-robot interaction by Sattar et al. in CRV ’07 (DOI: 10.1109 / CRV.2007.34). As such, the skilled person has no trouble selecting one or more multi-scale markers.

[0041] At this time, it is noted that the fiducial object itself may further facilitate recognition from far away and from up close, because the protrusions may be distinguishable even from far away.

[0042] Preferably, the body has a polyhedron shape, preferably a convex polyhedron shape. Such a shape presents faces visible from many different orientations, so that markers arranged on these faces can be used for determining the position of a camera system with respect to the fiducial object.

[0043] A particularly well-suited shape for the body is that of any isohedral polyhedron shape, wherein the faces are regular polygons. Even more in particular, the body may be of dodecahedron shape. This allows use of pentagonal markers, generation of which is possible for instance as described in Object Localization with Multiplanar Fiducial Markers: Accurate Pose Estimation by Garcia-Ruiz et al. in ibPRIA 2023, pp 454-465.

[0044] According to a further embodiment of the fiducial object, at least one, preferably each, of the faces may further include an object-specific, preferably unique, identification marker for identifying the fiducial object. That is, the face includes the identification marker in addition to the pattern acting as a visual marker. This facilitates the use of multiple fiducial objects as described further on, as these can then be distinguished from each other on the basis of the respective identification markers. Although different fiducial objects could also be distinguished from each other on the basis of being provided with different patterns acting as visual markers, the additional identification markers enhance the accuracy, speed and / or efficiency of the algorithm or algorithms used as described further on.

[0045] The identification marker is preferably a visible marker. This may offer several advantages, one thereof being that an operator can distinguish relatively easily between the respective identification markers of different fiducial objects. Moreover, some camera systems use chips sensitive specifically to colours in the visible range, so that maximal use can be made of the camera’s sensitivity. The invention also relates to a method of obtaining sensor data, such as non-destructive testing (NDT) data and / or non-destructive inspection (NDI) data, relating to a part using an inspection device having a range that corresponds to only a portion of the part.

[0046] Such a method is in itself known, for instance from GB 2605989 A. This document describes using a non-destructive evaluation (NDE) probe attached to a stereo camera. The stereo camera is used to perceive the environment in order to perform simultaneous localisation and mapping (SLAM).

[0047] Whilst the principles described in GB 2605989 A may be employed effectively, the system is not sufficiently accurate.

[0048] The invention aims to at least partially address this problem. Accordingly, a method is provided according to the independent method claim.

[0049] This claim describes:

[0050] - moving the inspection device with respect to the part between a plurality of mutual positions, thereby bringing different portions of the part into the range of the inspection device for each mutual position;

[0051] - determining the mutual position of the part and the inspection device for each of the plurality of mutual positions;

[0052] - capturing sensor data at each of the plurality of mutual positions, thereby creating sensor data of each of the different portions of the part; and

[0053] - optionally, creating a map of composite sensor data by composing said sensor data based on the determined mutual position corresponding to each of the different portions to which the sensor data relates.

[0054] By moving the inspection device, it becomes possible to inspect the part from different positions and / or orientations. Accordingly, a larger part may be inspected completely, or a larger portion of a relatively large part may be inspected with an inspection device that would normally have a limited range. As an example, an ultrasonic sensor may be used as an inspection device. Ultrasonic sensors can be used to derive information about the material under test locally, by applying ultrasonic vibrations and measuring the response. Such a sensor however has a range, meaning the area within which inspection can take place, approximately equal to the size of the sensor itself. To inspect a relatively large part, the inspection device has to be moved over the part.

[0055] It is known that if the mutual position of the part and the inspection device is determined, it becomes possible to create a map by composing sensor data from different positions.

[0056] According to the invention, the method comprises:

[0057] - keeping one of a) a camera system and b) one or more fiducial objects stationary with respect to the part, whilst keeping the other of a) the camera system and b) the one or more fiducial objects stationary with respect to the inspection device; - taking one or more images, or a stream of images, of the one or more fiducial objects using the camera system; and

[0058] - by a processing system, processing the one or more images of the one or more fiducial objects to determine a mutual position of the camera system and the one or more fiducial objects; and

[0059] - determining the mutual position of the inspection device and the part based on the mutual position of the camera system and the one or more fiducial objects.

[0060] Using a camera system and a fiducial object, a mutual position of these two can be determined. By keeping them stationary with respect to the part and the inspection device respectively, it becomes possible to infer also the mutual position of the inspection device and the part. The other way around is also possible. To illustrate, when keeping the camera system stationary with respect to the inspection device and the fiducial object stationary with respect to the part, the mutual position of the inspection device and the part can also be inferred.

[0061] It is noted that by using a fiducial object the accuracy of the determined mutual position may be increased, and / or the determination may become more efficient.

[0062] It is of course possible to use the fiducial object as described hereabove, with any one or more above-described features, in the method as described herein, but this is not strictly necessary. An advantage can already be achieved over, e.g., the disclosure of GB 2605989 A if a fiducial object is used instead of the environment. After all, the environment may be dynamic and / or may have features that are hard to identify in images taken by the cameras of GB 2605989 A.

[0063] It is possible, not only specifically in the method as herein described but also more in general relating to the fiducial object as described hereabove, to provide the fiducial object on a stand. For applications where the part is relatively large and stationary, the fiducial object can be placed in the vicinity of the part to allow inspection. The same fiducial object can later be used to inspect another part by displacing the fiducial object. Using a stand it has proven possible to enlarge the inspection area. For instance, the inspection area can be enlarged to about 4 m around the fiducial object. Sensor data can be placed with an accuracy of 2 mm or even less within this range. In specific conditions, even an accuracy of about 0.2 mm can be achieved within this range. For even bigger parts, multiple fiducial objects can be used.

[0064] It should be noted that the accuracy achieved is influenced by several factors, such as the resolution of the camera system, the size of the fiducial, etc. As an example, increased resolution will increase the accuracy (e.g., from 3 mm to 2 mm).

[0065] It is also possible to attach the fiducial object to the part.

[0066] In the opposite case, the fiducial object can be attached to the inspection device. The camera system in that case could be stationary with respect to the part, for instance by keeping both the part and the camera system in a stand still. It is for instance possible to fix the camera system in a frame with respect to the outside word. Multiple parts can then be presented, for instance within or after an assembly or production line, in an area seen by the camera system. In this case, during inspection, the parts are kept stationary, but they need not necessarily be lined up the same way for each part. The inspection device, having the fiducial object attached thereto, can be seen using the camera system so that its position with respect to the part can be determined.

[0067] In a variation of the method described herein, it is possible to determine the mutual position of the inspection device and the part if both of these are stationary with its own camera system and / or fiducial object. As an example, both may be provided with fiducial objects, and a camera system may be provided externally (i.e. fixed in a frame). Alternatively, a fiducial object may be fixed in a frame or on a stand, a part may be provided with a fiducial object, and the inspection device can be provided with a camera system. It is thus, in principle, possible to account for movement of the part with respect to a reference point by tracking the movement of the part, similar to how movement of the inspection device could be tracked.

[0068] It is advantageous if the processing system and the camera system are integrated. Accordingly, a compact sensor package can be obtained, that is easily attached to, e.g., an inspection device or easily installed in an inspection system.

[0069] In particular, the processing system may operate as the inspection is taking place, i.e., the map may be created live, possibly real-time or with a delay of less than one second.

[0070] The method may further comprise taking inertial measurements during said moving of the inspection device with respect to the part, and wherein determining the mutual position of the camera system and the one or more fiducial objects is based further on the inertial measurements.

[0071] Depending on which of the part and the inspection device is / are being moved, inertial measurements may be taken of that movement. The inertial measurements may allow a more accurate determination of position than via visual positioning only. The corresponding technique is called visual-inertial-odometry (VIO).

[0072] Practically, the inertial measurements are taken by an inertial measurement unit (IMU), the IMU optionally being integrated with the processing system and the camera system.

[0073] By integrating the IMU with the processing system and the camera system, a self- contained sensor package is obtained, that can be used with existing inspection devices relatively easily.

[0074] The method may further comprise retrieving a digital model of each of the one or more fiducial objects, wherein determining the mutual position of the camera system and the one or more fiducial objects comprises comparing one or more images with each digital model.

[0075] Using the retrieved digital model, a comparison can be made to quickly and accurately determine the mutual position of the camera system with respect to the fiducial object. For this purpose, the digital model may comprise information on the visual markers on the fiducial object and their mutual placement, as well as the position and size of the protrusions. Information about the colour of edges, and / or protrusions and / or visual markers may also be present in the digital model.

[0076] Practically, the processing system further comprises a memory having stored therein each digital model, and wherein the retrieving of each digital model comprises retrieving each digital model from the memory.

[0077] If the memory is part of the processing system, the method can be performed relatively self-contained, e.g. no connection to a remote server is needed. This greatly increases the versatility of the method because parts to be inspected are not always mobile and not always in locations where such connections can be made easily.

[0078] As an example, an image taken using the camera system may be matched with the digital model by recognising features in the image, and comparing these to features in the model. Of course, other strategies may be used to achieve the same. Algorithms for matching an image with a digital model are known but in many applications do not meet demands in terms of, e.g., speed or efficiency. It has been found that using the fiducial object as described herein in combination with a suitable algorithm increases the accuracy, speed and / or efficiency to an unforeseen extent. Following the strategy used, the orientation of the camera system and the fiducial object is known. Based on size information which may be present in the digital model, it is possible to determine the distance between the camera system and the fiducial object. The size information may relate to the fiducial object as a whole, or parts thereof, in particular of the protrusions. To converge to an optimal distance calculation, an estimate for the distance may be made iteratively. A loss function can then be used to optimise.

[0079] The invention also relates to a set for obtaining sensor data relating to a part using an inspection device having a range that corresponds to only a portion of the part, the set comprising:

[0080] - one or more fiducial objects; and

[0081] - a sensor package, the sensor package including a camera system and a processing system, wherein the camera system is configured for taking one or more images, or a stream of images, of the one or more fiducial objects; wherein the processing system is configured for receiving the one or more images; wherein the processing system is further configured to, repeatedly:

[0082] - process the one or more images of the one or more fiducial objects to determine a mutual position of the camera system and the one or more fiducial objects. The set can facilitate creating a map of composite sensor data relating to the part and may be used for example in a method as described hereabove. If the mutual position of the fiducial object and the camera system is known, a mutual position of an inspection device and a part may be derived.

[0083] At this point, it is noted that using the technology described herein, the mutual position can be derived, including the orientation. In that regard, the position can be defined as a position along all three lateral dimensions and an angular position along three rotational axes, all dimensions and axes being defined mutually perpendicularly.

[0084] While it is known to determine the position of a camera system with respect to a fiducial object per se, this is usually done off-line and remote, e.g. on a desktop computer away from an inspection site. This is however unsuitable for many inspection applications. Accordingly, by providing the processing system in the sensor package with the camera system, it becomes possible to do online processing, e.g. live or real-time or with a less than one second delay. This enhances usability of the sensor package.

[0085] The sensor package may further comprise an inertial measurement unit (IMU) configured for taking inertial measurements, wherein the processing system is further configured to the IMU for receiving the inertial measurements, and to base the determining of the mutual position on the inertial measurements.

[0086] The determined mutual position may be more accurate if inertial measurements are accounted for. In particular, it becomes possible to increase the frequency of determining positions as IMUs generally have higher update frequencies than cameras, which allows virtually updating a position between images taken at a lower frequency using the camera system. In addition, the IMU can increase the accuracy in case an image has motion blur.

[0087] Advantageously, the camera system comprises one or more directional cameras, and / or one or more wide-angle cameras.

[0088] Using wide-angle cameras, it is possible to expand the area within which the fiducial object can be seen, and thus in which inspection is possible. By using a fiducial object, the influence of aberrations and distortions associated with a wide-angle camera can be reduced or minimised.

[0089] However, for smaller parts or particular applications, it is possible to use directional cameras, thereby avoiding the influence of such aberrations and distortions altogether. This makes it possible to increase the accuracy of position determination. Using multiple cameras, it is possible to still obtain a relatively large field of view for the camera system.

[0090] It is noted that multiple fiducial objects may be part of the set to further expand the working area. If the mutual position of fiducial objects is known, only one fiducial object needs to be in view to accurately perform position determination. The application of multiple fiducial objects is thus advantageous in the method as described herein and / or in the set as described herein.

[0091] In any one or both cases, it is possible to predetermine the mutual positions of the fiducial objects, for instance using a calibration procedure. If multiple cameras are used, the mutual positions of the cameras may also be predetermined, e.g. via a calibration procedure. The predetermined positions (of fiducial objects and / or cameras) can be stored, for instance in the memory, and retrieved at will to be used as basis for the position determination. Further information, for instance about the fixed position of a fiducial object and / or camera with respect to the inspection device or the part respectively, may also be used to base determination of the mutual position on.

[0092] It may further be advantageous to perform a calibration procedure for each camera in the camera system, to account for the lenses. This becomes more important when wide-angle lenses are used.

[0093] Finally, it is noted the fiducial described herein may be used in the method or set described herein, with any one or more optional features as desired. The set may be used in the method, with any one or more optional features as desired.

[0094] It is further noted that instead of a single fiducial object, multiple fiducial objects can be used. For instance, in the method described herein, multiple fiducial objects can be used to enlarge the area within which inspection can be performed.

[0095] It is however also possible to use two or more fiducial objects that are fixed together in a predefined positional and rotational relation. These fiducial objects form together a fiducial constellation. The predefined relation between the fiducial objects can be used to infer even more accurate positioning information, e.g. via triangulation or trilateration (or multi-angulation or multi-lateration in general).

[0096] The invention will be further elucidated with reference to the figures, wherein: Figure 1 schematically shows a fiducial object;

[0097] Figure 2 schematically shows an inspection device with a sensor package;

[0098] Figure 3 schematically shows a part and the inspection device and the fiducial object; Figure 4 schematically illustrates a use of the fiducial object.

[0099] Throughout the figures, like elements are referred to using like reference numerals.

[0100] Figure 1 shows a fiducial object 1. The fiducial object 1 can be used for determining the mutual position of a camera system on the one hand and the fiducial object 1 on the other hand. To this end, images are taken by the camera system of the fiducial object 1, and these are processed. For this purpose, the fiducial object 1 is provided with several features that make processing the images relatively easy and / or fast, and / or enable more accurate detection. Firstly, the fiducial object 1 has a body 2, which in this case is shaped as a dodecahedron. Of course, other shapes could be used, such as optionally convex polyhedrons. In any case, the body 2 has an outer surface defining faces 3. The faces 3 meet at edges 4. For the shape depicted here, the edges meet at vertices, but this is not required. Protrusions 5 are provided at the vertices, but the protrusions 5 may be placed anywhere along the edges 4. In this particular example, the protrusions 5 are spherical in shape. This is however not required. It is possible, as is shown here, that all protrusions are of identical size. The faces 3 are provided with two-dimensional patterns which are visible, thereby creating a visual marker 6. The markers 6 in this example are provided to the faces 3 as stickers 9, however any other way of creating visible patterns is possible. Each pattern includes a foreground colour 7 and a background colour 8, which are spread in a manner so as to create the pattern. The manner in which the colours are spread is unique for each face 3, so that each face 3 can be identified by its marker 6. Here, the patterns are formed as a set of polygons. Of course, it is possible to use different kinds of visual markers 6. As an example, while the markers shown are essentially a composition of triangles, other types of markers 6 could be used. The markers 6 in this case are multi-scale markers 6 because, from up close, the position and specific constellation of a first set of small triangles 61 of the marker 6 can be detected with great accuracy whereas, at a larger distance, the position and specific constellation of a second set of larger polygons 62 of the marker 6 can be detected with great accuracy. This way, the marker 6 can be detected with great accuracy from different distances. While efficient for the dodecahedron body 1, it is also not required the markers 6 are of pentagonal shape as is shown in figure 1. Although likely not visible in the print of the present patent (application) document, the protrusions 5 of figure 1 are all the same colour blue. The foreground colour 7 of the markers 6 is red, while the background colour 8 is white. Finally, the edges 4 are green. Thus, the additive primary colours have been used, but it is possible to use other primary colours, divided across the elements of the fiducial object 1 differently. It is also possible multiple colours are used for each element, as long as the colours used for different elements do not match.

[0101] Figure 2 shows schematically a sensor package, and an inspection device 10. The inspection device 10 in this case is an ultrasonic sensor which has a base 11 and a head 12. The ultrasonic sensor works by providing ultrasonic vibrations 16 and registering their echoes. For the application of the technology described herein, it is however not important what type of inspection device 10 is used. The ultrasonic sensor is mentioned merely as an example. The sensor package includes a camera system with cameras 13 facing in different directions, a processing system 14 and an IMU 15, the latter being optional. The camera system of course can comprise more cameras 13, or only one camera 13, of the direction or wide-angle type as desired. The camera system 13, processing system 14 and the IMU 15 in this case are all integrated together by them being integrated with the inspection device 10. It is also possible to provide the sensor package 13, 14, 15 as a unitary component, the parts of which are integrated with each other, that can be attached and detached from inspection devices.

[0102] In a highly schematical setup shown in figure 3, a part 20 is shown ready for inspection. The part 20 in this case is a part of a wing, but other types of parts could be inspected. The fiducial object 1 is shown on a stand 21, so that it is stationary with respect to the part 20 which is also stationary. The inspection device 10 can be moved over the entire part 20. Since the cameras 13 on the inspection device can see the fiducial object 1, they can determine their position with respect to the part 20, to create a map including sensor data taken at different locations at the part 20.

[0103] In figure 4, the system of cameras 13 is shown in four different positions with respect to a part 20 to be inspected, to illustrate the movement of the camera system relative to the part 20. At each position, the camera system can see both the part 20 and the fiducial object 1 either in the field of view of a single camera 13 of the system or in the fields of view of different cameras 13 of the system. In figure 4, the fields of view are represented by respective rectangular frames. As the fiducial object 1 is stationary with respect to the part 20, their mutual position is known throughout the movement of the camera system. For instance, the mutual position of the fiducial object 1 and the part 20 may have been determined in an initialisation process before the start of the inspection. During movement of the camera system, the relative position of the camera system with respect to the part 20 and the relative position of the camera system with respect to the fiducial object 1 change, and a stream of images of the fiducial object 1 is taken using the camera system. Based on the image stream, the mutual position of the camera system and the fiducial object 1 can be determined for the various positions of the camera system. Based on both the mutual position of the fiducial object 1 and the part 20 and the mutual position of the camera system and the fiducial object 1, the mutual position of the part 20 and the camera system, and therefore the mutual position of the part 20 and the inspection device 10, can be determined. In figure 4, the respective mutual positions are illustrated as dashed lines. Based on the mutual positions of the part 20 and the moving inspection device 10, a map of composite sensor data can be created by composing the sensor data from the inspection device 10 at the various positions corresponding to the various portions of the part 20.

[0104] Although only one fiducial 1 is shown, inspection of bigger parts can be performed accurately using multiple fiducials 1 at different locations around and / or on the part.

[0105] Of course, it is also possible to apply the fiducial object to the inspection device, and to apply the sensor package stationary with respect to the part 20. It should be noted that the above-mentioned embodiments illustrate rather than limit the present invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word “comprising” does not exclude the presence of elements or steps not listed in a claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In claims enumerating several means, several of these means can be embodied by one and the same item of hardware. The usage of the words “first”, “second”, “third”, etc. does not indicate any ordering or priority. These words are to be interpreted as names used for convenience.

Claims

CLAIMS1. Fiducial object for determining the mutual position of a camera system on the one hand and the fiducial object on the other hand, by processing images taken of the fiducial object using the camera system, wherein the fiducial object comprises a body of a three-dimensional shape, the body having an outer surface presenting a plurality of faces and a plurality of edges along which said faces meet, wherein a series of protrusions is arranged on the edges, wherein each of the faces includes a two-dimensional pattern acting as a visual marker, wherein the visual markers are of one or more first colours, and the protrusions are of one or more second colours, wherein the first colours and second colours are mutually exclusive.

2. Fiducial object according to the previous claim, wherein the protrusions are spherical.

3. Fiducial object according to any of the preceding claims, wherein the protrusions are equi sized.

4. Fiducial object according to any of the preceding claims, wherein the edges are of one or more third colours, wherein the third colours are mutually exclusive with the first colours and with the second colours.

5. Fiducial object according to any of the preceding claims, wherein:- the protrusions are identical to each other in spectral appearance; and / or- the edges are identical to each other in spectral appearance; and / or- the patterns are identical to each other in spectral appearance.

6. Fiducial object according to any of the preceding claims, wherein:- each of the protrusions has a single colour; and / or- each of the edges has a single colour; and / or- each of the patterns has exactly two colours.

7. Fiducial object according to any of the preceding claims, wherein each pattern is different from the others, so that each visual marker is unique.

8. Fiducial object according to any of the preceding claims, wherein the visual markers are multi-scale markers.

9. Fiducial object according to any of the preceding claims, wherein the protrusions, edges and visual markers are of contrastive colour, optionally each being of a different primary colour, optionally one of the additive primary colours red, green and blue.

10. Fiducial object according to any of the preceding claims, wherein the body has a polyhedron shape, preferably a convex polyhedron shape.

11. Fiducial object according to the previous claim, wherein the body has an isohedral polyhedron shape, wherein the faces are regular polygons, preferably wherein the body is of dodecahedron shape.

12. Fiducial object according to any of the preceding claims, wherein at least one, preferably each, of the faces further includes an object-specific identification marker for identifying the fiducial object.

13. Fiducial constellation, including at least two fiducial objects according to any of the preceding claims, the fiducial objects being fixed to each other in a predefined positional and rotational relation.

14. Method of obtaining sensor data relating to a part using an inspection device having a range that corresponds to only a portion of the part, the method comprising:- moving the inspection device with respect to the part between a plurality of mutual positions, thereby bringing different portions of the part into the range of the inspection device for each mutual position;- determining the mutual position of the part and the inspection device for each of the plurality of mutual positions; and- capturing sensor data at each of the plurality of mutual positions, thereby creating sensor data of each of the different portions of the part, wherein determining the mutual position comprises:- keeping one of a) a camera system and b) one or more fiducial objects stationary with respect to the part, whilst keeping the other of a) the camera system and b) the one or more fiducial objects stationary with respect to the inspection device;- taking at least one image of the one or more fiducial objects using the camera system; and- by a processing system, processing the at least one image of the one or more fiducial objects to determine a mutual position of the camera system and the one or more fiducial objects; and- determining the mutual position of the inspection device and the part based on the mutual position of the camera system and the one or more fiducial objects.

15. Method according to the previous claim, wherein the processing system and the camera system are integrated.

16. Method according to any of the preceding method claims, wherein the method further comprises taking inertial measurements during said moving of the inspection device with respect to the part, and wherein determining the mutual position of the camera system and the one or more fiducial objects is based further on the inertial measurements.

17. Method according to the previous claim, wherein the inertial measurements are taken by an inertial measurement unit (IMU), the IMU optionally being integrated with the processing system and the camera system.

18. Method according to any of the preceding method claims, the method further comprising retrieving a digital model of each of the one or more fiducial objects, and wherein determining the mutual position of the camera system and the one or more fiducial objects comprises comparing one or more images with each digital model.

19. Method according to the previous claim, wherein the processing system further comprises a memory having stored therein each digital model, and wherein the retrieving of each digital model comprises retrieving each digital model from the memory.

20. Method according to any of the preceding method claims, wherein multiple fiducial objects are used, which are optionally fixed to each other in a predefined positional and rotational relation to form a fiducial constellation.

21. Method according to any of the preceding method claims, further comprising:- creating a map of composite sensor data by composing said sensor data based on the determined mutual position corresponding to each of the different portions to which the sensor data relates, wherein the range of the inspection device corresponds to only a portion of the map.

22. Method according to any of the preceding method claims, wherein the sensor data is nondestructive testing (NDT) data and / or non-destructive inspection (NDI) data.

23. Set for obtaining sensor data relating to a part using an inspection device having a range that corresponds to only a portion of the part, the set comprising:- one or more fiducial objects; and- a sensor package, the sensor package including a camera system and a processing system, wherein the camera system is configured for taking at least one image of the one or more fiducial objects; wherein the processing system is configured for receiving the at least one image; wherein the processing system is further configured to, repeatedly:- process the at least one image of the one or more fiducial objects to determine a mutual position of the camera system and the one or more fiducial objects.

24. Set according to the previous claim, wherein the sensor package further comprises an inertial measurement unit (IMU) configured for taking inertial measurements, and wherein the processing system is further configured to the IMU for receiving the inertial measurements, and to base the determining of the mutual position on the inertial measurements.

25. Set according to any of the preceding set claims, wherein the camera system comprises one or more directional cameras, and / or one or more wide-angle cameras.

26. Set according to any of the preceding set claims, wherein the one or more fiducial objects are comprised by one or more fiducial constellations, the fiducial objects of each constellation being fixed to each other in a predefined positional and rotational relation.

Images