Fiducial marker detection in spherical images

Abstract

A method is provided performed by a computing device for fiducial marker detection in a plurality of spherical images. The method includes sampling (410) the plurality of spherical images including at least one fiducial marker into a plurality of sets of perspective images. The method further includes detecting (420) the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker. The method further includes registering (430) the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

Description

FIDUCIAL MARKER DETECTION IN SPHERICAL IMAGESTECHNICAL FIELD

[0001] The present disclosure relates to detecting fiducial marker detection in spherical images, and related methods and devices.BACKGROUND

[0002] Accurate three dimensional (3D) scene geometry may be captured with 3D reconstruction approaches, for example based on the Structure-from-Motion (SfM) concept. Such approaches may allow depth in the scene to be estimated from an unstructured set of two dimensional (2D) images. Performing the scan with a 360 degree camera and visualizing the reconstructed geometry in a SkyBox -rendered 3D scene may be of particular interest.

[0003] The SfM pipeline may include physical placement and detection of fiducial markers, such as an AprilTag or ArUco, as part of the image processing during the early stage of the reconstruction process. Fiducial markers may be put in the scene by either attaching them to the walls or placing them on the ground, for example. Later, when those fiducial markers are detected during the reconstruction, they can be utilized in a number of ways. For example, since the physical dimensions of the fiducial markers are known beforehand, the physical dimension information can be used to bring a global scale to the reconstructed scene to perform absolute measurements within the scene, as is done in a Skybox environment for example. In another example, fiducial markers can serve as fixed identified locations within the scene. Those locations can be detected and automatically processed depending on the exact needs (e.g., automated global positioning system (GPS) location labeling).SUMMARY

[0004] There currently exist certain challenges. Fiducial marker detection in spherical images may be a difficult task. For example, existing detection methods may be tuned on perspective images.

[0005] Moreover, in some approaches for some fiducial tags (e.g., AprilTags), the best detection quality may occur when the camera directly faces, or is manipulated to have the camera viewing direction center on, the fiducial marker. Such manipulations, however, may not be possible when a 360°camera is used, as the camera is often placed on a tripod and cannot be tilted to make the camera lenses face the fiducial marker(s); and / or tilting the camera to face afiducial marker(s) on the ground / floor / ceiling can decrease image quality in an area(s) of interest (e.g., on walls and not on the ground / floor / ceiling).

[0006] Additionally, the number of pixels per unit of space in 360°camera images, for example taken by Ricoh Theta or Insta360, is much lower compared to regular small field of view (FoV) images taken by, e.g., a modem phone camera. Generally, for example, the detection distance may be low in spherical images, such as down to about 10 meters with a fiducial marker on the wall, and about 2.5 meters if fiducial markers are scattered on the ground in a scene.

[0007] In an approach for detection of fiducial markers in panoramic images, a homography may be used to warp a fiducial marker. Such an approach may rectify the fiducial marker image to a conventional perspective on which detection can be performed. However, this approach requires knowledge of the outer four comers of the fiducial marker to compute the homography; and those points typically may not be readily available in most real-world scenarios.

[0008] Another approach may use a circular fiducial marker. However, the circular marker may need to be tailored to a camera type, which reduces generalization of the approach.

[0009] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Examples of the present disclosure include a method for detecting fiducial markers in spherical images. The method includes sampling of spherical (e.g., equirectangular) images into multiple perspective images. Such sampling may increase the detection of fiducial markers (e.g., fiducial markers placed on the ground / floor).

[0010] Some embodiments disclosed herein are directed to a method performed by a computing device for fiducial marker detection in a plurality of spherical images. The method includes sampling the plurality of spherical images including at least one fiducial marker into a plurality of sets of perspective images. The method further includes detecting the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and registering the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

[0011] Some other embodiments are directed to a computing device for fiducial marker detection in a plurality of spherical images. The computing device includes processing circuitry; and at least one memory storing instructions executable by the processing circuitry to perform operations to sample the plurality of spherical images including at least one fiducial marker into a plurality of sets of perspective images. The operations further include to detect the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and to register the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

[0012] Some other embodiments are directed to a computer program product including a non-transitory computer readable medium storing instructions executable by processing circuitry of a computing device for fiducial marker detection in a plurality of spherical images. The instructions executed by the processing circuitry perform operations including to sample the plurality of spherical images including at least one fiducial marker into a plurality of sets of perspective images. The operations further include to detect the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and to register the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

[0013] Certain embodiments may provide one or more of the following technical advantage(s). As a consequence of the sampling and detecting, any fiducial marker (e.g., circular, square, on a ceiling / floor / ground, between images, etc.) may be detected in perspective images, with reduced overhead. Moreover, the detection rate and the detection distance of fiducial markers in spherical images taken by a 360°camera may be increased, while keeping the runtime of the detection method sufficiently low.

[0014] Other methods, computing devices, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional computing devices, methods, and computer program products be included within this description and protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings:

[0016] Figure 1 is a flow chart of operations for 3D reconstruction from 360° images according to some embodiments;

[0017] Figure 2A is a schematic drawings showing an example of directions of rotation of yaw and pitch according to some embodiments;

[0018] Figures 2B and 2C correspond to Figure 2A and are schematic drawings of an example of a reprojection according to some embodiments;

[0019] Figure 3A is a schematic drawing showing central axes of cubemap directions for a perspective projection;

[0020] Figure 3B is a schematic drawing showing an example of sampling directions of rotation according to some embodiments;

[0021] Figure 4 is a flow chart of operations performed by a computing device according to some embodiments;

[0022] Figure 5 is a block diagram of a computing device according to some embodiments; and

[0023] Figure 6 is a block diagram of a cloud-based computing device communicatively connected to a 360° camera according to some embodiments.DETAILED DESCRIPTION

[0024] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of the present disclosure are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0025] Figure 1 is a flow chart of operations for 3D reconstruction from 360° images according to some embodiments. The process of Figure 1 and various examples herein are discussed in the context of 3D scenes captured by a 360° camera and to present to a user device a virtual tour (e.g., SkyBox-type environment) of a digitized scene with correct dimensions. Such a process may allow a user / user device to navigate in the scene, perform accurate measurement(s), and insert labels associated with the 3D pointcloud. As discussed further herein, the process of Figure 1 includes obtaining a specific set of perspective projections that may increase the detection rate of a fiducial marker(s) in the spherical images, considering the expected relative locations of the fiducial marker(s) in relation to the camera locations (e.g., a fiducial marker(s) located on the walls and on the ground).

[0026] Scene capturing is performed in block 110. First, a scene may be prepared by a technician by placing a number of fiducial markers, such as AprilTags or ArUco fiducial markers, within the scene. Placement locations can be chosen according to the specific use-case for the fiducial markers. Non-limiting examples of locations may include textureless walls / floor to make visual connections or distinctions between areas; arbitrary locations in unoccupied areas if the fiducial markers are used to determine the scale of the scene; or some specific locations of interest, (e.g., to overlay extended reality (XR) assets). In some examples, the fiducial markersare primarily placed on the ground / floor or on a wall not higher than about 2m above the ground / floor, as those locations typically are accessible and convenient.

[0027] Then, a 360° camera is used as a capturing device to obtain spherical (e.g., commonly equirectangular) images of the scene. Alternatively, another way of obtaining the input set of 360 images can be implemented by extracting frames from a captured 360° video. Figure 2A is a schematic drawing illustrating an example of an equirectangular image which has a full view of an area 200. As shown on Figure 2A, directions of rotation are shown as axes having 2D-coordinates of the equirectangular image, where yaw runs along the illustrated horizontal line labelled “yaw” and pitch runs along the illustrated vertical line labelled “pitch”.

[0028] During the capture, the camera may be attached to a tripod (such that the camera is about 1.5 m above the floor / ground, for example) for stable horizontally levelled high-quality images. Additionally, the camera may be placed on a tripod in several locations with good visibility of the scene and fiducial markers. In other examples, the camera may not be placed on a tripod. In these examples, the captured spherical images may be horizontally levelled with a software operation.

[0029] In block 120 of Figure 1, fiducial marker(s) detection 120 is performed. As previously discussed, some fiducial marker detector approaches use perspective images as an input to operate with geometric transformations of the visible fiducial markers in terms of homographies, for example. Because the images from a 360° camera are in the form of spherical images (e.g., equirectangular images), the spherical images need to be reprojected into several perspective images (of a pinhole camera model, for example).

[0030] One approach to represent spherical images with perspective images is to use a cube mapping of six images, where each image plane coincides with a side of a fixed cube around the center of the camera. When the spherical images are equirectangular images, each projection may be obtained through interpolation of image pixels from the pixel values of the original equirectangular image. Such sampling of a spherical image may be beneficial for a SkyBox-type visualization 140 generated at the final step of the example in Figure 1. However, such sampling may not yield an acceptable detection rate since the fiducial markers may (e.g., often) be located away from the image center and get distorted.

[0031] Thus, fiducial marker detection 120 of examples herein includes spherical image sampling for fiducial marker detection, as discussed further with reference to Figure 4.

[0032] As discussed further herein, a computing device 1100, 1200 is configured to operate according to embodiments. Computing device 1100 includes processing circuitry 1110, 360° camera 1120, memory 1130, and communication interface 1140. Non-limiting examples of computing device 1100 include a 360° camera, an XR / virtual reality (VR) / augmented reality(AR) headset, or a mobile computing device. In contrast, computing device 1200 is a cloudbased computing device that does not include a 360° camera. Computing device 1200 includes processing circuitry 1220, memory 1230, and communication interface 1240. Communication interface 1240 can interface with communication interface 1340 of camera 1300 (e.g., to obtain spherical images from 360° camera 1300). Camera 1300 further comprises a 360° camera, processing circuitry 1320, and memory 1330.

[0033] Operations of the computing device 1100, 1200 are discussed with reference to the flow chart of Figure 4 according to some embodiments. For example, modules may be stored in memory 1130, 1230 of computing device 1100, 1200 and these modules may provide instructions so that when the instructions of a module are executed by a respective computing device 1100, 1200 at processing circuitry 1110, 1220, the processing circuitry 1110, 1220 performs respective operations of the flow chart.

[0034] Some embodiments are directed to a method performed by a computing device for fiducial marker detection in a plurality of spherical images. The method includes sampling (operation 410) the plurality of spherical images including at least one fiducial marker into a plurality of sets of perspective images. The method further includes detecting (operation 420) the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker. The method further includes registering (operation 430) the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

[0035] The plurality of spherical images, in some embodiments, include horizontally levelled spherical images from a 360° camera.

[0036] In some embodiments, the sampling (operation 410) includes reprojecting a spherical image from the plurality of spherical images into a perspective image based on a yaw angle of a respective perspective image center, a pitch angle of the respective perspective image center and a FoV of the at least one respective perspective image, and a roll angle of about 0° degrees.

[0037] For example, when an image is sampled with X° yaw and Y° pitch, the sampling process focuses on the area 200 of an equirectangular image with coordinates (X, Y) as shown in Figure 2A.

[0038] The plurality of sets of perspective images of some embodiments include (i) at least one first set of perspective images with a pitch angle of about 0°, (ii) at least one second set of perspective images with a pitch angle in a range from about -15° to about -75°, and (iii) at least one third set of perspective images with a pitch angle of about -90°.

[0039] The plurality of sets of perspective images of some embodiments include a number of sets of perspective images in a range from three to twelve sets of perspective images.

[0040] In some embodiments, the plurality of sets of perspective images include three sets of perspective images.

[0041] Each sampled image can be characterized by the yaw angle of the image center, the pitch angle of the center, and a FoV of the image. In some examples, the roll angle is fixed at zero degrees. An example of such sampling is presented in Figures 2A-2C, where as discussed, Figure 2A depicts the directions of rotation for the yaw and the pitch, and Figures 2B and 2C give an example of one reprojection.

[0042] In some embodiments, three sets of perspective images include (i) a first set of perspective images with a pitch angle of about 0°, (ii) a second set of perspective images with a pitch angle of about -45°, and (iii) a third set of perspective images with a pitch angle of about - 90°.

[0043] The plurality of spherical images, in some embodiments, include (i) images of an area including a floor or a ceiling and at least one wall, (ii) respective spherical image from a 360° camera positioned at about a same height above the floor or below the ceiling, and (ii) at least one fiducial marker is positioned in at least one of the respective spherical images on at least one of the floor or the ceiling and the at least one wall.

[0044] In some embodiments, the first set of perspective images includes a first set of eight perspective images distributed across eight yaw angles and the pitch angle of about 0°.

[0045] Further, in some embodiments, the first set of eight perspective images include a FoV of 120°.

[0046] In some embodiments, the second set of perspective images includes a second set of eight perspective images distributed across eight yaw angles and the pitch angle of about -45°.

[0047] Further, in some embodiments, the second set of eight perspective images include a FoV of 120°.

[0048] In some embodiments, the third set of perspective images include one perspective image with the pitch angle of about -90°. The one perspective image can include a FoV of 150°.

[0049] Thus, in some examples, the sampling can include sampling of eight perspective images (for respective 8 yaw angles) at pitch 0°, eight perspective images (for respective 8 yaw angles) at pitch —45° and one high FoV image at pitch —90°.

[0050] An example of such sampling is discussed further. Figure 3B is a schematic drawing showing sampling directions of rotation in this example, as discussed further below.

[0051] In the first set, the sampling can include sampling of eight perspective images (for respective 8 yaw angles of the solid lines 1-8 shown in Figure 3B) at pitch 0°).

[0052] In a first example, the eight perspective images can be evenly distributed across the yaw angles (e.g., cardinal and intercardinal directions N, NW, W, SW, S, SE, E, NE) and zeropitch (directed parallel to the ground / floor) are sampled. These perspective images can have a FoV of 120°. Images in the first set are meant to include images that are directed towards a fiducial marker(s) positioned on a wall(s) approximately on the level of the camera.

[0053] In the second set, the sampling can include sampling of eight perspective images (for respective 8 yaw angles of the dashed lines la-8a shown in Figure 3B) at pitch —45°.

[0054] Continuing with the first example, the sampled eight perspective images of the second set can have the same distribution of yaw angles as in the first set, but with a pitch of —45° (e.g., tilted downwards) and a FoV 120°. Images in the second set are meant to include a fiducial marker(s) on the ground / floor, as well as lower parts of a wall(s), to place such sub-areas in the center of an image.

[0055] In the third set, the sampling can include sampling of one high FoV image at pitch —90° as shown by the dotted line 9 in Figure 3B.

[0056] Continuing with the first example, the sampled one image of the third set can face directly downwards, that is with a pitch of —90°. This image can have a larger FoV of 150° for a fiducial marker(s) positioned on the ground / floor (e.g., on the ground / floor about 3.5m away from the base of a tripod to appear in the image). In this example, this image has its projective plane parallel to the ground, removing distortion from the fiducial marker(s) lying on the ground / floor.

[0057] An example of a reprojection from an equirectangular image to a perspective image is shown in Figures 2A-2C. Figure 2 A shows area 200 of an equirectangular image having direction of rotation axes labelled yaw and pitch; Figure 2B shows sub-area 202 visible at yaw = 45°, pitch = 0° with a 120° FoV; and Figure 2C shows the sub-area 202 reprojected into a perspective image.

[0058] In some embodiments, detecting at least one fiducial marker(s) includes identifying a location of the at least one fiducial marker within at least one respective perspective image.

[0059] For example, when a fiducial marker(s) is detected in at least one projection using a canonical fiducial marker detection process, its location (e.g., comer points / bounding box) is reprojected back to the original spherical image (e.g., equirectangular image), providing a detection of the fiducial marker(s) in the 360° scene. The detected fiducial marker(s), including their locations (e.g., comer points / bounds) together with an identifier for each fiducial marker, are then passed to a 3D reconstmction process together with the spherical images (e.g., equirectangular images).

[0060] In some embodiments, registering at least one detected fiducial marker includes reprojecting the location of the at least one detected fiducial marker on the spherical image.

[0061] Continuing with the above example and referring to Figure 1 , a 3D reconstruction process 130 can take a set of 2D images from the scene to build a sparse 3D model using Structure-from-Motion, for example. This process 130 can also obtain the image positions with respect to each other. COLMAP, OpenSfM or another such process can be used.

[0062] Continuing with the above example, as shown in Figure 1 , the camera poses and sparse model can then be used in a densification step, which increases the density (e.g., the number of points per unit of volume). Example densification processes include, without limitation, multi-view stereo (MVS) and neural radiance fields (NeRF).

[0063] As shown in Figure 1, visualization 140 can be performed in a Skybox-type environment using the available data from the 3D reconstruction process 130. Continuing with the above example, the spherical images (e.g., equirectangular images) are reprojected into perspective images using standard cube mapping, for example, for the purpose of immersive view rendering. Figure 3A is a schematic drawing showing central axes of cubemap directions for the sampled perspective projections. It is noted that in this example, these reprojections are disconnected from the reprojections used for fiducial marker(s) detection. The camera poses estimated during the 3D reconstruction 130 are used for relative positioning of different views for correct navigation between them. A dense point cloud is then used in this example for geometric analysis of the scene. An example of such analysis is depth estimation and distance measurements, which is one area where, e.g., fiducial marker(s) detection is applied for global scale correction.

[0064] In some embodiments, the respective perspective images include an overlap between the respective perspective images and the overlap is greater than or equal to 50%. Such overlap may increase detection of a fiducial marker(s). For example, when a fiducial marker(s) is positioned at the edges of two respective images.

[0065] Further, in some embodiments, respective spherical images in the plurality of spherical images include respective equirectangular images.

[0066] As previously discussed, as a consequence of sampling of the present disclosure, the detection rate of fiducial markers in spherical images taken by a 360°camera may be increased, while keeping the runtime of the detection method sufficiently low. Empirical improvements of examples of the method of the present disclosure are outlined below in Table 1 compared to sampling of six overlapping perspective images with a FoV of 120° in a cubemap arrangement. The cubemap arrangement in these examples used conventional sampling as used in applications exposing digitized 3D scene by means of SkyBox.

[0067] Table 1. Detection rate improvements on datasets having different amounts of data / images, where each number shows how many time fiducial markers were detected in the scene:

[0068] As previously referenced herein, a computing device 1100, 1210 may be any of a wide variety of computing devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with other devices.

[0069] In some examples, a computing device 1110, 1210 is configured to perform operation discussed herein without direct human interaction. For instance, a computing device 1110, 1210 may be designed to transmit information about detected fiducial markers to another device.

[0070] Figures 5 and 6 are block diagrams of components of a computing device 1100 and a cloud-based computing device 1210, respectively, configured to operate according to some embodiments. As discussed further herein, the computing device can be configured as computing device 1100 or cloud-based computing device 1210. Each of computing device 1100 and cloudbased computing device 1210 includes communication interface 1140, 1240, processing circuitry 1110, 1220, at least one memory 1130, 1230. As previously discussed, computing device 1100 includes a 360° camera 1300; and cloud-based computing device 1210 is configured to communicatively connect to 360° camera 1320. Cameras 1120, 1300 can capture spherical images as discussed herein.

[0071] The processing circuitry 1110, 1220, 1320 may include at least one processor (processor), and the memory 1130, 1230, 1330 may include at least one memory (memory). The processing circuitry 1110, 1220, 1320 is operationally connected to the various components inFigures 5 and 6. The memory 1130, 1230, 1330 stores executable instructions that are executed by the processing circuitry 1110, 1220, 1320 to perform operations. The processing circuitry 1110, 1220, 1320 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processing circuitry 1110, 1220, 1320 is configured to execute the instructions in the memory 1130, 1230, 1330, described herein as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for a computing device.

[0072] Certain computing devices 1100, 1210 may utilize all or a subset of the components shown in Figures 5 and 6. The level of integration between the components may vary from one computing device 1100, 1210 to another computing device 1100, 1210. Further, certain computing devices 1100, 1210 may contain multiple instances of a component, such as multiple processors, memories, communication interfaces, cameras, etc.

[0073] Referring to Figures 5 and 6, computing device 1100, 1210 includes components needed to enable a communication interface 1140, 1240 with another device (for example, a 360° camera, a server, a computer, a cloud-based computing device, or other device). The communication interface 1140, 1240, for example can include a modem that can be used to send and receive data to and from an application that is used in the computing device 1100, 1210. The application can be a game, social media application, map application, etc.

[0074] Processing circuitry 1110, 1220 can process data that is received from the communication interface 1140, 1240 and can prepare data that can be transmitted. Processing circuitry 1110, 1220 also can run at least one application in the computing device 1100, 1210.

[0075] Memory 1130, 1230 can be used to store data that is received, data that can be transmitted, and data that is calculated, stored, and / or used for the processing circuitry 1110, 1220 to function, etc. The memory 1130, 1230 can be organized in at least one hierarchy. Memory 1130, 1230 technology includes, without limitation, static random access memory (SRAM), dynamic random access memory (DRAM), etc. or any combination thereof.

[0076] Functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one ormore virtual environments hosted by one or more of hardware nodes, such as a hardware computing device that operates as a computing device.

[0077] Applications (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0078] Although the computing devices described herein (e.g., XR, AR, VR headsets or glasses) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the computing device, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0079] In certain embodiments, a computing device (1100, 1210) is provided. The computing device (1110, 1210) includes processing circuitry (1110, 1220); and at least one memory (1130, 1230) connected to the processing circuitry (1110, 1220) and storing program code that is executed by the processing circuitry to perform operations. The operations include to perform some or all of the functionality described herein.

[0080] In certain embodiments, a computer program product including a non-transitory storage medium (1130, 1230) storing instructions executable by processing circuitry (110, 1220) of a computing device (1110, 1210) for fiducial marker detection in a plurality of spherical images. Instructions executed by the processing circuitry causes the computing device to perform operations. The operations include to perform some or all of the functionality described herein.

[0081] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users.

[0082] In the above-description of various embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting on the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0083] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0084] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed asecond element / operation in other embodiments without departing from the teachings of the present disclosure.

[0085] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

[0086] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0087] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of the present disclosure may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0088] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocksshown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0089] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present disclosure. All such variations and modifications are intended to be included herein within the scope of the present disclosure. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. A method performed by a computing device for fiducial marker detection in a plurality of spherical images, the method comprising: sampling (410) the plurality of spherical images comprising at least one fiducial marker into a plurality of sets of perspective images; detecting (420) the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and registering (430) the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

2. The method of Claim 1, wherein the plurality of spherical images comprise horizontally levelled spherical images from a 360° camera.

3. The method of any one of Claims 1 to 2, wherein the sampling (410) comprises reprojecting a spherical image from the plurality of spherical images into a perspective image based on a yaw angle of a respective perspective image center, a pitch angle of the respective perspective image center and a field of view of the at least one respective perspective image, and a roll angle of about 0° degrees.

4. The method of Claim 3, wherein the plurality of sets of perspective images comprise (i) at least one first set of perspective images with a pitch angle of about 0°, (ii) at least one second set of perspective images with a pitch angle in a range from about -15° to about -75°, and (iii) at least one third set of perspective images with a pitch angle of about -90°.

5. The method of any one of Claims 1 to 4, wherein the plurality of sets of perspective images comprise a number of sets of perspective images in a range from three to twelve sets of perspective images.

6. The method of any one of Claims 1 to 5, wherein the plurality of sets of perspective images comprises three sets of perspective images.

7. The method of Claim 6, wherein the three sets of perspective images comprise (i) a first set of perspective images with a pitch angle of about 0°, (ii) a second set of perspective imageswith a pitch angle of about -45°, and (iii) a third set of perspective images with a pitch angle of about -90°.

8. The method of Claim 7, wherein the plurality of spherical images comprise (i) images of an area comprising a floor or a ceiling and at least one wall, (ii) respective spherical image from a 360° camera positioned at about a same height above the floor or below the ceiling, and (ii) the at least one fiducial marker is positioned in at least one of the respective spherical images on at least one of the floor or the ceiling and the at least one wall.

9. The method of Claim 8, wherein the first set of perspective images comprises a first set of eight perspective images distributed across eight yaw angles and the pitch angle of about 0°.

10. The method of Claim 9, wherein the first set of eight perspective images comprise a field of view of 120°.

11. The method of any one of Claims 9 to 10, wherein the second set of perspective images comprises a second set of eight perspective images distributed across eight yaw angles and the pitch angle of about -45°.

12. The method of Claim 11 , wherein the second set of eight perspective images comprise a field of view of 120°.

13. The method of any one of Claims 7 to 12, wherein the third set of perspective images comprises one perspective image with the pitch angle of about -90°.

14. The method of Claim 13, wherein the one perspective image comprises a field of view of 150°.

15. The method of any one of Claims 1 to 14, wherein detecting the at least one fiducial marker comprises identifying a location of the at least one fiducial marker within the at least one respective perspective image.

16. The method of any Claim 15, wherein the registering the at least one detected fiducial marker comprises reprojecting the location of the at least one detected fiducial marker on the spherical image.

17. The method of any one of Claims 1 to 16, wherein the respective perspective images include an overlap between the respective perspective images and the overlap is greater than or equal to 50%.

18. The method of any one of Claims 1 to 17, wherein respective spherical images in the plurality of spherical images comprise respective equirectangular images.

19. A computing device (1110, 1210) for fiducial marker detection in a plurality of spherical images, the computing device comprising: processing circuitry (1110, 1220); and at least one memory (1130, 1230) storing instructions executable by the processing circuitry to perform operations to: sample the plurality of spherical images comprising at least one fiducial marker into a plurality of sets of perspective images; detect the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and register the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

20. The computing device (1110, 1210) of Claim 19, wherein the at least one memory stores further instruction executable by the at least one processor to perform further operations comprising operations of any one of Claims 2 to 19.

21. A computer program product comprising a non-transitory computer readable medium (1130, 1230) storing instructions executable by processing circuitry (1110, 1220) of a computing device (1110, 1210) for fiducial marker detection in a plurality of spherical images, the instructions executed by the processing circuitry to perform operations comprising: sample the plurality of spherical images comprising at least one fiducial marker into a plurality of sets of perspective images; detect the at least one fiducial marker in at least one respective perspective image from the plurality of sets of perspective images to obtain at least one detected fiducial marker; and register the at least one detected fiducial marker on a spherical image from the plurality of spherical images.

22. The computer program product of Claim 21, wherein the non- transitory computer readable medium storing further instruction executable by the processing circuitry of the computing device, the further instructions executed by the processing circuitry to perform further operations comprising operations of any one of Claims 2 to 19.

Images