Accurate Camera-to-Display Calibration for Telepresence Videoconferencing

Abstract

A technique for calibrating a telepresence videoconferencing display and camera includes defining a position and orientation vector of a 6DoF camera in a display reference frame based on a plurality of images showing designated mirror surface points and designated reflected display surface points of a mirror. In some implementations, the designated mirror surface points are located at origin markers printed on the mirror. In some implementations, the designated reflected display surface points are positioned in a checkerboard pattern of origin markers on the display.

Description

[Background technology]

[0001] background Telepresence refers to any set of technologies that allow a person to feel as if they are present in a location other than their physical location. Telepresence may involve the interaction of a user's senses with specific stimuli, such as sight, sound, touch, smell, etc. In applications such as telepresence videoconferencing, visual and auditory stimuli are considered. Summary of the Invention [Problem to be solved by the invention]

[0002] overview Implementations described herein relate to calibrating multiple cameras to obtain accurate user eye positions within a stereoscopic three-dimensional (3D) display for telepresence videoconferencing. Because a stereoscopic 3D display displays interleaved pixels for the left and right eyes, it is desirable to obtain accurate estimates of the user's eye positions relative to the display pixels. Such accurate measurement of the user's eye positions within the display requires measuring six degrees of freedom (6DoF), i.e., three position coordinates and three orientation coordinates of the camera-to-display transformation. Provided herein is a method for determining the position and orientation vector of a 6DoF camera within a display reference frame based on multiple images showing designated mirror surface points and designated reflected display surface points of a mirror. In some implementations, the designated mirror surface points are located at origin markers printed on the mirror. In some implementations, the designated reflected display surface points are positioned on the display in a checkerboard pattern of the origin markers. [Means for solving the problem]

[0003] In one general aspect, a method may include receiving image data representing a plurality of images, each of the plurality of images showing (i) a designated point in a plane of a mirror held at a respective one of a plurality of mirror positions relative to a camera and (ii) a designated point in a virtual image display plane containing an image of a display. The method may also include determining a position and orientation of the camera within a reference frame of the display based on the designated point in the plane of the mirror and the designated point in the virtual image display plane. The method may further include causing an image on the display to be presented to a user according to the determined position and orientation of the camera.

[0004] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit, causes the processing circuit to perform a method. The method may include receiving image data representing a plurality of images, each of the plurality of images showing (i) a designated point in a plane of a mirror held at a respective one of a plurality of mirror positions relative to a camera and (ii) a designated point in a virtual image display plane containing an image of a display. The method may also include determining a position and orientation of the camera within a reference frame of the display based on the designated point in the plane of the mirror and the designated point in the virtual image display plane. The method may further include causing an image on the display to be presented to a user according to the determined position and orientation of the camera.

[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will become apparent from the following description and the accompanying drawings, and the appended claims. [Brief explanation of the drawings]

[0006] [Figure 1A] 1 is a front view illustrating an exemplary telepresence videoconferencing system according to implementations described herein. [Figure 1B] 1 is a side view illustrating an exemplary telepresence videoconferencing system consistent with implementations described herein. [Figure 2] FIG. 1 illustrates an exemplary processing circuit configured to determine camera positions and orientations in a stereoscopic 3D display. [Figure 3] 1 is a flowchart illustrating an exemplary method for determining camera positions and orientations in a stereoscopic 3D display. [Figure 4] FIG. 10 illustrates an exemplary mirror with fiducial marks reflecting an image of a checkerboard pattern of fiducial marks on a stereoscopic 3D display. [Figure 5] 1A-1C illustrate exemplary geometries for determining the position and orientation of a camera in a mirror, a virtual image, and a display. [Figure 6] FIG. 10 illustrates an exemplary geometry for determining camera position within a display using a virtual camera that is a reflection of the camera by a mirror. DETAILED DESCRIPTION OF THE INVENTION

[0007] Detailed Description The present disclosure relates to expanding access to and improving telepresence technology. Telepresence refers to any set of technologies that enable a person to feel as if they are present in a location other than their actual location. Telepresence may involve the interaction of a user's senses with certain stimuli, such as sight, sound, touch, smell, etc. In applications such as telepresence videoconferencing, only sight and sound stimuli are considered.

[0008] A telepresence videoconferencing system 100 is shown in FIG. 1A. The system 100 includes a monitor 110 on which three cameras 130(1)-130(3) are positioned. As shown in FIG. 1A, the system 100 is fixed in a room occupied by a user facing the monitor 110. The user facing the monitor views fellow telepresence videoconference participants. In some implementations, the image seen by the user on the monitor 110 is configured to make the fellow participants appear to be present in the room with the user. For example, the cameras 130(1)-130(3) may provide images of the user from various viewpoints (e.g., angles). Such information may be used to provide depth imaging information. The depth imaging information may be combined with texture information to simulate a three-dimensional image of the user within the space occupied by fellow telepresence videoconference participants.

[0009] The monitor 110 may include a stereoscopic three-dimensional display. A stereoscopic 3D display presents a 3D image to a viewer by transmitting a slightly different perspective to each of the viewer's eyes to provide an immersive experience. The viewer's visual system may process the two perspective images to translate the image, including depth perception, resulting in binocular stereopsis so that the viewer can see the image in 3D. Some stereoscopic displays transmit stereoscopic images to each of the viewer's left and right eyes via left and right channels, respectively.

[0010] Because depth translation is performed using stereoscopic images, it is desirable to accurately measure the user's eye position relative to the display coordinates. To accurately measure the user's eye position within the display, six degrees of freedom (6DoF) must be measured: three position coordinates and three orientation coordinates of the camera-to-display transform. To achieve this, conventional telepresence videoconferencing displays provide an estimate of the 6DoF camera-to-display transform.

[0011] A technical problem with the above-mentioned conventional telepresence video conferencing displays is that such estimates provided in conventional telepresence video conferencing displays require empirical head offset errors that are not based on any fundamental geometry of the telepresence system. Therefore, such inaccurate errors result in a small working head volume when measures are taken to reduce LR crosstalk.

[0012] According to implementations described herein, a technical solution to the above-mentioned technical problem includes defining a position and orientation vector of a 6DoF camera in a display reference frame based on multiple images showing designated mirror surface points and designated reflective display surface points of a mirror. In some implementations, the designated mirror surface points are located at origin markers printed on the mirror. In some implementations, the designated reflective display surface points are positioned on the display in a checkerboard pattern of the origin markers.

[0013] The technical advantage of the solution is that it improves telepresence videoconferencing displays, eliminating the need for empirical head offset errors, thereby improving the accuracy of the camera-to-display calibration, and therefore providing a larger field of view with little trade-off between working head volume and LR crosstalk reduction.

[0014] In some implementations, the designated point in the plane of the mirror is located at an origin marker printed on the mirror.

[0015] In some implementations, the fiducial markers are printed directly onto the mirror using an ultraviolet printer.

[0016] In some implementations, the designated points in the virtual image display surface are arranged in a checkerboard pattern of origin markers on the virtual image display surface.

[0017] In some implementations, receiving the image data includes performing an inversion operation on the checkerboard pattern of the origin marker to generate an inverted checkerboard pattern, and in such implementations, determining the position and orientation of the camera within the frame of reference of the display includes recording the positions of designated points located on the inverted checkerboard pattern.

[0018] In some implementations, determining the position and orientation of the camera within the reference frame of the display includes determining a camera position within the plane of the mirror, the camera position within the plane of the mirror being based on the positions of points of a checkerboard pattern on the virtual image display surface.

[0019] In some implementations, the camera position in the plane of the mirror is a constant multiple of the checkerboard virtual image point, where the constant is based on the specified point in the plane of the mirror, the position of the checkerboard pattern point on the virtual image display surface, and the orientation of the mirror.

[0020] In some implementations, determining the position and orientation of the camera within the frame of reference of the display includes determining the position and orientation of the display within the plane of the mirror.

[0021] In some implementations, determining the position and orientation of the camera within the frame of reference of the display includes averaging the position of the camera within the frame of reference of the display for multiple mirror poses.

[0022] In some implementations, the method further includes averaging the camera rotation within the display's reference frame for multiple mirror poses, the averaging including generating a quaternion projection matrix corresponding to the mirror poses and calculating an eigenfunction corresponding to the largest eigenvalue of the quaternion projection matrix.

[0023] 1A shows a front view of an exemplary telepresence videoconferencing system 100. As shown in FIG. 1A, telepresence videoconferencing system 100 includes a display 110, a processing circuit 120, and three camera pods 130(1)-130(3), each including a respective camera.

[0024] Display 110 is a stereoscopic 3D display used for telepresence videoconferencing. Stereoscopic 3D displays present 3D images to an observer by transmitting slightly different perspective views to each of the observer's eyes to provide an immersive experience. The observer's visual system can process the two perspective images to translate the images, including depth perception, resulting in binocular stereopsis so that the observer can see the images in 3D. Some stereoscopic displays transmit stereoscopic images to each of the observer's left and right eyes via left and right channels, respectively.

[0025] In some implementations, the display 110 is a 3D light field display. A 3D light field display is a stereoscopic display that produces an autostereoscopic effect, allowing a viewer to perceive a stereoscopic image without the viewer wearing special headgear. An exemplary 3D light field display uses lenticular optics to produce the autostereoscopic effect. The lenticular optics may be formed as a series of vertically oriented cylindrical lenses formed on a sheet that is fitted over the display screen.

[0026] 1A, cameras 130(1)-130(3) surround display 110 around its periphery and point outward from it to capture images of users sitting in front of display 110. These cameras may record images of the users from various viewpoints and form a three-dimensional or high-resolution aggregate image for display to another user.

[0027] Because display 110 is a stereoscopic 3D display, it is desirable to accurately position the user's eyes at the display. This means that the camera positions 140 and orientations at display 110 are carefully calibrated. To this end, processing circuitry 120 is configured to perform this calibration and determine the exact positions of cameras, e.g., 130(1), at display 110.

[0028] One way to perform calibration is to use a mirror to indicate the display content to the camera. Locations on the mirror may be indicated using fiducial marks, and locations on the image of the display within the mirror may be indicated using fiducial marks arranged in a checkerboard pattern. This is described in more detail in Figures 4-7.

[0029] 1B is a side view of an exemplary telepresence videoconferencing system 100. The side view of system 100 shows a user 150 seated in front of a display 110. When properly calibrated, camera 130(2) accurately records the user's eye position 160 within the display, i.e., in display coordinates. For example, the improved telepresence videoconferencing display 110 has a geocal (geometric camera calibration) accuracy level of less than 2 mm, as opposed to the 15 mm to 25 mm level of inaccuracy in conventional telepresence videoconferencing displays.

[0030] "Accuracy" is understood herein to mean static, or time-independent, error determined by geocal fidelity. This is distinct from dynamic accuracy, which is determined by system latency and is not addressed here. One source of inaccuracy or error may be due to mirrors moving in different styles during calibration, which is suspected to introduce systematic errors into the tracker camera pose.

[0031] 2 illustrates an example of processing circuitry 120. Processing circuitry 120 includes a network interface 222, one or more processing units 224, and non-transitory memory 226. Network interface 222 may include, for example, an Ethernet adapter, a token ring adapter, etc., to convert electronic and / or optical signals received from a network into an electronic format for use by processing circuitry 120. The set of processing units 224 includes one or more processing chips and / or assemblies. Memory 226 may include both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid-state drives, etc. Together, the set of processing units 224 and memory 226 form a processing circuit configured and arranged to perform various methods and functions as described herein.

[0032] In some implementations, one or more of the components of processing circuit 120 may be or include a processor (e.g., processing unit 224) configured to process instructions stored in memory 226. Examples of such instructions shown in Figure 2 include an image manager 230, a calibration manager 240, and a least mean squares manager 250. Additionally, as shown in Figure 2, memory 226 is configured to store various data, which is described with respect to each manager that uses such data.

[0033] The image manager 230 is configured to obtain the image data 232. In some implementations, the image manager 230 obtains the image data 232 over a network via the network interface 222. In some implementations, the image manager 230 obtains the image data 232 via a direct connection. In some implementations, the image manager 230 obtains the image data 232 from a local storage device.

[0034] Image data 232 represents images of display 110 on the mirror at several specific poses. Image data 232 is used by calibration manager 240 to determine the precise position and orientation of a camera (e.g., camera 130(2)) within the display reference frame, i.e., display coordinates. Each image in the image data represents a specific pose of the mirror held by the camera. Exemplary images of the display on the mirror are shown in FIG. 4.

[0035] FIG. 4 illustrates an exemplary mirror 400 with fiducial marks 410(1)-410(4) reflecting an image of a checkerboard pattern 420 of fiducial marks on a stereoscopic 3D display (e.g., display 110).

[0036] It is assumed that a direct geocal internal calibration using a printed chart was performed on the camera (e.g., camera 130(2)) when generating the image, and that lighting, camera focus, exposure time, signal-to-noise ratio (SNR), and motion were optimized for a clear image of the mirror markers and display.

[0037] In some implementations, the origin markers 410(1)-410(4) are square ArUco markers. ArUco markers are composite square markers consisting of a wide, black border and an internal binary matrix that determines its identifier. The black border facilitates fast detection of the border in the image, and the binary coding allows for its identification and the application of error detection and correction techniques. The marker size determines the size of the internal matrix. For example, a 4x4 marker size is composed of 16 bits. Note that the marker can be found rotated within the environment. However, the detection process must be able to determine its original rotation so that each corner can be clearly identified. This may be done based on the binary coding.

[0038] In some implementations, the ArUco markers are printed directly onto the substrate of the mirror 400 using an ultraviolet (UV) printer. An example of such a UV printer is the Hewlett-Packard (HP) Scitex FB550, which supports 1200x600, 600x600, and 600x300 dots per inch (DPI). In some implementations, the substrate is flat to 1 wavelength per inch, which indicates a flatness deviation of less than 5 nm across a 12-inch mirror. In some implementations, the mirror is a 300 mm semiconductor wafer.

[0039] In some implementations, the checkerboard pattern 420 of origin marks on a display (e.g., display 110) is a ChArUco pattern. ArUco markers and boards are very useful because they can be quickly detected and are versatile. However, one problem with ArUco markers is that their corner locations are not very accurate, even after applying subpixel refinement. In contrast, the corners of a chessboard pattern can be more precisely refined because each corner is surrounded by two black squares. However, finding a chessboard pattern is not as flexible as finding an ArUco board; it must be fully visible and does not tolerate occlusion. The ChArUco pattern combines the advantages of these two approaches. Because the ArUco portion allows for occlusion or partial visibility, it is used to interpolate the locations of chessboard corners to have the versatility of a marker board. Furthermore, because the interpolated corners belong to the chessboard, they are very accurate with subpixel precision.

[0040] Returning to Figure 2, image data 232 includes a plurality of origin marker locations 234(1)...234(N) (e.g., ArUco marker locations), as shown in Figure 2, where N is the number of orientations represented in image data 232. Each origin marker location, e.g., 234(1), includes the location of four markers at each corner of mirror 400 (or, for a 300 mm semiconductor wafer, a square inscribed in a circular mirror). Calibration manager 240 uses the four markers at each orientation to calculate camera coordinates, i.e., the camera reference frame

[0041]

number

[0042] Calculate the 6DoF pose of the mirror with respect to where R is the orientation vector and T is the translation (position) vector.

[0043] The image data 232 also includes a plurality of checkerboard pattern locations 236(1)...236(N), e.g., ChArUco pattern marker locations, as shown in Figure 2. Each checkerboard pattern location, e.g., 236(1), includes the location of each of the origin marker corners at the checkerboard pattern locations 236(1)...236(N). These locations are then used by the calibration manager 240 to calculate the camera coordinates, i.e., the camera reference frame

[0044]

number

[0045] Calculate the 6DoF pose of the virtual image of the display relative to the object. In some implementations, image manager 230 includes checkerboard inversion manager 231. Checkerboard inversion manager 231 is configured to invert, i.e., flip, checkerboard pattern locations 236(1)...236(N) to be detected along an axis in the plane of mirror 400. Once these locations are detected by image manager 230, checkerboard inversion manager 231 is configured to invert the flips along an axis in the plane of mirror 400 to record image points of virtual checkerboard pattern 420.

[0046] The image data 232 also includes multiple mirror orientations 238(1)...238(N), as shown in Figure 2. Each mirror orientation, e.g., 238(1), indicates the direction of the mirror surface normal pointing outward from the mirror surface. In some implementations, the multiple mirror orientations 238(1)...238(N) are determined from multiple origin marker positions 234(1)...234(N).

[0047] Calibration manager 240 is configured to determine the position and orientation of a camera, e.g., 130(2), within the frame of reference of display 110 based on image data 232. In some implementations, calibration manager 240 performs intermediate calculations based on image data 232 to generate calibration data 242, which is used to generate the position and orientation of a camera, e.g., 130(2), within the frame of reference of display 110. As shown in FIG. 2, the calibration data includes:

[0048]

number

[0049] Includes: The calibration manager 240 is also configured to estimate the reflection points on the mirrors, which in some implementations are used to perform subsequent reflection transformations, as shown in FIG.

[0050] FIG. 5 illustrates the reflection points on a mirror 510, a virtual image 530, and a display 520, given image data 232.

[0051]

number

[0052] 5 illustrates an example geometry 500 for determining the reflection points. The reflection points can be calculated by solving a line-plane intersection problem as follows:

[0053]

number

[0054] That is, the reflection point on the mirror 510

[0055]

number

[0056] is a constant multiple of the position of the checkerboard pattern points on the virtual image display surface. The calibration manager 240 is also configured to calculate the camera-to-display position vector, where the display position is calculated in the mirror frame of reference (i.e., mirror coordinates).

[0057]

number

[0058] The display position in the reference mirror frame is given by:

[0059]

number

[0060]

number

[0061] In some implementations, it is useful to consider the role of a virtual camera, as shown in Figure 6. Figure 6 illustrates an example geometry for determining the position of a camera 640 within a display 620 using a virtual camera 650, which is a reflection of the camera by a mirror 610. Taking into account the mirror 610 image of the camera 640, a geocal experiment can be envisioned as the virtual camera 650 views the physical display from various poses. As shown in Figure 6, the pose of the virtual camera 650 is determined from the pose of the mirror 610.

[0062] The camera-to-display translation is generated by considering the triangle Δ in Figure 6. The virtual camera geometry simplifies the vector equation structure and explicitly expresses the camera-to-display translation (i.e., the camera in display coordinates) in terms of measurands. By substituting the above equations, the camera-to-display translation is:

[0063]

number

[0064] Finally, a camera-to-display orientation rotation matrix is ​​calculated for each image of the image data 232. The reflection operator M is used to reflect the camera-to-virtual image rotation matrix, but must be applied in a "sandwich" fashion to maintain mirror (i.e., polar) symmetry.

[0065]

number

[0066] Least squares averaging manager 250 is configured to average the camera-to-display positions over several poses to find a least squares estimate of the camera-to-display translation vector as least squares averaging data 252 .

[0067]

number

[0068] In contrast, rotation vector and / or matrix estimates cannot be averaged to find an "average" camera-based rotation vector. However, under unitary constraints, a rotation matrix estimate (3DoF) can be found that minimizes the Frobenius norm.

[0069]

number

[0070] In some implementations, a least-squares estimate of the rotation matrix may be computed using quaternions. Advantages of using quaternions include avoiding iterative nonlinear optimizers and being able to trap in local minima during gradient descent. The quaternion that minimizes the least-squares error is the eigenvector (e.g., eigenvector data 254(1)...254(N)) corresponding to the largest eigenvalue of the quaternion projection matrix Q251. Note that the i-th quaternion q i Note that is constructed for each mirror pose using the Rodrigues vector definition.

[0071]

number

[0072] The end result is the camera position in display coordinate data 260.

[0073]

number

[0074] The effect of this 6DoF data is to cause the image on the display to be presented to the user according to this determined position and orientation of the camera.

[0075] The components of processing circuitry 120 (e.g., modules, processing unit 224) may be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of processing circuitry 120 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, the functionality and processing of the components of processing circuitry 120 may be distributed across several devices of the cluster of devices.

[0076] The components of processing circuitry 120 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of processing circuitry 120 of FIG. 2 may be or include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions executable on a computer). For example, in some implementations, one or more portions of the components of processing circuitry 120 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of a component may be included in different modules and / or different components than those shown in FIG. 2, including combining functionality shown as two components into a single component.

[0077] Although not shown, in some implementations, components of processing circuit 120 (or portions thereof) may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, components of processing circuit 120 (or portions thereof) may be configured to operate within a network. Thus, components of processing circuit 120 (or portions thereof) may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or include a local area network (LAN), a wide area network (WAN), etc. The network may be or include a wireless network and / or a wireless network implemented using, for example, gateway devices, bridges, switches, etc. The network may include one or more segments and / or have portions based on various protocols, such as Internet Protocol (IP) and / or proprietary protocols. The network may include at least a portion of the Internet.

[0078] In some implementations, one or more of the components of the search system may be or include a processor configured to process instructions stored in a memory, for example, image manager 230 (and / or portions thereof), calibration manager 240 (and / or portions thereof), and least mean squares manager 250 (and / or portions thereof) are examples of such instructions.

[0079] In some implementations, memory 126 may be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, memory 226 may be implemented as two or more memory components (e.g., two or more RAM components or disk drive memory) associated with components of processing circuit 120. In some implementations, memory 226 may be database memory. In some implementations, memory 226 may be or include non-local memory. For example, memory 226 may be or include memory shared by multiple devices (not shown). In some implementations, memory 226 may be associated with a server device (not shown) in a network and configured to function for components of processing circuit 120. As shown in FIG. 2, memory 226 is configured to store various data, including image data 232 and image calibration data 242.

[0080] 3 is a flowchart illustrating an exemplary method 300 for generating a correction wavefront for image calibration. Method 300 may be performed by a software configuration described in connection with FIG. 2 , residing in memory 226 of processing circuit 120 and executed by set of processing units 224.

[0081] At 302, image manager 230 receives image data (232) representing a plurality of images, each of which shows (i) a designated point in the plane of a mirror held at a respective one of a plurality of mirror positions relative to a camera (e.g., 130(2)) and (ii) a designated point in the plane of a virtual image display containing an image of display (110).

[0082] At 304, calibration manager 240 determines the position and orientation of the camera within the display's frame of reference based on the specified points in the plane of the mirror and the specified points in the plane of the virtual image display.

[0083] At 306, processing circuitry 120 causes an image on a display to be presented to a user according to the determined camera position and orientation.

[0084] While several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present specification.

[0085] It will also be understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, the element may be directly on, connected to, or coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, connected to, or directly coupled to another element, there are no intervening elements present. The phrases directly on, directly connected, or directly coupled may not be used throughout the detailed description, but elements shown as directly on, directly connected, or directly coupled may be referred to in this manner. The appended claims of this application may be amended to recite the example relationships described herein or shown in the figures.

[0086] While several features of the described implementations have been illustrated as described herein, many variations, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such variations and modifications as fall within the scope of the implementations. It should be understood that these have been presented by way of example only, and not limitation, and that various changes in form and details may be made. Any portions of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the various implementations described.

[0087] Additionally, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve desirable results. Additionally, other steps may be provided or steps may be eliminated from the described flows, and other components may be added to or eliminated from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A method comprising: receiving image data representing a plurality of images, each of the plurality of images showing (i) a designated point in a plane of a mirror held at a respective mirror pose of a plurality of mirror poses relative to a coordinate system of a camera, and (ii) a designated point in a virtual image display surface, the virtual image display surface including an image of a display, the method further comprising: determining a position and orientation of the camera within a reference frame of the display based on the designated point in the plane of the mirror and the designated point in the plane of the virtual image display, the determining including providing an estimate of a rotation matrix defining the orientation, the estimate based on the plurality of mirror poses, the method further comprising: and causing an image on the display to be presented to a user according to the determined position and orientation of the camera.

2. The method of claim 1 , wherein the designated point in the plane of the mirror is located at an origin marker printed on the mirror.

3. The method of claim 2 , wherein the fiducial markers are printed directly onto the mirror using an ultraviolet printer.

4. The method of claim 1 , wherein the designated points in the virtual image display surface are arranged in a checkerboard pattern of origin markers on the virtual image display surface.

5. The step of receiving the image data includes: performing an inversion operation on the checkerboard pattern of the origin marker to generate an inverted checkerboard pattern; Determining the position and orientation of the camera within a frame of reference of the display comprises:

5. The method of claim 4, further comprising the step of recording the locations of designated points disposed on said inverted checkerboard pattern.

6. Determining the position and orientation of the camera within a frame of reference of the display comprises: The method of claim 4 , comprising determining a camera position within the plane of the mirror, the camera position within the plane of the mirror based on positions of points of the checkerboard pattern on the virtual image display surface.

7. 7. The method of claim 6, wherein the camera position in the plane of the mirror is a constant multiple of the position of the point of the checkerboard pattern on the virtual image display surface, the constant being based on the specified point in the plane of the mirror, the position of the point of the checkerboard pattern on the virtual image display surface, and an orientation of the mirror.

8. Determining the position and orientation of the camera within a frame of reference of the display comprises: The method of claim 1 , comprising determining a position and orientation of the display within the plane of the mirror.

9. Determining the position and orientation of the camera within a frame of reference of the display comprises: The method of claim 1 , comprising averaging the position of the camera in the frame of reference of the display for the plurality of mirror poses.

10. 1. A method comprising: receiving image data representing a plurality of images, each of the plurality of images showing (i) a designated point in a plane of a mirror held at a respective one of a plurality of mirror positions relative to a camera, and (ii) a designated point in a virtual image display surface, the virtual image display surface including an image of a display, the method further comprising: determining a position and orientation of the camera within a reference frame of the display based on the designated point in the plane of the mirror and the designated point in the plane of the virtual image display; causing an image on the display to be presented to a user according to the determined position and orientation of the camera; Determining the position and orientation of the camera within a frame of reference of the display comprises: averaging the position of the camera within the frame of reference of the display for the plurality of mirror poses; The method further includes averaging rotations of the camera within the reference frame of the display for the plurality of mirror poses, the averaging comprising: generating a quaternion projection matrix corresponding to the mirror pose; and calculating an eigenfunction corresponding to the largest eigenvalue of the quaternion projection matrix.

11. A computer program comprising code which, when executed by a processing circuit, causes said processing circuit to carry out a method according to any one of claims 1 to 10.

Images