Method of object registration in video perspective augmented reality system

Abstract

In one embodiment, a method includes capturing a stereoscopic image of a predetermined calibration object having a predetermined pose using a pair of cameras of a video perspective AR system, and obtaining position data for each of the pair of cameras as the stereoscopic image is captured. The method includes generating a 3D reconstruction of the calibration object. The method includes performing a registration process by: generating a stereoscopic virtual view including a predetermined calibration object using parameters of each of a pair of stereoscopic virtual cameras, and determining, for each of the stereoscopic virtual views, one or more differences between a real-world calibration object corresponding to a predetermined calibration object in the stereoscopic image and a virtual rendering of the calibration object, and using these differences to adjust virtual camera parameters and repeat a registration process or store parameters for the system.

Description

Technical Field

[0001] The present application generally relates to video see-through augmented reality (AR). More specifically, the present application relates to a method for object registration in a video see-through augmented reality (AR) system. Background Art

[0002] Video perspective augmented reality (AR) outputs an image (such as a video) of a real-world environment and one or more virtual objects. One or more cameras (such as a pair of stereo cameras) capture images of the physical environment near (one or more) cameras. One or more virtual objects may be superimposed on at least some of the images captured by the one or more cameras. For example, a portion of a graphical user interface (GUI) may be superimposed on at least some of the images. As another example, virtual animals, tools, displays, etc. may be superimposed on at least some of the images.

[0003] In contrast to optical see-through systems (which pass light from the physical environment near the optical see-through system through one or more lenses so that the light remains near the optical see-through system), video see-through systems generate video of the environment near the camera(s). Thus, the video may be displayed on any suitable display, which may also be near the camera(s) (e.g., as part of a head mounted display system), may be remote from the camera(s) (e.g., may be video displayed on a device that is not near the camera(s)), and / or may be part of an integrated device that includes at least some of the camera(s) (e.g., a video head mounted device may include at least a pair of cameras and may include a display for displaying the video from the cameras and any superimposed virtual objects). Summary of the invention

[0004] Technical Solution

[0005] In an embodiment of the present disclosure, a method for object registration in a video perspective augmented reality (AR) system includes capturing a stereoscopic image of a predetermined calibration object with a predetermined pose using a pair of cameras of the video perspective AR system. The method includes obtaining position data of each of the paired cameras by a position sensor when the stereoscopic image is captured. The method generates a 3D reconstruction of the predetermined calibration object based on the stereoscopic image and the position data. The method includes performing a virtual object registration process. The virtual object registration process includes using parameters of each of the paired stereo virtual cameras to generate a stereoscopic virtual view including the predetermined calibration object based on the stereoscopic image, the position data and the 3D reconstruction, wherein each virtual camera in the paired stereo virtual cameras is located at an eye position for viewing content on the video perspective AR system. The virtual object registration process includes determining one or more differences between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicating a virtual rendering of the predetermined calibration object for each of the stereoscopic virtual views. The virtual object registration process includes: when at least one of the one or more differences determined is greater than a corresponding threshold, adjusting one or more parameters of at least one of the paired stereo virtual cameras and re-performing the virtual object registration process. The virtual object registration process includes: when none of the one or more differences is greater than a corresponding threshold, storing parameters of each of the paired stereo virtual cameras in association with the video see-through AR system.

[0006] In an embodiment of the present disclosure, one or more computer-readable storage media contain instructions for capturing a stereoscopic image of a predetermined calibration object with a predetermined pose using a pair of cameras of a video perspective AR system. The one or more computer-readable storage media contain instructions for obtaining position data of each of the paired cameras by a position sensor when the stereoscopic image is captured. The one or more computer-readable storage media contain instructions for generating a 3D reconstruction of the predetermined calibration object based on the stereoscopic image and the position data. The one or more computer-readable storage media contain instructions for performing a virtual object registration process. The virtual object registration process includes using parameters of each of the paired stereo virtual cameras to generate a stereoscopic virtual view including the predetermined calibration object based on the stereoscopic image, the position data, and the 3D reconstruction, wherein each virtual camera in the paired stereo virtual cameras is located at an eye position for viewing content on the video perspective AR system. The virtual object registration process includes determining one or more differences between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicating a virtual rendering of the predetermined calibration object for each of the stereoscopic virtual views. The virtual object registration process includes: when at least one of the one or more differences determined is greater than a corresponding threshold, adjusting one or more parameters of at least one of the paired stereo virtual cameras and re-performing the virtual object registration process. The virtual object registration process includes: when none of the one or more differences is greater than a corresponding threshold, storing the parameters of each of the paired stereo virtual cameras in association with the video see-through AR system.

[0007] In an embodiment of the present disclosure, a system includes one or more memories storing instructions. The system includes one or more processors coupled to the one or more memories. The one or more processors are operable to execute instructions to capture a stereoscopic image of a predetermined calibration object with a predetermined posture using a pair of cameras of a video perspective AR system. The one or more processors are operable to execute the instructions to obtain position data of each of the paired cameras through a position sensor when capturing a stereoscopic image. The one or more processors are operable to execute the instructions to generate a 3D reconstruction of the predetermined calibration object based on the stereoscopic image and the position data. The one or more processors are operable to execute instructions to perform a virtual object registration process. The virtual object registration process includes using parameters of each of the paired stereo virtual cameras to generate a stereoscopic virtual view including a predetermined calibration object based on the stereoscopic image, the position data, and the 3D reconstruction, wherein each virtual camera in the paired stereo virtual cameras is located at an eye position for viewing content on the video perspective AR system. The virtual object registration process includes determining one or more differences between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicating a virtual rendering of the predetermined calibration object for each of the stereoscopic virtual views. The virtual object registration process includes: when at least one of the one or more determined differences is greater than a corresponding threshold, adjusting one or more parameters of at least one of the paired stereo virtual cameras and re-performing the virtual object registration process. The virtual object registration process includes: when none of the one or more differences is greater than a corresponding threshold, storing the parameters of each of the paired stereo virtual cameras in association with the video see-through AR system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example representation of a video see-through AR system is shown.

[0009] Figure 2A and Figure 2B An example method for correcting image transformations caused by components of a video see-through AR system is shown.

[0010] Figure 3 Shows the implementation Figure 2A and Figure 2B An example method for an example method.

[0011] Figure 4 An example model for a video perspective camera is shown.

[0012] Figure 5 An example model for image correction is shown.

[0013] Figure 6 An example model for image correction is shown.

[0014] Figure 7 An example map for image rectification is shown.

[0015] Figure 8 Example projections for image correction are shown.

[0016] Fig. 9 An example model for transforming image frames from a perspective camera to a virtual camera is shown.

[0017] Fig.10 An example transformation is shown in which there is a translation.

[0018] Fig.11 An example model showing a lens is shown.

[0019] Fig.12 An example process for correcting distortion caused by a display lens is shown.

[0020] Fig.13 An example model of a virtual camera is shown.

[0021] Fig.14 An example set of transformations is shown.

[0022] Fig.15A and Fig. 15B An example method for calibrating a video see-through AR system is shown.

[0023] Fig.16 Shows the implementation Fig.15A and Fig. 15B Example processing of the example method.

[0024] Fig.17 Shown is the method for executing Fig.16 Example processing and Fig.15A and Fig. 15B Example schema for an example method.

[0025] Fig.18 An example of determining the field of view of a display lens and a perspective camera is shown.

[0026] Fig.19 An example system including 4 cameras for capturing real-world content is shown.

[0027] Fig. 20 An example of a virtual object being misregistered is shown.

[0028] Fig.21 An example method for virtual object registration is shown.

[0029] Fig. 22 Shows the implementation Fig.21 An example process of the method.

[0030] Fig.23 An example architecture and coordinate system for performing dynamic object registration is shown.

[0031] Fig.24 An example of video transformation in dynamic object registration is shown.

[0032] Fig.25 An example algorithm for dynamic object registration is shown.

[0033] Fig.26 An example computer system is shown. DETAILED DESCRIPTION

[0034] Figure 1 An example representation of a video see-through AR system 100 is shown. A target object 101 in the environment of the system 100 is captured by a see-through camera 103, which may include a see-through camera lens 102 through which light from the target object 101 passes. Figure 1 In an example of , the system 100 may include a display panel 104 for displaying video AR content to a user. For example, the system 100 may be a head-mounted device worn by a user. The content displayed on the display panel 104 passes through a display lens 105 and reaches a user eye viewpoint 106.

[0035] like Figure 1 As shown in the example of , light from a target object 101 passes through a series of system components, and when the light reaches the eye viewpoint 106, it has been transformed (e.g., distorted) by the system 100. The overall transformation created by the system 100 can be considered as a series of separate transformations performed by the various components. For example, one transformation can be caused by the perspective camera itself, which can be modeled using the perspective camera model K1. As another example, light can be transformed by a display lens, which can be modeled using the display lens model K5. As another example, a user view 106 of displayed content on the display 104 can be modeled as a virtual camera, such as using a virtual camera model K3 at the user's viewing position (i.e., the position of the user's eyes for viewing the content on the display). Figure 1 An example representation of a video see-through AR system for a single camera and a single eye is shown, and applies to systems that use multiple cameras and display content to both eyes. For example, for a system that displays stereo images, a separate model can be made for each camera, each display lens, and each virtual camera corresponding to each position of the user's eyes.

[0036] Figure 2A and Figure 2B An example method for correcting image transformations caused by components of a video see-through AR system is shown. As described herein, Figure 2AThe method creates a transformation map prior to real-time image rendering. This can significantly speed up the image rendering process when the user is using a video see-through AR system because the transformation that must be implemented for a given image frame in order to correct distortion caused by system components does not need to be determined for each frame during rendering, which is a time-sensitive process. For example, if latency is introduced into the rendering process, the displayed content may not temporally match the movement of the system. For example, if a user is wearing a head-mounted device and suddenly turns their head, the delay in the rendering stage causes the displayed content to move delayed relative to the user's head movement, resulting in an inaccurate scene display and potentially causing the user to feel disoriented in viewing the content (such as dizziness).

[0037] Figure 2A Step 205 of the example method of the video see-through augmented reality (AR) system includes accessing an image captured by a camera in a pair of stereo cameras. In embodiments, step 205 may include, for example, capturing an initial image using a camera in a pair of stereo cameras. In embodiments, step 205 may include accessing the image from a memory of a client or server computing device, and may include receiving the image from another electronic device, such as via a Wi-Fi or Bluetooth connection. In embodiments, Figure 2A The steps of the example method may be performed by a single computing device, or may be performed by more than one computing device. For example, Figure 2A Some or all of the steps of the example method may be performed by a video see-through system, such as by one or more computers integrated within the system. As another example, Figure 2A Some or all steps of the example method may be performed by one or more other computing devices. For example, some or all of the steps may be performed by a server device or by a client device (e.g., a personal computer, a smart phone, etc.).

[0038] like Figure 2A As shown in the example method of , the camera can be one of a pair of stereo cameras that can ultimately be used to display three-dimensional content to a user. For example, a left camera can be used to capture a left image of a scene and a right camera can be used to capture a right image of the scene. The left image can be displayed to the user's left eye (e.g., through a left display lens, such as Figure 1 ) and the right image may be displayed to the user's right eye, for example, through a separate right display lens for that eye. In embodiments, one or more pairs of stereo cameras may be integrated into an overall display system, such as, for example, a head-mounted device including a display structure for viewing captured content and AR / VR content. Although Figure 2AThe example method describes processing of images captured by one camera in a pair of stereo cameras, but the same method can be used for the other camera, although as described herein, the specific model parameters used to generate the transformation mapping for that camera may be different, for example, due to physical differences in the two cameras or associated display lenses, etc.

[0039] Figure 2A Step 210 of the example method includes determining a modification to an image based on each of one or more models of one or more components of a video see-through AR system. Figure 3 An example method of implementing the example method of FIG. 2 is shown.

[0040] exist Figure 3 2 , process 310 illustrates a specific implementation of the example method of FIG. 2 . As described more fully herein, the process generates a transform map 319 that identifies a frame-independent transform to apply to render one or more subsequent images captured by the camera. Generating the transform map prior to rendering subsequent images eliminates the need to determine a corrected transform for the video see-through AR system during the rendering process, thereby improving latency and saving computational resources during rendering. As described more fully below, process 330 illustrates an example rendering process.

[0041] As long as the video see-through system configuration does not change, process 310 provides an accurate transformation map 319 for any subsequent image rendering. The process is device-specific in that any particular instance of a device will need to undergo process 310 in order to determine the transformation map for that particular device because of, for example, physical differences in the mounting and components used. If the system does change significantly, for example, the system is a head-mounted device and the user puts down their head-mounted device, process 310 may need to be repeated to generate an updated transformation map. In embodiments, camera calibration and / or system calibration (e.g., as described more fully below) may be run prior to performing process 310, for example, to determine camera properties and / or lens properties for a particular system. In embodiments, process 310 may be run for a system prior to deployment of the system (e.g., by the manufacturer of the system), and the resulting transformation map may be associated with a particular system corresponding to the transformation map, for example, by storing the map as a file on the system or in an online location associated with the system.

[0042] exist Figure 3 In the example of , after accessing an image captured by a camera, a grid (such as a grid) is created 311 using the image coordinates. The present disclosure contemplates that the grid can take any suitable shape (e.g., rectangular, triangular, etc.) and can have any suitable size. In embodiments, the accessed image can be an image of a known scene, such as a calibration image taken by the camera. In embodiments, the accessed image can be any suitable image taken by the camera. Figure 3 In the example of FIG. 2 , the method of determining a modification to an image based on each of one or more models of one or more components of a video see-through AR system may include creating a coordinate grid from the captured image, and then determining subsequent transformations by using only the grid, i.e., the transformations are not dependent on any particular image or frame captured by the camera, and the transformations are generally applied to the image coordinates of the image captured by that camera.

[0043] In embodiments, the grid may have any suitable size, and the grid element size may be larger than the pixel size of the image. For example, for an image having 3000×3000 pixels, a grid of 300×300 elements may be used, such that each grid element corresponds to a 10×10 block of image pixels, although these image and grid sizes and their relative sizes are merely examples. In embodiments, to determine the transform for each pixel in a subsequent image during rendering, the GPU may interpolate the grid points in the transform map to determine the appropriate transform for that pixel.

[0044] Figure 3 An example grid 305 is shown that has been distorted, for example, by a lens associated with a camera that captured the image. Figure 2A Step 210 of the example method may include creating a mesh 312 of undistorted images using a computational model 313 of the effect of the perspective camera's transformation on the captured images. The computational model 313 of the perspective camera may also be referred to as a model of the perspective camera. Figure 3 As shown in the example of 1 , C 1 ), external parameters (R 1 , T 1 ) or both.

[0045] Figure 4 An example model for a video perspective camera is shown. Figure 4 In the example, O w , (X, Y, Z) is the world coordinate system, O c (x, y, z) is the camera coordinate system, and o i (x, y) is the image coordinate system. The camera matrix of the perspective camera is:

[0046]

[0047] Where (f 1x , f 1y ) represents the focal length, (c 1x , c 1y ) represents the optical center, and s 1 Represents the skew coefficient.

[0048] exist Figure 4 In the above equation, P(X 1 , Y 1 , Z 1 ) is a point in 3D space. If there is no lens distortion, the projection of this point on the image is p(x 1 ,y 1 ). However, due to the distortion created by the camera lens, the projection of the point on the image is p d (x 1d ,y 1d , z 1d ). The lens distortion model uses the lens distortion coefficients available through camera calibration to identify p(x 1 ,y 1 ) and p d (x 1d ,y 1d , z 1d ):

[0049]

[0050] The terms in square brackets represent tangential distortion, while the remaining terms represent radial distortion. In Equation 2, (x 1d ,y 1d ) is the distorted coordinate, (x 1 ,y 1 ) is the undistorted coordinate, (k 11 , k 12 , k 13 ) represents the radial distortion coefficient, and and (p 11 , p 12 ) represents the tangential distortion coefficient. Equation (2) can be expressed as:

[0051]

[0052] in yes Then, the distortion model can be expressed as C 1 =(k 11 k 12 P 11 P 12 k 13 As a result, the perspective camera model is (K 1 , C 1 ). After obtaining the model, it can be used to undo the distortion caused by the perspective camera, e.g. Figure 3 For example, a lookup table can be created by pre-computing an undistorted grid from a distorted grid using the model described herein, and this can be used to undo the distortion in each subsequent frame from the camera, for example, in a video such as Figure 3During the rendering process 330 .

[0053] In an embodiment, Figure 2A Step 210 of the example method may include correcting the initial image, such as by creating a corrected mesh 317 of the image using one or more of a model 313 of the perspective camera that captured the image, a correction model 314, and a model 315 of the virtual camera corresponding to the image (i.e., the "right" virtual camera if the image corresponds to the right side of the system). Correction model 314 may also be referred to as a model of a paired stereo camera.

[0054] The purpose of image rectification is to transform the image pair into a plane and align their corresponding epipolar lines, as represented by:

[0055]

[0056] Where (x 1 ,y 1 ) are the coordinates of the undistorted image, and (x r ,y r ) are the coordinates of the corrected image.

[0057] Figure 5 An example geometric model for image correction is shown. l ,i r ) can be obtained by stereo camera pair (C l , C r ) capture. Correction will be the perspective camera stereo image pair (i l ,i r ) is transformed into a virtual camera pair (I l , I r ), so that the polar line pair (I l , I r ) can be transformed into a modified epipolar pair (l rect_l , l rect_r ), where the modified epipolar lines are parallel and aligned. Thus, the perspective camera pair can be modified to a virtual camera pair. To perform this transformation, the position and orientation of the virtual camera pair can be used as input.

[0058] Stereo camera calibration provides the pose of the right camera relative to the left camera:

[0059] S r =[R r , T r ] (4)

[0060] The intrinsic parameters of the left camera are expressed as:

[0061]

[0062] And the intrinsic parameters of the right camera are expressed as:

[0063]

[0064] Embodiments may use two steps to convert a stereoscopic image pair (i l ,i r ) is corrected to (l l , l r ), including transforming the original image points into the corrected camera coordinate system and mapping the original image points to the image plane of the corrected camera coordinate system.

[0065] Figure 6 An example model for image correction is shown. As shown in the figure, Figure 6 An example model of may include offline processing 610 and online processing 620.

[0066] The offline process 610 may include defining a correction camera coordinate system 613 using a perspective camera model 612, a camera calibration 612 for intrinsic and extrinsic parameters of a stereo camera pair, and a virtual camera model 614. In the correction camera coordinate system, a left correction camera coordinate system and a right correction camera coordinate system are parallel. The defined correction camera coordinate system is then used in an online image correction process 620.

[0067] The online processing 620 may include coordinate system conversion and image point reprojection to rectify the captured image (or in an embodiment, a mesh representing the image) from the perspective camera coordinate system to the rectified camera coordinate system.

[0068] The process of coordinate system conversion may include determining a rotation matrix between the perspective camera coordinate system and the corrected camera coordinate system defined by the offline process 610 and rotating the perspective camera coordinate system to the corrected camera coordinate system 623. Additionally, the online process 620 may include rotating the left perspective image 621 from the left perspective camera coordinate system to the corrected camera coordinate system 623. The same process occurs for the right image 622 from the right perspective camera, as determined by Figure 6 As shown in element 624.

[0069] The process of reprojecting the image points may include mapping the image points in the corrected camera coordinate system from the rotated coordinates of the perspective camera coordinate system to the coordinates of the corrected image points in the corrected camera coordinate system. This occurs for the left image, as shown in element 625, and for the right image, as shown in element 626. The process includes projecting the corrected image points from the corrected camera coordinate system to the image plane coordinate system to create a corrected image. The process occurs for the left image, as shown in element 626. Figure 6 As shown in elements 627 and 629, and for the right image, as Figure 6Elements 628 and 630 are shown.

[0070] After the coordinate system conversion and image point reprojection, an embodiment may then correct the image captured by the perspective camera in the perspective camera coordinate system (or a grid representing the image, as discussed herein) to the correction camera coordinate system. After projecting the corrected image points to the image plane coordinate system, an embodiment may then obtain the final corrected image for use in the image transformation of the video perspective system, as described more fully below.

[0071] To transform the image points to the corrected camera coordinate system, the transformation of the left image can be expressed as:

[0072]

[0073] in, is the rotation around the y-axis and θ is the rotation around the z-axis, which can be derived from the relative translation T between the left and right cameras. r Calculated.

[0074] The transformation of the right image can be expressed as

[0075] M r =M l R r (8)

[0076] Among them, R r is the relative rotation between the left and right cameras.

[0077] To map image points to a modified coordinate system, an embodiment may determine:

[0078] p l (x l ,y l , z l )=M l p ol (x ol ,y ol , z ol ) (9)

[0079] Among them, p ol is the original camera coordinate system C l (x ol ,y ol , z ol ) in the image point, and p l (x l ,y l , z l ) is the corrected camera coordinate system C l (x l ,y l , z l) in the coordinates. Then, the embodiment can l (x l ,y l , z l ) is mapped to P in the corrected camera coordinate system rect_l (x rect_l ,y rect_l , z rect_l ),like Figure 7 As shown in the example of , where:

[0080] P rect_1 (x rect_l ,y rectl , z rect_l )=Φ(p l (x l ,y l , z l )) (10)

[0081] Then, the embodiment can be p l (x l ,y l , z l ) is projected onto the image plane of the corrected camera coordinate system, for example Figure 8 As shown in the example of , where:

[0082]

[0083] Similarly, the transformation of the right image can be calculated as:

[0084] P r =M r P or (12)

[0085] In the embodiment, the image point p r (x r ,y r , z r ) is mapped to Figure 7 The corrected camera coordinate system p shown in rect_r (x rect_r ,y rect_r , z rect_r ):

[0086] p rect_r (x rect_r ,y rect_r , z rect_r )=Φ(p r (x r ,y r , z r )) (13)

[0087] And then p r(x r ,y r , z r ) projected onto Figure 8 The image plane in the corrected camera coordinate system is shown in:

[0088]

[0089] In an embodiment, Figure 2A Step 210 of the example method may include transforming 318 an image frame from an image frame of a video perspective camera to an image frame of a virtual camera, such as by transforming a modified mesh representing the image using one or more of a model 315 of a virtual camera corresponding to the image and a model 316 for transforming a view from a perspective camera to a virtual camera. The transformation may be represented as:

[0090]

[0091] Among them, (x r ,y r ) are the coordinates of the corrected image, and (x m ,y m ) are the coordinates of the image where the viewpoint matches. Model 316 may also be referred to as a model of the rotation and translation of the virtual camera relative to the perspective camera.

[0092] Fig. 9 An example model for transforming an image frame from a perspective camera to a virtual camera using camera viewpoint matching is shown. v (X v , Y v , Z v ) is the virtual camera coordinate system and O s (X s , Y s , Z s ) is the perspective camera coordinate system, and there may be both a rotation R and a translation T between these two coordinate systems. The coordinates of the virtual image are:

[0093] P v =R(P s -O v ) (16)

[0094] Among them, P v is the coordinate of the virtual image, R is the rotation matrix between the virtual camera and the perspective camera, P s are the coordinates of the perspective image, and O v are the coordinates of the virtual camera. The coordinates of the virtual image are:

[0095]

[0096] And the coordinates of the perspective camera image are:

[0097]

[0098] Equation 16 can be rewritten as:

[0099]

[0100] The translation can be expressed as:

[0101] T=-RO v (20)

[0102] And therefore Equation 19 can be expressed as:

[0103]

[0104] Fig.10 An example is shown where there is only a translation in the z direction. In this example, since there is no rotation between the two coordinate systems, R = I and the translation is:

[0105]

[0106] In this example, t x =0,t y = 0, and t z =d. Using equation (21) and the results of this example:

[0107]

[0108] And therefore:

[0109]

[0110] 3D coordinates can be warped into display coordinates:

[0111]

[0112] Where w is the image width and h is the image height. Embodiments may then use the pre-warped virtual image coordinates to obtain a pre-distorted image for display lens geometric distortion and chromatic aberration correction, as described below.

[0113] In an embodiment, Figure 2A Step 210 of the example method may include correcting 320 distortion in the image introduced by the display lens, for example by using a model 321 of the display lens. Fig.11An example model for a display lens is shown. Display lenses can provide a large field of view scene on a display panel close to the user's eyes, and, for example, display lenses are useful in head-mounted displays. However, display lenses create geometric distortion and chromatic aberration. Geometric distortion refers to the phenomenon that lines or surfaces in an image appear distorted or twisted due to the characteristics of the lens. It can appear as barrel distortion, in which objects toward the edge of the frame appear to expand or expand, or pincushion distortion, in which objects toward the edge of the frame appear to shrink or contract. Chromatic aberration, on the other hand, refers to the phenomenon that different wavelengths of light are refracted differently by the lens, resulting in color fringing or color distortion in the image. It can occur as longitudinal chromatic aberration, resulting in out-of-focus colors in front of or behind the focal plane, or as lateral chromatic aberration, resulting in color shifts or fringing at the edges of the image.

[0114] like Fig.11 As shown, if there is no lens between the eye, the light ray traced from the eye to the panel will reach point p on the panel, but if there is a lens between the eye and the panel, it will reach point pd on the panel. d The difference between is due to the geometric distortion caused by the display lens. The display lens will create different geometric distortions in different color channels, known as chromatic aberration.

[0115] The radial distortion model for display lenses can be expressed as:

[0116]

[0117] Where (x 5c ,y 5c ) is the coordinate of the center of distortion, (x 5d ,y 5d ) is the coordinate in the distorted image, (x m ,y m ) are the pixel coordinates in the undistorted image, and (k 51 , k 52 , k 53 ) is the distortion coefficient of the display lens, which can be obtained by display lens calibration.

[0118] Fig.12 An example process for correcting distortion caused by a display lens is shown. Fig.12 In the example of , the display lens creates pincushion distortion. To correct the distortion, for example, as shown by the regular grid shown in the figure, the embodiment can create a grid with barrel distortion to display on the display panel. The barrel distortion is the inverse of the pincushion distortion of the display lens, and thus the light from the barrel-distorted image of the display is distorted so that the image with barrel distortion appears as an undistorted image.

[0119] To correct for chromatic aberration, embodiments may pre-distort separate grids in separate color channels. For example, the grid in the red channel may have a distortion of the coefficients of the red channel, the grid in the green channel may have a distortion of the coefficients of the green channel, and the grid in the blue channel may have a distortion of the coefficients of the blue channel. In this way, embodiments may compensate for lateral chromatic aberration caused by different geometric distortions associated with different wavelengths. Putting these steps together, the coordinates of the undistorted image are:

[0120]

[0121] Among them, (x vsc ,y vsc ) is the display screen coordinate of the virtual image.

[0122] Although the above model of the display lens uses radial distortion for geometric distortion and chromatic aberration, an embodiment may calibrate the display lens and store the geometric distortion and chromatic aberration into a lookup table. Such an embodiment may then directly use the lookup table to create a pre-distorted image to compensate for chromatic aberration and geometric distortion in different color channels. One benefit of using a lookup table is that the lookup table can store any kind of distortion, such as symmetric distortion (radial distortion), asymmetric distortion (tangential distortion), etc. On the other hand, in an embodiment, the lookup table may be too large for a large field of view and a high resolution virtual view. In contrast, a model using a polynomial equation requires only a few parameters to define the model, for example, for use by third-party developers.

[0123] In an embodiment, Figure 2A Step 210 of the example method may include transforming a mesh using a model of a virtual camera, for example, to transform an image from the perspective of a perspective camera to the perspective of a virtual camera. Fig.13 An example model for a virtual camera is shown. To render a virtual image corresponding to the real world environment, the example creates a pinhole model camera matrix K for the virtual camera 3 and the polynomial distortion model C 3 :

[0124]

[0125] Among them, (f 3x , f 3y ) is the focal length, (c 3x , c 3y ) is the optical center, and s 3 is the skewness coefficient, and:

[0126]

[0127] Among them, (x 3d ,y 3d ) is the distorted coordinate, (x 3,y 3 ) is the undistorted coordinate, (k 31 , k 32 , k 33 ) is the radial distortion coefficient, And (p 31 , p 32 ) is the tangential distortion coefficient.

[0128] The distortion model can be expressed as:

[0129] c 3 =(k 31 k 32 p 31 p 32 k 33 ) (30)

[0130] Therefore, the calculation model of the virtual camera model is (K 3 , C 3 ). The parameters of the camera matrix and the coefficients of the distortion model are determined by virtual camera calibration. Fig.13 As shown, for a stereo virtual camera pair, other additional parameters are required, such as inter-pupillary distance (ipd) and field of view Furthermore, when the virtual 3D object is located at the same distance from the virtual image plane, an embodiment may calculate the rotation angles of the left eye and the right eye as:

[0131]

[0132] Where θ is the angle that the left and right eyes need to rotate to see the 3D virtual object at the same distance from the virtual image plane, ipd is the interpupillary distance of the eyes, and f3 is the focal length of the virtual camera, where:

[0133]

[0134] And (f 3x , f 3y ) are the focal lengths in the x and y dimensions respectively.

[0135] Figure 2A Step 215 of the example method may include generating a transformation map for a camera that captured the accessed image based on the modification to the image, wherein the transformation map identifies a frame-independent transformation to apply to rendering a scene based on one or more subsequent images captured by the camera. Figure 3 , transform map 319 identifies a frame-independent transform to be applied to render the scene based on one or more subsequent images captured by the corresponding camera.

[0136] Figure 2BStep 250 of the example method may include obtaining one or more subsequent images captured by a perspective camera in a pair of stereo cameras of a video perspective augmented reality (AR) system.

[0137] Figure 2B Step 255 of the example method may include transforming the one or more subsequent images using a predetermined transformation map that identifies a frame-independent transformation to apply to the one or more subsequent images captured by the perspective camera.

[0138] Figure 2B Step 260 of the example method may include displaying the transformed image on a display panel on the video see-through AR system.

[0139] In the embodiment, according to Figure 2A The process of the example method of predetermines the transformation mapping. In an embodiment, the transformation mapping is predetermined by obtaining a previous image captured by a perspective camera in a pair of stereo cameras of the video perspective AR system, determining a modification to the previous image based on each of one or more models of one or more components of the video perspective AR system, and generating a transformation mapping for the perspective camera that captured the previous image based on the modification to the previous image. The previous image may represent an image captured before one or more subsequent images of steps 250 to 260.

[0140] Where appropriate, the examples may be repeated Figure 2A and Figure 2B Although the present disclosure describes and illustrates one or more steps of the method. Figure 2A and Figure 2B The specific steps of the method occur in a specific order, but the present disclosure contemplates Figure 2A and Figure 2B Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates the execution Figure 2A and Figure 2B Specific components, devices or systems (such as Fig.26 ), but the present disclosure contemplates executing Figure 2A and Figure 2B In addition, the present disclosure contemplates some or all of the computing operations described herein (including Figure 2A and Figure 2B The steps of the example method shown in FIG. 1 may be performed by a computing device (e.g. Fig.26 The invention may be executed by circuits of a computing device), by a processor coupled to a memory, or by any suitable combination thereof.

[0141] Fig.14 Tracking of those discussed above (e.g., Figure 3 A transformation set of examples of . Fig.14 In the example of , a real object (i.e., a rabbit) is captured by each camera of a pair of stereo cameras (although only one capture camera is shown in the figure for convenience). For each image of the left and right stereo pairs, the mesh is undistorted, rectified, and viewpoint matching is performed to transform the view from the view of the camera to the view of the virtual camera, which is offset from the capture camera by a distance d. Figure 3 As shown in the example of , during rendering, a virtual object is created and blended with an image of a real scene, display lens distortion (e.g., pincushion distortion) is corrected, and then the corrected blended image is displayed.

[0142] After the transformation map is obtained for a particular system, the transformation map can then be used to correct images captured by the system for display to a user. As explained above, this avoids having to calculate the transformation and correction for each captured image on the fly during image rendering, thereby reducing latency during the rendering process. For example, rendering process 330 shows an example use of a transformation map previously generated for a particular system. For each image captured by the system's camera, image 331 is distorted by the system components, as discussed above. Figure 3 As shown in element 332 of the example, the transformation map is accessed and used to determine the appropriate undistortion, rectification and viewpoint matching of the image. Figure 3 As shown in the example of , the correction 335 determined by the display lens model discussed above can be used to pre-distort the captured image for display so that when the pre-distorted image passes through the display lens, the captured image appears undistorted. Figure 3 As shown in the example of , one or more virtual objects 334 may be generated during rendering (e.g., by a GPU), and the mixed real and virtual images may be pre-distorted according to correction 335. Finally, the correct mixed image is rendered 333 on a display for viewing by a user. In an embodiment, correction 335 may be stored with transform map 319 and may be part of transform map 319.

[0143] Embodiments may use a model of video transformation in video perspective AR with two perspective cameras (including a left camera and a right camera). In embodiments, the system may include more than two perspective cameras, such as Fig.19 , and the model can be used for each of these multiple perspective cameras as different types of cameras with different lens parameters. Such an embodiment models each camera according to its camera and lens type, and then determines the contribution of each perspective camera to the virtual camera. In an embodiment, a single perspective camera can be used (where the perspective camera and the left display lens create the left "camera", and the perspective camera and the right display lens create the right "camera"), such as in combination with Fig.19 As described more fully herein, the single camera may be modified with a left virtual camera and a right virtual camera, respectively, and the contribution of the perspective camera to the left and right virtual cameras, respectively, is determined.

[0144] As described above, components of a video see-through AR system may introduce distortions and other transformations to content captured by the system and content viewed by a user. For example, light from a scene in the system's environment necessarily passes through a see-through camera lens and a display lens before being viewed by a user, and each of these lenses modifies the light (such as by introducing geometric distortion and / or chromatic aberration). Calibrating the system's display lens and the combination of a display lens and a corresponding see-through camera lens can reduce these distortions and improve the content viewed by the user. As discussed herein, the calibration may be specific to a particular instance of a device, e.g., because each instance may have slight differences due to variations during manufacture or installation of device components, and thus each calibration is performed specifically for a particular instance of a device and identifies parameters specific to that device. In addition, the calibration techniques discussed herein may provide device parameters that can be used to further correct content captured and displayed by the system, e.g., Figure 3 The example processing used by .

[0145] Fig.15A and Fig. 15B An example method for calibrating a video see-through AR system is shown. Fig.16 Shows the implementation Fig.15A and Fig. 15B An example method of example processing, and Fig.17 Shown is the method for executing Fig.16 Example processing and Fig.15A and Fig. 15B Example schema for an example method.

[0146] Fig.15A Step 1505 of the example method may include capturing, by a calibration camera, a calibration pattern displayed on a display of the video see-through AR system. For example, the calibration camera may be Fig.17 The example architecture shown in FIG. Fig.17 As shown, the calibration camera 1702 is located at the eye position for viewing content on the video see-through AR system 1700, that is, at the position where the user's eyes should be when the user views the content displayed by the display panel 1705 of the system (also called "good viewing distance", which can be, for example, 12 mm in an embodiment). In an embodiment, and Fig.17 In the example of , the video see-through AR system 1700 may include a head mounted device 1701, which may include one or more see-through cameras 1706 (e.g., a pair of stereo cameras), each see-through camera 1706 having a camera lens 1707. Fig.17In the example of FIG. 1 , the head mounted device 1701 may include a display panel 1705 having a left display and a right display for displaying real world content captured by the perspective camera and for displaying any virtual content generated for display. The user views the content displayed on the display panel 1705 through one or more display lenses 1704 (e.g., a left display lens for the user's left eye and a right display lens for the user's right eye). In an embodiment, the display panel 1705 of the video perspective AR system 1700 may include one of a pair of display panels.

[0147] In an embodiment, a displayed calibration pattern (which may also be referred to as a first calibration pattern) is shown on the display panel, such as in Fig.17 The calibration pattern shown on the display panel 1705 of the example corresponds to an image of the calibration pattern (which may also be referred to as a second calibration pattern) captured by the perspective camera of the video perspective AR system. For example, Fig.17 A display pattern displayed on an external display 170 is shown. The external display may be any suitable display, such as a TV screen, a monitor, etc. The calibration pattern may be a predetermined pattern with known display characteristics, such as a known geometric pattern and / or a known color profile. For example, the calibration pattern may be a checkerboard pattern of various colors. In an embodiment, calibration may be based on more than one calibration image. For example, the location, position, and / or orientation of the calibration pattern may be changed during the calibration process.

[0148] Fig.15A Step 1510 of the example method may include determining one or more system parameters of the video see-through AR system based on the captured calibration pattern, the one or more system parameters representing the combined distortion caused by a display lens of the video see-through AR system and by a camera lens of the see-through camera. Fig.16 Shown in Fig.17 An example of this step in the context of the sample architecture. Fig.16 The example process of may include three main parts: head mounted device display calibration 1601, perspective camera calibration 1620, and calibration camera calibration 1630. Head mounted device display calibration 1601 is used to determine the distortion (e.g., geometric distortion and chromatic aberration) introduced by a set (e.g., right or left) of head mounted device components as a whole; perspective camera calibration 1620 is used to determine the distortion (e.g., geometric distortion and chromatic aberration) introduced by the perspective camera; and calibration camera calibration 1630 is used to determine the distortion (e.g., geometric distortion and chromatic aberration) introduced by the calibration camera. Although the calibration camera is not part of the video see-through AR system itself, the camera will introduce its own distortions to the images it captures, and these distortions must be corrected in order to determine the distortion introduced by the actual components of the video see-through AR system.

[0149] like Fig.17 As shown in the example of , a calibration process may be performed for each color channel in a plurality of channels (eg, each of three channels in an RGB color representation). Fig.17 An example of may include rendering a calibration pattern 1602 in a selected color channel and then displaying the rendered calibration pattern 1603 on a TV display. The video see-through AR system (in this example, a head mounted device) is placed in a fixed position (e.g., on a table), and a calibration camera is placed behind the system so that it is viewing the entire calibration pattern displayed on the display of the head mounted device, which is displaying the entire calibration pattern captured by the head mounted device. In step 1604, the display (e.g., TV) is set to a specific posture (position and orientation), and in step 1605, the calibration camera captures a picture of the calibration pattern through the system, i.e., as displayed on the system display. In step 1606, the picture captured by the calibration camera is stored in a specific location (such as a folder) associated with the color channel in step 1602. Steps 1604 to 1607 are repeated for each of a plurality of display postures. For example, the user may be instructed to rotate the display (e.g., TV) by a specific amount or range (such as 1 degree, 2 degrees, etc.). When all applicable gestures have been captured for a particular color channel, steps 1602 to 1607 are repeated for different color channels until all color channels have been captured, as shown in step 1608 .

[0150] Step 1609 may include integrating the captured images into an RGB dataset, which is then used to determine camera parameters and a distortion model. Fig.17 In the example of , process 1601 treats the head mounted display as a camera having a display lens and having a viewpoint at the eye viewpoint. Step 1610 may include extracting the captured data in a selected color channel.

[0151] Steps 1611 and 1612 include determining system parameters that represent the distortion of the combined system camera, here the perspective camera and the lens (e.g., Figure 1 1706 and 1707) and a display lens (e.g., Fig.17 1704) together with the calibration camera and lens are considered as elements of a single camera, assuming that a user viewing the content on the display 1705 of the head mounted device 1701 will see the content that has been modified by both components. For example, step 1611 may include determining a combined camera matrix, which may include contributions from the perspective camera and display lens of the system, and may also include contributions from the calibration camera that will be removed after processing 1630, as described below. The camera matrix (which may also be referred to as part of the "camera intrinsics") may be represented as:

[0152]

[0153] Where (f x , f y ) is the focal length of the camera, (c x , c y ) is the optical center, and s is the skewness coefficient. Therefore, Fig.16 Step 1611 of the exemplary method may include determining the matrix for the combined system components (eg, in this step (f x , f y ) is the focal length of the combined system components, (c x , c y ) are the optical centers of those components, etc.).

[0154] Step 1612 may include determining a distortion model for the camera lens assembly. The distortion model may include radial distortion and tangential distortion. For example, radial distortion may be expressed as:

[0155]

[0156] Where (x d ,y d ) is the distorted coordinate, (x, y) is the undistorted coordinate, (k 1 , k 2 , k 3 ) is the radial distortion coefficient, and r 2 =x 2 +y 2 . The tangential distortion can be expressed as:

[0157]

[0158] Where (x d ,y d ) is the distorted coordinate, (x, y) is the undistorted coordinate, (p 1 , p 2 ) is the tangential distortion coefficient.

[0159] The distortion coefficient C can be used as 1 k 2 p 1 p 2 k 3 ) to make the distortion model. Display calibration is used to determine the camera matrix K and distortion model C for each camera (i.e., for the perspective camera, for the calibration camera, for the "camera" representing the display lens, and for the "camera" representing the head mounted device, i.e., the combination of the perspective camera and the lens and the display lens). For example, Fig.16 The example calibration process 1601 determines K and C for the combined camera.

[0160] Fig.16 Step 1613 of the example method of 1601 determines whether steps 1610 to 1612 have been performed for all color channels. If so, the results are combined in step 1614 to obtain a combined camera matrix and distortion model for all color channels. These results can then be combined with the results of processes 1620 and 1630, as discussed below. Process 1601 can be performed for each camera-lens combination of the system, for example, as described below in conjunction with Fig.19 More fully described.

[0161] In order to fully calibrate the headset and determine the parameters necessary to correct the distortion introduced by the system, Fig.16 The example of obtains parameters (such as camera matrices and distortion models) for the perspective camera and the calibration camera. Process 1620 calibrates the perspective camera, while process 1630 calibrates the calibration camera, and in Fig.16 In the example above, the two processes are similar.

[0162] Process 1620 for calibrating a perspective camera may include step 1621, which may include rendering a calibration pattern in a selected color channel. Step 1622 may include displaying the rendered calibration pattern on a display (such as on a TV). The pose of the display is set in step 1623, and then in step 1624, the perspective camera captures and stores a picture of the display with the specific pose. Note that in this calibration step, the perspective camera is capturing the original display of the pattern, as opposed to process 1601, in which the calibration camera captures the display of the pattern on the display of the head mounted device.

[0163] Step 1625 is a determination of whether steps 1623 to 1625 have been completed for all desired display poses. Step 1626 may include determining parameters representing the distortion caused by the perspective camera, such as by determining the camera matrix for the perspective model and the distortion model for the camera in all selected color channels using equations 33-35 and the distortion model C for the perspective camera. Step 1627 determines whether steps 1621-1626 have been performed for all color channels, and if so, step 1628 may include creating a complete camera matrix and distortion model for the perspective camera based on the complete set of color channels.

[0164] Process 1630 is similar to process 1620, except that in process 1630 the calibration camera is capturing the calibration pattern shown on a display (e.g., TV), and the process provides parameters representing the distortion created by the calibration camera, e.g., the camera matrix and distortion model for that camera. Then, using the results of process 1630, the contribution of the calibration camera can be removed from step 1614 using the parameters representing the distortion created by the calibration camera to determine the distortion caused by the actual components of the system (e.g., perspective camera and display lens).

[0165] After completing processes 1601, 1620, and 1630, the resulting information can then be used in step 1615 to determine, for each side: (1) the head mounted device camera matrix, which represents the display lens and the combined effect of the perspective camera and lens without the contribution from the calibration camera, whose contribution is removed from the combined camera parameters obtained in step 1614 after process 1630; (2) the distortion model for the display lens; (3) the center of distortion; and (4) the field of view. This shows Fig.15A In a specific example of step 1515 of the example method of , this may include determining one or more display lens parameters representing the distortion caused by the display lens based on one or more system parameters and based on one or more camera parameters representing the distortion caused by the perspective camera. For example, the relationship between the distortion between the perspective camera, the display lens, and the head mounted device is:

[0166]

[0167] Where D 2 is the distortion of the display lens, D h is the distortion of the headset, and D 3 is the distortion of the perspective camera, and:

[0168]

[0169] And (k 1 , k 2 , k 3 ) is defined above with respect to Equation 34 (i.e., is a radial distortion coefficient specific to each optical component). The focal length of the display lens can be expressed as:

[0170]

[0171] where F 3 is the focal length matrix of the perspective camera, F h is the focal length matrix of the headset, and F 2 is the focal length matrix of the display lens, and:

[0172]

[0173] And (f x , f y ) as defined above with respect to Equation 3 (i.e., is the focal length specific to each component). Using the focal length matrix F of the perspective camera 3 and the focal length matrix of the head-mounted device, the focal length matrix of the display lens can be obtained.

[0174] Fig.18 An example of determining the field of view of a display lens and a perspective camera is shown. Fig.18 In the configuration, the distance between the eye viewpoint and the perspective camera viewpoint is d 1 And the minimum distance d from the perspective camera viewpoint 2 , at which the object can be viewed at the edge of the field of view (points a, b, c, d). Therefore, the field of view of the display lens can be calculated as follows:

[0175]

[0176] in is the field of view of the display lens after distortion correction, and is the field of view of the perspective camera after distortion correction.

[0177] Fig.15A Step 1520 of the example method may include storing one or more system parameters and one or more display lens parameters as calibration parameters for the video see-through AR system. In addition, the obtained parameters (such as camera matrix, distortion model, and Fig.16 Other calibration parameters resulting from the processing of the image) can be stored and used for future renderings. As an example, Fig.16 The processing is provided in Figure 3 Several parameters used in the example processing of , such as the camera matrix and distortion model of the perspective camera and display lens. In addition, for example, using Fig.17 Any AR rendering process for an example head mounted device requires certain parameters (such as the center of distortion).

[0178] As another example, the parameters may be used to correct for distortions (such as geometric distortion and chromatic aberration) caused by the head mounted device, such as Fig.16As shown in step 1616 of . Importantly, the calibration process results in parameters that can be used to correct both distortion in the display of the real world scene captured by the perspective camera and distortion in virtual objects rendered on the system display. From the viewer's perspective, the real world content is modified by both the perspective camera and the display lens, while the virtual content only passes through the display lens (i.e., the virtual content is not affected by the perspective camera because the virtual content is not captured by the camera). Therefore, corrections determined from both the perspective camera and the display lens can be made to the real content, while corrections based only on the display lens can be made to the virtual content. As a result, when the real and virtual content are mixed and rendered together on the system's display, different corrections are made to these different content types, so the user is able to see the undistorted real and virtual content together because the distortion correction addresses the related components that affect the different types of content.

[0179] In embodiments, a video see-through AR system may include multiple cameras for capturing real-world content. In embodiments, a system may include several cameras. Fig.19 An example system is shown that may include four cameras for capturing real-world content. Fig.19 As shown in the example of , the four system cameras result in four left system "cameras": (1) camera A1 combined with left lens Dleft (2) camera A2 combined with left lens Dleft (3) camera A3 combined with left lens Dleft and (4) camera A4 combined with left lens Dleft. Each of these cameras represents the system's effect on what the user sees on the left panel through the left lens, where the user's left eye is located at the position of the left virtual rendering camera Vleft. Similarly, Fig.19 The four system cameras in the example of yield four right system "cameras": (1) camera A1 in combination with right lens Dright (2) camera A2 in combination with right lens Dright (3) camera A3 in combination with right lens Dright and (4) camera A4 in combination with right lens Dright. Each of these cameras represents the system's effect on what the user sees on the right panel through the right lens, where the user's right eye is located at the position of the right virtual rendering camera Vright.

[0180] Including multiple cameras can provide the user with more perspectives of a scene and can increase the field of view of the system. For example, an object located near camera A1 can substantially block A1's view of the scene behind the object, but by including other cameras (e.g., A2 and A4), views of the scene behind the object can still be obtained and displayed by the system.

[0181] In an embodiment, a calibration process may be performed for each system "camera", such as Fig.15A Example processing and Fig.16 For example, Fig.16 In the example of processing 1601, the Fig.19 1615 and 1616 may be performed for each of the four left system cameras and each of the four right system cameras shown in the example system of , while process 1620 may be performed four times (i.e., for each of the four perspective cameras used in the system). In an embodiment, steps 1615 and 1616 may integrate the contributions of the system cameras of all corresponding sides to determine the overall parameters of that side. For example, in Fig.19 In the example of , the parameters (e.g., distortion models) of each of the four left system cameras can be combined to determine the parameters of the entire left "camera" of the system, and similarly for the right side. Thus, each camera-display lens pair contributes to the overall system parameters (e.g., distortion correction, distortion center, field of view, etc.) of the side corresponding to that display lens.

[0182] Although Fig.19 The example of shows the use of several perspective cameras, but the present disclosure contemplates that the system may include other types of cameras, such as tracking cameras, etc. These cameras may also undergo a calibration process, such as Fig.16 processing.

[0183] Although Fig.16 The example uses a polynomial function to model the distortion of the lens and determines the coefficients in the polynomial function through a calibration process to obtain the display lens distortion, but an optional embodiment uses a calibration process to determine the distortion of each pixel and creates a lookup table using the determined distortion of each pixel. A lookup table can be created for each color channel. During distortion correction, such an embodiment applies the distortion lookup table to the corresponding color channel, thereby correcting both the geometric distortion and chromatic aberration of the lens. The distortion center on the left and right sides can be obtained by finding the pixel with zero distortion in the lens using the lookup table. Within the lookup table, a calibrated horizontal field of view can be obtained by identifying the leftmost pixel in the scene in the lens and the rightmost pixel in the scene, and a calibrated vertical field of view can be obtained by identifying the topmost pixel in the scene in the lens and the bottommost pixel in the scene.

[0184] Although Equation 34 models the radial distortion of the lens with a polynomial function having three coefficients (k1, k2, k3), an optional embodiment adaptively changes the number of coefficients n. For example, a function can be used to detect how much distortion the lens creates. If there is no distortion, no distortion correction is needed, and n=0. If there is a small distortion, an embodiment can use one or two coefficients, n=1 or 2. If there is a large distortion, more coefficients can be used so that n>=3. In addition, although the radial distortion function is used to model the symmetrical distortion of the lens, an optional embodiment can perform asymmetrical distortion calibration by applying other asymmetric functions (such as spline functions or some non-continuous functions) according to the properties of the lens that needs to be calibrated instead of using a radial distortion function.

[0185] Fig. 15B Step 1550 of the example method may include accessing a real-world image captured by a perspective camera 1706 of the video perspective AR system 1700 .

[0186] Fig. 15B Step 1555 of the example method may include rendering one or more virtual objects for display within the real-world scene.

[0187] Fig. 15B Step 1560 of the example method may include distorting the real-world image according to one or more predetermined system parameters for the video see-through AR system 1700 .

[0188] Fig. 15B Step 1565 of the example method may include distorting one or more virtual objects according to one or more predetermined display lens parameters of the display lens 1704 for the video see-through AR system 1700.

[0189] Fig. 15B Step 1570 of the example method may include displaying a distorted real-world image and a distorted mixed image of one or more virtual objects on a display panel 1705 of the video see-through AR system 170 .

[0190] In the embodiment, according to Fig.15A The calibration process of the example method determines predetermined system parameters and predetermined display lens parameters.

[0191] Where appropriate, the examples may be repeated Fig.15A and Fig. 15B Although the present disclosure describes and illustrates one or more steps of the method. Fig.15A and Fig. 15B The specific steps of the method occur in a specific order, but the present disclosure contemplates Fig.15A and Fig. 15B Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates the execution Fig.15A and Fig. 15B Specific components, devices or systems (such as Fig.26 ), but the present disclosure contemplates executing Fig.15A and Fig. 15B In addition, the present disclosure contemplates some or all of the computing operations described herein (including Fig.15A and Fig. 15B The steps of the example method shown in FIG. 1 may be performed by a computing device (e.g. Fig.26The invention may be executed by circuits of a computing device), by a processor coupled to a memory, or by any suitable combination thereof.

[0192] In a video see-through AR system, rendered virtual objects may not match their intended placement within a real image of a real scene. For example, the rendered object may be in the wrong location, have the wrong orientation, have the wrong size, and / or may have incorrect shading or geometry given the geometric distortion and chromatic aberration of the system, etc., and these types of errors may be referred to as misregistration of the virtual object. Fig. 20 Examples of these kinds of errors are shown. For example, assume that a real object (such as a rabbit 2004 or a turkey 2007) is captured by a video perspective camera and displayed on a system display having a 2D image plane 2002, as shown in FIG. Fig. 20 2000 (actually, the objects will be rendered on the left and right displays separately, although for convenience, the Fig. 20 Only one display is shown in the example of ). The system attempts to render virtual objects 2005 and 2006 that overlap identically with the corresponding real objects, but due to misregistration, the displayed virtual objects do not overlap with the real objects. Scene 2050 shows an example of errors caused by misregistration, including angular errors, lateral errors, depth errors, and linear errors. Misregistration can be complicated because the object or the video see-through AR system or both may be moving, and delays in video processing and object rendering may cause the rendered objects to be misplaced within a dynamic scene.

[0193] In order to correctly register rendered virtual objects and correct errors caused by misregistration, embodiments of the present disclosure calibrate the rendering process by creating a virtual rendering camera with a corresponding camera matrix and distortion model. Fig. 20 As shown in the example of , a virtual camera 2001 is placed at the eye position for viewing content on a display. Separate virtual cameras are used for the left eye / left display and the right eye / right display. The virtual camera can be viewed as rendering the real world information captured by the perspective camera and the virtual objects generated by the graphics pipeline.

[0194] Fig.21 An example method for virtual object registration is shown. Fig. 22 Shows the implementation Fig.21 An example of the method is processed, and Fig.23 An example architecture and coordinate system for performing dynamic object registration is shown. Fig.21The example method treats the rendering process as using a pair of virtual cameras or rendering cameras to perform object registration. As described below, the method iteratively adjusts the virtual camera parameters, if necessary, until the virtual object misregistration is below a threshold level. Among other things, the method corrects errors in the head-mounted device as a whole. For example, during any given manufacturing and installation of components (such as lens assemblies, camera assemblies, etc.) of a video perspective AR system, the actual parameters of the components may be different from the expected parameters. For example, stereo cameras may not be completely aligned with each other, or the distance between such cameras may not be the same as the expected or planned distance, or the camera may be rotated from the expected orientation, or one or more lenses may have optical properties that are different from the planned or expected or may be tilted from the expected orientation, or the good viewing distance may be different from the plan, or the intrinsic properties (e.g., camera matrix, focal length, etc.) may not be exactly as planned. Any and all of these errors, as well as other manufacturing and installation errors when assembling system instances, may result in object misregistration. Fig.21 The example method of addresses these errors and provides corrections for these system-level errors so that displayed virtual objects are correctly aligned. As described more fully herein, the calibration process determines virtual camera parameters that are specific to the particular instance of the system (e.g., head mounted device) undergoing the calibration process.

[0195] Fig.21 Step 2105 of an example method of may include capturing stereo images of a predetermined calibration object having a predetermined pose using a pair of cameras of a video see-through AR system. The predetermined object may be any suitable object having a known shape, size, and orientation (or pose). In embodiments, a user may select an object (e.g., a cube) and enter values ​​for these parameters, or the object may be designed for use with a particular system. For example, the object may be included in the system before the system is provided to the user, or may be performed, for example, by a manufacturer using a specific object. Fig.21 Furthermore, although this example describes the use of a single predetermined calibration object, the present disclosure contemplates Fig.21 and / or Fig. 22 The process may use multiple calibration objects, each calibration object having known object properties.

[0196] Fig. 22 An example of capturing a stereoscopic image using a left ultra-wide field of view camera 2201 and a right ultra-wide field of view camera 2202 in step 2204 is shown. Fig.21 Step 2110 of the example method may include obtaining position data of each camera in the pair of cameras via a position sensor when the stereoscopic image is captured. For example, Fig. 22An IMU position sensor is shown capturing position data for the camera (e.g., by capturing position data for a system that holds the camera) in step 2205. This disclosure contemplates that any suitable position sensor or any number of position sensors may be used to capture position data.

[0197] Fig.21 Step 2115 of the example method may include generating a 3D reconstruction of the calibration object based on the stereo image and the position data. For example, Fig. 22 It is shown that the captured images can be corrected, for example, by using camera parameters 206 (such as camera matrices, focal lengths, and distortion models) for each of the left and right cameras. For example, the correction process using the camera model discussed above can be used to correct the left and right images. The camera parameters together with the image data and the position data can be used to perform camera pose tracking 2208, for example, by tracking the camera pose in 6 degrees of freedom (6DoF). In step 2209, the calibration objects are detected and extracted from the corrected stereo image pair, and in step 2210, a depth map of the calibration objects is reconstructed. In step 2212, the 3D shape of the (one or more) calibration objects is reconstructed using the depth map and camera pose tracking. In an embodiment, a volume-based 3D reconstruction (such as TSDF) can be used to perform 3D reconstruction. An optional embodiment can alternatively use point cloud technology, and after capturing images and reconstructing depth maps of these calibration objects, a point cloud of these objects is created in 3D space. 3D objects are detected and extracted from the point cloud, and the position and orientation of the extracted objects are calculated.

[0198] Fig.21 Step 2120 of the example method may include performing a virtual object registration process including steps 2125 and 2130. Fig.21 Step 2125 of the example method may include generating a stereoscopic virtual view including a predetermined calibration object based on the captured stereo images, the position data, and the 3D reconstruction using parameters of each of the pair of stereoscopic virtual cameras, wherein each virtual camera in the pair of stereoscopic virtual cameras is located at an eye position for viewing content on a video see-through AR system. For example, Fig. 22 It is shown that virtual camera parameters 2211 (such as camera matrix, field of view, distortion model, etc.) are used to generate left and right virtual views using captured stereo images, 3D reconstruction, and camera poses. Each virtual camera (e.g., left and right virtual cameras) has its own set of associated virtual camera parameters. As shown, in e.g. Fig.23 , each virtual camera 2316 is located at an eye position for viewing content on the system 2300 , that is, at the position where the user's eyes (left eye and right eye, respectively) should be when viewing content on the system 2300 .

[0199] like Fig. 22 As shown, in step 2220, the graphics pipeline may take a known pose of a predetermined calibration object and generate a virtual calibration object in step 2221. The known parameters of the predetermined calibration object may be used to generate a virtual object, such as shape, position, etc., which may be reconstructed from the depth data and camera position data discussed above. Then, as described, for example, with respect to Fig.25 As illustrated, the generated virtual objects and virtual views may be blended and rendered for display on the system, ie, a left blended image is displayed on a left display and a right blended image is displayed on a right display.

[0200] Fig.21 Step 2130 of the example method may include determining, for each of the stereoscopic virtual views, one or more differences between a real-world calibration object corresponding to a predetermined calibration object in the stereoscopic image and the virtual rendering of the calibration object. For example, once the hybrid view is displayed on a display, Fig. 22 It is shown that a match between the rendered real world calibration object and the rendered virtual calibration object occurs in step 2217, and in step 2218, a difference between the two is determined. Since the images are stereo images, this determination is effectively made in 3D space. The difference can be any suitable difference, such as a difference in size, shape, orientation (e.g., rotation), color, and / or amount of object distortion.

[0201] Fig.21 Step 2135 of the example method may include adjusting one or more parameters of at least one of the pair of stereoscopic virtual cameras and re-performing the virtual object registration process when at least one of the determined one or more differences is greater than a corresponding threshold. For example, each type of difference may be associated with a specific threshold, such as a specific translation amount, a specific rotation amount, a specific distortion amount, etc. between the two objects. In an embodiment, the differences relative to the predetermined threshold may be considered as a whole, for example, if the two objects differ in only one aspect, the allowable threshold may be higher, but if the objects differ significantly in two aspects, the corresponding two thresholds may be lower than the thresholds when they are considered separately. Fig. 22 A decision 2219 is shown to determine whether each difference is less than a threshold, and if not, the parameters of the virtual camera are updated in step 2214, and steps 2213, 2217, 2218, and 2219 are repeated using the updated virtual camera parameters. The virtual camera parameters may be updated by, for example: adjusting the distortion coefficient to adjust the object shape; adjusting the focal length to adjust the scale, and adjusting the principal point to shift the object.

[0202] Fig.21Step 2140 of the example method may include storing the parameters of each of the pair of stereoscopic virtual cameras in association with the video see-through AR system when one or more differences are not greater than the corresponding threshold. For example, Fig. 22 The camera parameters are shown to be stored as registration parameters in step 2223, and these parameters may subsequently be used to blend the real world and virtual content in step 2222, and render the content to a display panel in step 2224. The parameters may be stored in any suitable location, such as on the system, on a device associated with the system, on a server device, etc. As explained above, these parameters are specific to the system being calibrated because they are based on correcting for specific errors present in the system.

[0203] In an embodiment, each time the system undergoes a physical change, Fig.21 Calibration processing such as Fig. 22 For example, if a user sets down their system, the calibration process may be repeated, for example, to account for changes in the relative positions of system components.

[0204] Where appropriate, the examples may be repeated Fig.21 Although the present disclosure describes and illustrates one or more steps of the method. Fig.21 The specific steps of the method occur in a specific order, but the present disclosure contemplates Fig.21 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates specific components, devices, or systems for performing Fig.21 Specific steps of the method, such as Fig.26 However, the present disclosure contemplates any suitable combination of any suitable components, devices, or systems to perform Fig.21 In addition, the present disclosure contemplates some or all of the computing operations described herein (including Fig.21 The steps of the example method shown in FIG. 1 may be performed by a computing device (e.g. Fig.26 The invention may be executed by circuits of a computing device), by a processor coupled to a memory, or by any suitable combination thereof.

[0205] In an embodiment, the registration calibration process may also process depth values. Fig. 22 An example process for calibrating depth values ​​is shown. The depth map obtained in step 2210 is matched with the virtual depth information of the virtual calibration object generated during steps 2220 and 2221. In step 2215, the depth information of the rendered real calibration object and the rendered virtual calibration object are aligned using a depth scale factor, and the depth scale factor is adjusted in step 2216 until the depth difference is less than a corresponding threshold. Fig. 22In the example of , the depth scale factor is stored together with the registration / virtual camera parameters.

[0206] Although Fig.21 and Fig. 22 Example methods of relate to video perspective AR, but these methods can be extended to optical perspective AR. Before the registration process, a scene is created by placing (one or more) calibration objects in a designed predetermined manner. In the designed scene configuration, the poses of these calibration objects are known. The calibration objects are rendered to the optical perspective AR device by the graphics pipeline. The position and orientation of these calibration objects in 3D space are extracted by the optical perspective AR device. After comparing the poses of real-world objects and virtual objects, the optical perspective system can align the virtual objects with the real-world objects to calibrate the virtual camera for the optical perspective system.

[0207] Fig.23 An example coordinate system used during dynamic object registration is shown, such as Fig.21 and Fig. 22 In FIG2 , a world coordinate system 2301 is defined as a global coordinate system for measuring all activities of dynamic object registration. A camera tracking coordinate system 2302 is defined for camera tracking to obtain 6DoF camera poses. Fig.23 In the example of , this coordinate system is attached to the left camera of the stereo camera pair. The HMD (head mounted display) marker coordinate system 2303 is defined to measure head movement to obtain the 6DoF head pose. It is attached to the HMD device. The perspective camera coordinate system 2304 is defined to measure the perspective camera movement to obtain the 6DoF pose of the perspective camera 2312. Fig.23 In the example of , the coordinate system is attached to the left perspective camera. If there are multiple perspective cameras, the coordinate system is attached to one of them. The virtual camera coordinate system 2305 is defined for the virtual rendering camera to render the virtual view to the head mounted device panel. The virtual camera viewpoint is configured at the eye viewpoint determined by the head mounted device parameters (e.g., good viewing distance). The camera tracking coordinate system 2302, the HMD marker coordinate system 2303, and the perspective camera coordinate system 2304 are all rigidly connected. Therefore, once the movement is known in a coordinate system such as the camera tracking coordinate system 2302, the movement can be determined in any other rigidly connected coordinate system.

[0208] Fig.24 An example of video transformation in dynamic object registration is shown. The goal is to transfer a perspective camera to a virtual camera in a world coordinate system so that an embodiment can register real world objects captured by the perspective camera with virtual objects created by a computer (e.g., by a GPU) and rendered by the virtual camera. The example uses HMD markers to associate the HMD and all elements rigidly connected to the HMD to the world coordinate system.

[0209] P M is the posture of the HMD marker 2313 and can be expressed as:

[0210] P M =[R M |T M ] (41)

[0211] Where R M is the rotation matrix of the HMD marker’s pose, and T M is the translation vector of the HMD marker’s pose. M is a 4×4 homogeneous transformation matrix that maps the HMD marker coordinate system 2403 of the HMD marker to the perspective coordinate system 2405. P can be measured on the rigid connection between the HMD marker 2313 and the perspective camera 2312. M In addition, the transformation P from the perspective camera 2312 to the HMD marker 2313 SM It can be defined as follows:

[0212] P SM =R SM [I]-T SM (42)

[0213] Where R SM is the rotation matrix between the perspective camera 2312 and the virtual camera, and T SM is the translation vector between the perspective camera 2312 and the virtual camera, as described below.

[0214] P SM is the 3×4 projection matrix of the perspective camera transform 2312 that defines the coordinates relative to the HMD markers 2313. For a given system (such as a head mounted device), the distance between the perspective camera 2312 and the center of the virtual camera 2316 is measured to obtain the translation vector T SM , and the angular distance between the perspective camera 2312 and the center of the virtual camera 2316 is measured to calculate the rotation matrix R SM Therefore, the overall transformation matrix P that maps the virtual camera center to the world coordinate system is SV Can be defined as:

[0215] P SV =P SM P M P H (43)

[0216] Where P SV is a 3×4 projection matrix that transforms world coordinates into virtual camera coordinates, and P H is a 4×4 homogeneous transformation matrix that maps world coordinates to perspective camera coordinates.

[0217] In addition, the transformation P from the pose tracking camera 2310 to the HMD marker 2313 TM Defined by:

[0218] P TM =R TM [I]-T TM (44)

[0219] Where R TM is the 3×4 projection matrix of the pose tracking camera transform 2310 that defines the coordinates relative to the HMD markers 2313. For a given head mounted device, the distance between the pose tracking camera 2312 and the center of the virtual camera 2316 can be measured to obtain the translation vector T TM , and measure the angle between the posture tracking camera 2310 and the virtual camera 2316 to calculate the rotation matrix R TM Therefore, the overall transformation matrix P that maps the virtual camera center to the world coordinate system is TV Can be defined as:

[0220] P TV =P TM P M1 P H (45)

[0221] Where P TV is a 3×4 projection matrix that transforms world coordinates into virtual camera coordinates, and P H is a 4×4 homogeneous transformation matrix that maps world coordinates to the tracking coordinates of the pose tracking camera.

[0222] P M1 is a 4×4 homogeneous transformation matrix that maps the HMD marker coordinate system 2403 of the HMD marker to the pose tracking camera coordinate system 2402. P can be measured on the rigid connection between the HMD marker 2313 and the pose tracking camera 2310. M In addition, the transformation P from the pose tracking camera 2310 to the perspective camera 2312 TS It can be defined by the following formula:

[0223] P TS =R TS [I]-T TS (46)

[0224] Where R TS is the rotation matrix between the pose tracking camera 2310 and the perspective camera 2312, and T TS is the translation vector between the pose tracking camera 2310 and the perspective camera 2312. TSis the 3×4 projection matrix that defines the pose tracking camera transform 2310 relative to the coordinates of the perspective camera 2312. The matrix P TS is the desired projection matrix.

[0225] For a given head mounted device, the distance between the center of the pose tracking camera 2312 and the perspective camera 2312 can be measured to obtain the translation vector T TS And measure the angle between the posture tracking camera 2310 and the perspective camera 2312 to calculate the rotation matrix R TS . We can transform the coordinates from the pose tracking camera coordinate system 2402 to the perspective coordinate system 2405.

[0226] Fig.25 An example algorithm for dynamic object registration is shown. The algorithm may be, for example, Fig.21 or 22. Example method uses. Fig.25 In FIG. 1 , a calibration object (or objects in an embodiment) 2511 is captured by a video perspective camera 2510, and a depth map 2512 of the calibration markers and objects is reconstructed from the images. The 6DoF camera pose is calculated using the image sequence. The 3D shapes 2530 of the calibration markers and objects are reconstructed using the images, depth map, and camera pose. Virtual object images 2521 and depth maps 2522 of the calibration markers and objects are generated by the graphics pipeline 2520. The virtual calibration markers and objects are generated to overlap with the real world calibration markers and objects when viewed from the HMD, as described above with reference to FIG. Fig. 22 discussed.

[0227] The real world calibration markers and objects 2530 reconstructed from the perspective camera and the virtual calibration object(s) are blended together at step 2532. In step 2534, the blended information is rendered and displayed to the HMD panel using virtual camera parameters 2536, which will be discussed more fully herein.

[0228] Fig.25 An example of using an HMD to extract the position and orientation of virtual calibration markers and objects in 3D space 2535:

[0229] P v =[R v |T v ] (47)

[0230] Where P v is the pose of the virtual object, R v is the rotation matrix that identifies the rotation of the virtual object, and T v is the translation vector that identifies the translation of the virtual object. v , R v and T vEach of is generated when the virtual object is generated. The position and orientation 2531 of the real-world calibration markers and objects are calculated from the 3D object reconstruction 2530:

[0231] P r =[R r |T r ]) (48)

[0232] Where P r is the pose of the real-world object, R r is the rotation matrix, and T r is the translation vector.

[0233] Fig.25 The example compares and extracts the differences in position, orientation, etc. between real-world objects and virtual objects 2533. This is done for all real-world objects and all virtual objects 2538. The extracted differences 2533 can then be used to determine parameters of the virtual camera 2539, such as by updating the existing virtual camera parameters with the new calibrated virtual camera parameters (e.g., based on Fig.16 Example process for generating a virtual camera matrix and distortion model during head mounted device calibration).

[0234] If the difference between the real-world calibration markers and objects captured from the perspective camera and the virtual calibration markers and objects generated by the graphics pipeline is small enough 2540 (e.g., less than a corresponding threshold), the algorithm outputs the calibration parameters of the virtual camera. Otherwise, the algorithm replaces one or more current parameters 2537 of the virtual camera with the calibration parameters 2539 of the camera model 2536, and repeats the alignment steps 2534, 2535, 2533, 2538, 2539 and 2540 until the difference between the real-world calibration markers and objects captured from the perspective camera and the virtual calibration markers and objects generated by the graphics pipeline is small enough (e.g., less than a corresponding threshold), and then saves the calibration result.

[0235] In an embodiment, Fig.25 The example can be extended to the depth calibration of the system. For example, the marker can be placed in the real 3D space. The depth of the marker is known. According to Fig.25 The depth of the virtual markers is then extracted and compared to the depth of the real-world markers to calibrate the depth rendering.

[0236] Fig.26An example computer system 2600 is shown. In an embodiment, one or more computer systems 2600 perform one or more steps of one or more methods described or illustrated herein. In an embodiment, one or more computer systems 2600 provide functionality described or illustrated herein. In an embodiment, software running on one or more computer systems 2600 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. An embodiment may include one or more portions of one or more computer systems 2600. Herein, reference to a computer system may include a computing device and vice versa, where appropriate. Furthermore, reference to a computer system may include one or more computer systems, where appropriate.

[0237] The present disclosure contemplates any suitable number of computer systems 2600. The present disclosure contemplates computer systems 2600 in any suitable physical form. As an example and not by way of limitation, computer system 2600 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on module (COM) or a system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive computer kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 2600 may: include one or more computer systems 2600; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 2600 may perform one or more steps of one or more methods described or shown herein without substantial spatial or temporal limitations. As an example and not by way of limitation, one or more computer systems 2600 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 2600 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein.

[0238] In an embodiment, computer system 2600 may include processor 2602, memory 2604, storage 2606, input / output (I / O) interface 2608, communication interface 2610, and bus 2612. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0239] In an embodiment, the processor 2602 may include hardware for executing instructions (such as those constituting a computer program). As an example and not by way of limitation, to execute instructions, the processor 2602 may: retrieve (or fetch) instructions from an internal register, an internal cache, memory 2604, or storage 2606; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, memory 2604, or storage 2606. In an embodiment, the processor 2602 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 2602 that includes any suitable number of any suitable internal caches. As an example and not by way of limitation, the processor 2602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 2604 or storage 2606, and the instruction cache may speed up the retrieval of those instructions by the processor 2602. The data in the data cache may be: a copy of the data in the memory 2604 or storage device 2606 for the instruction operation executed at the processor 2602; the result of the previous instruction executed at the processor 2602 for access by the subsequent instruction executed at the processor 2602 or for writing to the memory 2604 or storage device 2606; or other suitable data. The data cache may speed up the read or write operation by the processor 2602. The TLB may speed up the virtual address translation of the processor 2602. In an embodiment, the processor 2602 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 2602 including any suitable number of any suitable internal registers. Where appropriate, the processor 2602 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 2602. Although the present disclosure describes and illustrates a specific processor, the present disclosure contemplates any suitable processor.

[0240] In an embodiment, the memory 2604 may include a main memory for storing instructions for execution by the processor 2602 or data for operation by the processor 2602. As an example and not by way of limitation, the computer system 2600 may load instructions from the storage device 2606 or another source (such as, for example, another computer system 2600) to the memory 2604. The processor 2602 may then load the instructions from the memory 2604 to an internal register or an internal cache. To execute the instructions, the processor 2602 may retrieve the instructions from the internal register or the internal cache and decode them. During or after execution of the instructions, the processor 2602 may write one or more results (which may be intermediate results or final results) to the internal register or the internal cache. The processor 2602 may then write one or more of those results to the memory 2604. In an embodiment, the processor 2602 executes only instructions in one or more internal registers or internal caches or in memory 2604 (opposite to storage device 2606 or elsewhere), and operates only on data in one or more internal registers or internal caches or in memory 2604 (opposite to storage device 2606 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 2602 to the memory 2604. Bus 2612 may include one or more memory buses, as described below. In an embodiment, one or more memory management units (MMUs) reside between the processor 2602 and the memory 2604 and facilitate access to the memory 2604 requested by the processor 2602. In an embodiment, the memory 2604 may include a random access memory (RAM). Where appropriate, the RAM may be a volatile memory, where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 2604 may include one or more memories 2604. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0241] In an embodiment, the storage device 2606 may include a mass storage device for data or instructions. As an example and not by way of limitation, the storage device 2606 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the storage device 2606 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 2606 may be inside or outside the computer system 2600. In an embodiment, the storage device 2606 is a non-volatile solid-state memory. In an embodiment, the storage device 2606 may include a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a mass storage device 2606 in any suitable physical form. Where appropriate, storage 2606 may include one or more storage control units that facilitate communication between processor 2602 and storage 2606. Where appropriate, storage 2606 may include one or more storage 2606. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

[0242] In an embodiment, the I / O interface 2608 may include hardware, software, or both of one or more interfaces for communication between the computer system 2600 and one or more I / O devices. Where appropriate, the computer system 2600 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and the computer system 2600. As an example and not by way of limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more of these. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 2608 for them. Where appropriate, the I / O interface 2608 may include one or more devices or software drivers so that the processor 2602 can drive one or more of these I / O devices. Where appropriate, the I / O interface 2608 may include one or more I / O interfaces 2608. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0243] In an embodiment, the communication interface 2610 may include hardware, software, or both that provide one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 2600 and one or more other computer systems 2600 or one or more networks. As an example and not by way of limitation, the communication interface 2610 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 2610 for it. As an example and not by way of limitation, the computer system 2600 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 2600 may communicate with a wireless PAN (WPAN) (such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, the computer system 2600 may include any suitable communication interface 2610 for any of these networks. Where appropriate, the communication interface 2610 may include one or more communication interfaces 2610. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0244] In an embodiment, bus 2612 may include hardware, software, or both that couple components of computer system 2600 to each other. As an example and not by way of limitation, bus 2612 may include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnect, industry standard architecture (ISA) bus, infinite bandwidth interconnect, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCIe) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus. Or another suitable bus or a combination of two or more of these. Where appropriate, bus 2612 may include one or more buses 2612. Although the present disclosure describes and illustrates a specific bus, the present disclosure contemplates any suitable bus or interconnect.

[0245] In an embodiment of the present disclosure, a method for object registration in a video perspective augmented reality (AR) system includes capturing a stereoscopic image of a predetermined calibration object with a predetermined pose using a pair of cameras of the video perspective AR system. The method includes obtaining position data of each camera in the pair of cameras by a position sensor when the stereoscopic image is captured. The method generates a 3D reconstruction of the predetermined calibration object based on the stereoscopic image and the position data. The method includes performing a virtual object registration process. The virtual object registration process includes using parameters of each of the paired stereo virtual cameras to generate a stereoscopic virtual view including the predetermined calibration object based on the stereoscopic image, the position data and the 3D reconstruction, wherein each virtual camera in the paired stereo virtual camera is located at an eye position for viewing content on the video perspective AR system. The virtual object registration process includes determining one or more differences between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicating a virtual rendering of the predetermined calibration object for each of the stereoscopic virtual views. The virtual object registration process includes, when at least one of the one or more differences determined is greater than a corresponding threshold, adjusting one or more parameters of at least one of the paired stereo virtual cameras and re-performing the virtual object registration process. The virtual object registration process includes: when one or more differences are not greater than a corresponding threshold, storing parameters of each of the pair of stereo virtual cameras in association with the video see-through AR system.

[0246] In an embodiment, the method comprises determining depth information of a virtual calibration object based on a stereo image and a depth scale factor. The method comprises determining a difference between the depth information of the virtual calibration object and known depth information of a predetermined calibration object. The method comprises adjusting the depth scale factor based on the difference.

[0247] In an embodiment, the position sensor comprises an inertial measurement unit.

[0248] In an embodiment, the parameters of each of the pair of stereo virtual cameras include a camera matrix for each virtual camera and a distortion model for each of the pair of stereo virtual cameras.

[0249] In an embodiment, the one or more differences include a positional difference.

[0250] In an embodiment, the one or more differences include an orientation difference.

[0251] In an embodiment, the one or more differences include a distortion difference.

[0252] Herein, where appropriate, a computer-readable storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable storage medium, or any suitable combination of two or more of these. Where appropriate, a computer-readable storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0253] As used herein, unless expressly indicated otherwise or indicated by context, "or" is inclusive and not exclusive. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0254] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that would be understood by a person of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or illustrated herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or arrangement of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person of ordinary skill in the art.

Claims

1. A method for object registration in a video perspective augmented reality (AR) system, comprising: Capturing a stereoscopic image of a predetermined calibration object having a predetermined pose using a pair of cameras of the video see-through AR system 2105; When the stereoscopic image is captured, position data for each camera in the pair of cameras is obtained by a position sensor 2110; generating a 3D reconstruction of the predetermined calibration object based on the stereo image and the position data 2115; Perform virtual object registration processing 2120, including: generating a stereoscopic virtual view including the predetermined calibration object based on the stereo images, the position data, and the 3D reconstruction using parameters of each virtual camera in a pair of stereoscopic virtual cameras, wherein each virtual camera in the pair of stereoscopic virtual cameras is located at an eye position for viewing content on the video see-through AR system; for each of the stereoscopic virtual views, determining one or more differences 2130 between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicative of a virtual rendering of the predetermined calibration object; When at least one of the one or more determined differences is greater than a corresponding threshold, adjusting one or more parameters of at least one virtual camera in the pair of stereoscopic virtual cameras and re-performing the virtual object registration process 2135; and When the one or more differences are not greater than the corresponding threshold, the parameters of each virtual camera in the pair of stereoscopic virtual cameras are stored 2140 in association with the video see-through AR system.

2. The method according to claim 1, further comprising: determining depth information of the virtual calibration object based on the stereo image and the depth scale factor; determining a difference between depth information of a virtual calibration object and known depth information of said predetermined calibration object; as well as A depth scale factor is adjusted based on the difference.

3. The method according to any one of claims 1 and 2, wherein: The position sensor includes an inertial measurement unit.

4. The method according to any one of claims 1 to 3, wherein: The parameters of each virtual camera in the pair of stereo virtual cameras include a camera matrix for each virtual camera and a distortion model for each virtual camera in the pair of stereo virtual cameras.

5. The method according to any one of claims 1 to 4, wherein: The one or more differences include a positional difference.

6. The method according to any one of claims 1 to 5, wherein: The one or more differences include an orientation difference.

7. The method according to any one of claims 1 to 6, wherein: The one or more differences include a distortion difference.

8. One or more computer-readable storage media containing instructions and coupled to one or more processors, the one or more processors being operable to perform the method according to any one of claims 1 to 7.

9. A system 2600, comprising: One or more memories 2604 for storing instructions; as well as One or more processors 2602, coupled to the one or more memories 2604, the one or more processors 2602 are operable to execute instructions to perform the following operations: capturing, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose; obtaining position data of each of the pair of cameras by a position sensor while the stereoscopic images are being captured; generating a 3D reconstruction of the predetermined calibration object based on the stereo image and the position data; Perform virtual object registration processing, including: generating a stereoscopic virtual view including the predetermined calibration object based on the stereo images, the position data, and the 3D reconstruction using parameters of each virtual camera in a pair of stereoscopic virtual cameras, wherein each virtual camera in the pair of stereoscopic virtual cameras is located at an eye position for viewing content on a video see-through AR system; determining, for each of the stereoscopic virtual views, one or more differences between a real-world calibration object corresponding to the predetermined calibration object in the stereoscopic image and a virtual calibration object indicative of a virtual rendering of the predetermined calibration object; When at least one of the one or more determined differences is greater than a corresponding threshold, adjusting one or more parameters of at least one virtual camera in the pair of stereoscopic virtual cameras and re-performing the virtual object registration process; and When the one or more differences are not greater than the corresponding threshold, the parameters of each virtual camera in the pair of stereoscopic virtual cameras are stored in association with the video see-through AR system.

10. The system according to claim 9, wherein: The one or more memories 2604 also store instructions that the one or more processors are operable to execute to perform the following operations: determining depth information of the virtual calibration object based on the stereo image and the depth scale factor; determining a difference between depth information of the virtual object and known depth information of the predetermined calibration object; as well as A depth scale factor is adjusted based on the difference.

11. The system according to any one of claims 9 and 10, wherein: The position sensor includes an inertial measurement unit.

12. A system according to any one of claims 9 to 11, wherein: The parameters of each virtual camera in the pair of stereo virtual cameras include a camera matrix for each virtual camera and a distortion model for each virtual camera in the pair of stereo virtual cameras.

13. A system according to any one of claims 9 to 12, wherein: The one or more differences include a positional difference.

14. A system according to any one of claims 9 to 13, wherein: The one or more differences include an orientation difference.

15. A system according to any one of claims 9 to 14, wherein: The one or more differences include a distortion difference.

Images