Segmented progressive and continuous calibration with coherent context

Abstract

Display of virtual content is improved using a piecewise progressive continuous calibration method with contextual coherence. When a set of frames is rendered to depict a virtual image, a VAR system can identify the location of the virtual content in the frames. The system can convolve a test pattern at the location of the virtual content to generate a calibration frame. The calibration frame is inserted within the set of frames in a manner that is imperceptible to the user.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 027,677, filed May 20, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This manual mainly covers image processing and display calibration. Background Technology

[0004] Display devices project images onto a display interface. However, if there are problems with the monitor or the display interface, the image may not appear as expected. For example, if the display interface is distorted or misaligned, the image may not appear as intended. Summary of the Invention

[0005] The innovative aspects of the subject matter described in this specification relate to the calibration of display devices used in virtual or augmented reality (VAR) systems. In particular, VAR systems can be used to display virtual content to enhance a view of physical reality. Calibration may be necessary to ensure that virtual content is displayed correctly when one or more display-related components of a VAR system are deformed or not functioning as expected.

[0006] According to the described implementation, a piecewise progressive continuous calibration method with contextual coherence is used to improve the display of virtual content. A set of frames is rendered to depict the virtual image. The VAR system can identify the position of the virtual content within this set of frames. The system can convolve a test pattern at the position of the virtual content to generate a calibration frame. This calibration frame is inserted into the set of frames in a way that is imperceptible to the user because the exposure of the calibration frame is used for such a short time.

[0007] The described calibration technique is advantageous because it calibrates the display only at the time and place when virtual content is displayed to the user. This calibration method performs exceptionally well if the virtual content has strong contextual coherence and spatial distribution. Other benefits include low computational burden, low dependence on eye tracking, and low risk of degradation. These advantages allow the calibration technique to perform well in high-frequency displays.

[0008] Other embodiments of this aspect include computer programs recorded on corresponding systems, apparatuses, and computer storage devices, each configured to perform the operation of the method.

[0009] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0010] Figure 1 An example implementation of a virtual or augmented reality (VAR) system is described.

[0011] Figure 2A A plan view depicting the aligned left and right eyepieces is shown.

[0012] Figure 2B A plan view depicting the misaligned left and right eyepieces is shown.

[0013] Figure 3A A top view depicting the frame structure with aligned left and right eyepieces is shown.

[0014] Figure 3B A top view depicting a frame structure with misaligned left and right eyepieces is shown.

[0015] Figure 4 A series of frames, including example calibration frames, are depicted.

[0016] Figure 5 An illustration depicts the calibration frame.

[0017] Figure 6 A series of frames, including calibration frames with test patterns and virtual content, are depicted.

[0018] Figure 7 A flowchart is depicted for a method to achieve display calibration.

[0019] The same reference numerals and markings in the various figures refer to the same elements. Detailed Implementation

[0020] Figure 1 An example implementation of a user 500 utilizing a VAR system 100 is depicted. The VAR system 100 includes a frame structure 102, a display subsystem 104, a speaker 106, an eyepiece 110, a user orientation module 112, a computer processing unit (CPU) 114, a graphics processing unit (GPU) 116, a frame buffer 118, a three-dimensional (3D) database 120, and a light sensing component 122. The VAR system 100 can operate as an augmented reality system that can provide images of virtual objects blended with physical objects within the user 50's field of view.

[0021] like Figure 1 As shown, the frame structure 102 can be worn on the head of the user 50. The frame structure 102 can be integrated, connected to, or coupled to the speaker 106 and the display subsystem 104. The display subsystem 104 may include a left eyepiece 110L, a right eyepiece 110R, a left projection system 108L, and a right projection system 108R.

[0022] The speaker 106 can include a single speaker or a pair of speakers. For example, one speaker can be configured to output audio data in one ear, and a second speaker can be configured to output audio data in a second ear of the user 50. The speaker 106 can be positioned adjacent to the ear canal of the user 50. Generally, various types of speakers can be used, for example, a headset with speakers, earphones, headphones, etc. The speaker 106 can be connected to other components of the VAR system 100 wirelessly or through a wired connection. For example, in some implementations, the speaker 106 can be connected to other components of the VAR system 100 through a short-range communication protocol, such as Bluetooth.

[0023] The display subsystem 104 can be positioned above the nose and in front of the eyes 52 of the user 50, similar to the position of the lenses of eyeglasses. The display subsystem 104 can be configured to present photo-based radiation patterns to the eyes 52 of the user 50 that can be comfortably perceived as an augmentation of the physical reality, with high-quality two-dimensional (2D) or three-dimensional (3D) image content. The display subsystem 104 can output a sequence of frames obtained from the frame buffer 118 at various frequencies. In some cases, the display subsystem 104 can output frames at a high frequency to provide a perception of a single coherent scene.

[0024] The eyepieces 110 include a partially transparent left eyepiece 110L and a partially transparent right eyepiece 110R. When images are projected onto the left and right eyepieces 110L and 110R, they effectively operate as display screens or display interfaces. In some implementations, the left and right eyepieces (or display screens) 110L, 110R displays can be“optical see-through” displays through which a user can directly view light from real objects via a transparent (or semi-transparent) element. The transparent element can superimpose light from the projection subsystems 108L, 108R onto the user’s real-world view.

[0025] In some implementations, the eyepieces 110L and 110R can take the form of a waveguide device that includes a planar light waveguide and one or more diffractive optical elements (DOEs) associated with the planar light waveguide. The waveguide device can include a plurality of planar light waveguides and DOEs respectively associated with the planar light waveguides.

[0026] The left and right projection subsystems 108L, 108R can project left and right monocular images onto left and right eyepieces 110L, 110R, respectively. The eyepieces 110L, 110R can be placed in front of the eyes 52 of the user 50 to view the monocular images as a binocular image. Additionally, the eyepieces 110L, 110R can be placed in the field of view of the user 50 between the eyes 52 of the user 50 and the surrounding environment such that direct light from the surrounding environment travels through the eyepieces 110L, 110R to the eyes 52 of the user 50.

[0027] The projection assemblies 108L, 108R can provide scanning light to the eyepieces 110L, 110R, respectively. In some implementations, the projection subsystems 108L, 108R can be implemented as scanning light fiber-based projection devices, and the eyepieces 110L, 110R can be implemented as waveguide-based displays into which the scanning light from the respective projection subsystems 108L, 108R is injected. Each of the projection subsystems 108L, 108R can include a spatial light modulator (“SLM”), such as a liquid crystal on silicon (“LCoS”) assembly or a microelectromechanical (“MEM”) scanning mirror.

[0028] The VAR system 100 can also include one or more sensors mounted to the frame structure 102 for detecting the position and motion of the head 54 of the user 50 and / or the eye position and interocular distance of the user 50. Such sensor(s) can include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyroscopes.

[0029] The user orientation detection module 112 can be configured to detect the instantaneous position of the head 54 of the user 50 and determine the position of the head 54 of the user 50 based on position data received from the sensor(s). Detecting the instantaneous position of the head 54 can also facilitate the determination of the object that the user 50 is viewing. The user orientation module 112 can also track the eyes 52 of the user 50 based on tracking data received from the sensor(s).

[0030] The light sensing assembly 122 can sense light rays exiting the two eyepieces 110L, 110R. Additionally, as explained in greater detail below, the light sensing assembly 122 can be configured to sense at least one parameter indicative of a mismatch between the displayed left and right monocular images as a binocular image.

[0031] The VAR system 100 can also include a control subsystem that includes various software and hardware components. In some implementations, the control subsystem can include a central processing unit (CPU) 114, a graphics processing unit (GPU) 116, one or more frame buffers 118, and a 3D database 120 for storing three-dimensional scene data. The CPU 114 can control the overall operation of the VAR system 100, while the GPU 116 renders frames (e.g., converts 3D scenes to 2D images) from the 3D data stored in the 3D database 120 and stores the frames in the frame buffer(s) 118.

[0032] In general, the control subsystem can include various controllers, such as a microcontroller, a microprocessor, a CPU, a digital signal processor, a GPU, an application-specific integrated circuit (ASIC), a programmable gate array (PGA), a field PGA (FPGA), and / or a programmable logic controller (PLU). The control subsystem can include one or more processors and / or be in communication with one or more processors, such as the CPU 114 and the GPU 116, that perform the operations described in this specification, e.g., by executing executable instructions. Although not shown, one or more integrated circuits can be used to control the reading of one or more frames into and / or out of the frame buffer 118, as well as the operation of the left and right projection subsystems 108L, 108R of the display subsystem 104.

[0033] The VAR system 100 can be configured to operate in different modes. For example, in one mode, the cameras in the VAR system 100 can be used to capture images of the surrounding environment. The VAR system 100 can mix virtual images into the data representing the images of the surrounding environment to render a mixed reality image for a user to view. In another mode, the VAR system 100 can include one or more partially transparent surfaces through which an observer can see the surrounding environment. The VAR system 100 generates images of virtual objects that are transposed onto the partially transparent surfaces.

[0034] The VAR system 100 and the various techniques disclosed herein can also be used in applications other than augmented reality and virtual reality subsystems. Although certain implementations are described in the context of an augmented reality subsystem or a virtual reality subsystem, the VAR system 100 is not limited to such subsystems.

[0035] In augmented reality applications, it can be desirable to spatially position various virtual objects in the field of view of the user 50 relative to corresponding physical objects. As described above, the projection assembly 108L, 108R can project virtual objects onto the eyepieces 110L, 110R for display. Virtual objects can be referred to as virtual markers, markers, or calls, and can be implemented in a variety of forms. Examples of virtual objects can include, but are not limited to, virtual text objects, virtual numeric objects, virtual alphanumeric objects, virtual marker objects, virtual field objects, virtual chart objects, virtual map objects, virtual instrument objects, or virtual visual representations of physical objects.

[0036] As described above, the VAR system 100 includes eyepieces 110L, 110R integrated with the frame structure 102. As the frame structure becomes lighter, thinner, and more flexible to facilitate transport, comfort, and aesthetics, the frame structure also becomes more susceptible to deformation. These deformations can introduce distortion and other errors into the virtual binocular images.

[0037] For example, as shown in FIGS. 2A and 2B, virtual content 72L, 72R can be presented and perceived by a left eye and a right eye, respectively, through a pair of eyepieces 70L, 70R. In Figure 2A In the ideal case, the two eyepieces 70L, 70R are aligned with each other in an ideal manner. For example, since the time of manufacture of the frame structure 102, the alignment of the two eyepieces 70L, 70R has not changed.

[0038] However, if the alignment of the two eyepieces 70L, 70R were to change, the virtual content 74 presented by the two eyepieces 70L, 70R can be distorted. For example, Figure 2B An example of a misalignment of a pair of eyepieces 70L, 70R about a pitch axis is depicted. Other types of misalignments include, but are not limited to, misalignments along a roll axis or a yaw axis. In general, misalignments between the left and right eyepieces 70L, 70R can cause translational and / or rotational misalignments between the perceived left virtual content 72L and right virtual content 72R. Misalignments can cause physiological strain on the eyes of the user 50. Furthermore, and more generally, humans can be sensitive to binocular rotational misalignments of virtual images about pitch, roll, and yaw axes down to 4, 6, and 10 arcminutes, respectively.

[0039] Figure 3A and 3BAnother example of a frame structure 102 with aligned and misaligned eyepieces 110L and 110R is depicted. The frame structure 102 includes left and right cantilever arms 310L, 310R, left and right temple arms 302L, 302R, left and right hinges 308L, 308R, a nosepiece 306, and a bridge 304. The left and right temple arms 302L, 302R are designed to engage the head 54 of the user 50 such that the left and right eyepieces 110L, 110R are in front of the eyes 52 of the user 50. The temple arms 302L, 302R include left and right hinges 308L, 308R, respectively, to facilitate bending of the arms 302L, 302R for proper fitting of the frame structure 102 to the head 54 of the user 52. The nosepiece 306 is configured to rest on the nose of the user 52 and can have a convex surface that conforms to the shape of the nose of the user 52.

[0040] The left and right cantilever arms 310L, 310R are connected to cantilever portions 312 that extend away from the bridge 304. Additional arm portions 314 extend from the respective cantilever portions 312 in a plane that is parallel to the plane of the eyes 52 of the end user 52. The left and right eyepieces 110L, 110R are attached to the additional arm portions 314, respectively. Left and right projection subassemblies 108L, 108R are attached to the outer ends of the additional arm portions 314 to facilitate providing light beams into the left and right eyepieces 110L, 110R, respectively. In this manner, light rays can exit the left and right eyepieces 110L, 110R to display left and right monocular images as a binocular image to the user 50.

[0041] The end portions of the left and right cantilever arms 310L, 310R away from the nose of the user 50 include cameras 103L, 103R, respectively. The left and right cameras 103L, 103R are configured to obtain images of the user’s environment, e.g., objects in front of the user 50.

[0042] References Figure 3AIn state A, the VAR system 100 displays a virtual monocular image through left and right eyepieces 110L and 110R. The left projection subsystem 108L projects light representing the virtual content onto the left eyepiece 110L, which then couples and directs the light to a diffractive optical element (DOE) configured to provide orthogonal pupil expansion (OPE) and / or exit pupil expansion (EPE). While most of the directed light exits the eyepiece 110L as it passes through one or more DOEs (e.g., directed towards the user's left eye), a portion of the light continues toward the coupled DOE 190L, where it exits the eyepiece 110L as light (represented by ray 203) and is at least partially intercepted by the light sensing component 122. The right projection subsystem 108R, together with the right eyepiece 110R and one or more DOEs (e.g., output element 190R, input element (ICE), OPE, and EPE), can operate in a similar manner to the projection subsystem 108L in state A. For example, the projection subsystem 108R, the right eyepiece 110R, and one or more DOEs can present virtual content to the user's right eye and couple the light representing the virtual content out via the output DOE 190R and direct it to the light sensing component 122.

[0043] In state B (e.g.) Figure 3B As shown, the left and right eyepieces 110L and 110R are misaligned about the yaw axis. This could be caused, for example, by deformation of the arm 312 of the right cantilever 310R or by the cantilever itself. When such misalignment occurs, the angle at which light (represented by ray 203) leaves the right eyepiece 110R at state B differs from the angle at which light (represented by corresponding ray 203) leaves the right eyepiece 110R at state A, and the angle at which light (represented by ray 203) leaves the left eyepiece 110L at states A and B. Based on data output from the light sensing component 122, the VAR system 100 can detect such deformation or mismatch between the two eyepieces 110L and 110R.

[0044] Specifically, the CPU 114 can receive and process data acquired by the light sensing component 122. The CPU 114 can compare the data derived from the light incident on the light sensing component 122 when the frame structure 102 is in state A with the data derived from the light incident on the light sensing component 122 when the frame structure 102 is in state B, and determine the relative deformation state of the left and right eyepieces 110L, 110R. In response to detecting a relative deformation state or misalignment of the virtual image, the VAR system 100 can perform one or more calibration procedures to compensate the virtual image or the displayed image according to the deformation / misalignment.

[0045] Figures 4 to 7An example implementation of a calibration technique that performs calibration using calibration frames and coherent context is depicted. The calibration technique can be performed by one or more processors (hereinafter simply referred to as processors), such as CPU 114 or GPU 116. The processors can determine a number of frames to render according to a refresh rate of display subsystem 104. For example, if the refresh rate of display subsystem 104 is 120 Hz, then display subsystem 104 can generate a total of 120 frames, including one calibration frame for every 119 frames. More generally, as shown in Figure 4

[0046] In some implementations, to perform calibration in a computationally efficient manner, the processors can select one of the N-1 frames as a representative virtual content frame. The processors can determine a location of the content in the representative virtual content frame. To do so, the processors can use various content detection methods. For example, edge detection and chromatic pattern detection techniques can be used to detect the location of the content in the frame.

[0047] In the example shown in Figure 4 The processors determine that the virtual content is located in the upper left region of the frame. The processors can identify the locations and pixel values of the pixels associated with the detected content in the frame and can store data indicating the pixel locations and pixel values in memory. In some implementations, the representative virtual content frame can be determined by averaging the pixel values across the N-1 rendered frames. The pixel values can include values such as intensity values or RGB color values.

[0048] After determining the location of the virtual content, the processors can generate a calibration frame. Referring to Figure 5 The processors can convolve frame 510 (such as the representative virtual content frame) with test pattern 520 to generate calibration frame 530, in the example shown in Figure 5 In the example shown in Figure 5 Although the chessboard pattern is depicted as being used as test pattern 520, in general, various types of test patterns can be used, such as grids and crosses. The calibration frame can be generated by applying a patterned mask to the frame of virtual content. The resulting calibration frame can be similar to the frame of virtual content, with the test pattern subtracted from the frame.

[0049] Figure 6 An example of N coherent frames that include one calibration frame is depicted. Figure 6 ​The calibration frame in FIG. 6A shows virtual content in the form of a cube, which is convolved with a test pattern in the form of a plus sign. Advantageously, image processing is performed only in areas where virtual content is present.

[0050] The generated calibration frame can be inserted into N-1 frames. In general, the calibration frame can be inserted after any of the plurality of frames. For example, the calibration frame can be inserted in the middle of the plurality of frames or after the first 10 frames. By inserting the calibration frame in this manner, the calibration frame is less noticeable to the user, as the human brain tends to fill in, filter, or ignore brief anomalies (e.g., “holes”) in images received from the eyes.

[0051] In some implementations, the VAR system 100 can calibrate for colors of virtual content being displayed to the user. For example, if only blue virtual content is being displayed, the processor can perform calibration using a blue test pattern. If only red virtual content is being displayed, the processor can perform calibration using a red test pattern. If only green virtual content is being displayed, the processor can perform calibration using a green test pattern. If virtual content with a combination of red, blue, and green is being displayed, the processor can perform calibration using a combination of red, blue, and green calibration frames.

[0052] To further reduce the perceptible difference of the calibration frame, various characteristics of the calibration frame, such as intensity, can be configured to match or resemble characteristics of representative virtual content frames. For example, if the intensity of the virtual content is determined to be greater than or equal to a minimum threshold level of intensity, the intensity of the calibration frame can be equal to the intensity of the corresponding virtual content. If the intensity of the virtual content is determined to be less than the minimum threshold level of intensity, the intensity of the calibration frame can be set to the minimum threshold level of intensity.

[0053] In some implementations, image characteristics of the calibration frame, such as contrast or brightness, can be configured to further reduce the perceptibility of the test frame. In some implementations, the calibration frame can be diluted by hiding the test image behind the edges of the virtual content. The calibration frame can be further camouflaged by using similar textures and colors as the virtual content.

[0054] Figure 7 A flowchart depicting a method for implementing display calibration is depicted. As explained above, in a VAR system, virtual reality content can be generated to be displayed on a display or to augment a user’s view of a physical reality. To provide virtual content on a display, a processor of the VAR system can generate a plurality of frames including virtual content (710). The number of generated frames can depend on the refresh rate. For example, if the refresh rate is 120 Hz, the processor can generate 120 frames per second.

[0055] Using the techniques described above with respect to Figures 4 to 6 the processor can determine the area of the generated frame in which the virtual content is positioned (720). For example, edge detection and color pattern detection techniques can be used to detect the location of the content in the frame.

[0056] After determining the location of the virtual content, the processor can generate a calibration frame (730). As explained above, the processor can apply one of several possible test patterns and convolve it with the virtual content. Since the processor has identified the location of the virtual content, the convolution can be performed in a computationally efficient manner by performing the convolution only in the area of the frame in which the virtual content is positioned. The convolution of the test pattern and the virtual content produces a calibration frame that resembles the virtual content with the test pattern subtracted from the virtual content. In some embodiments, multiple copies of the calibration frame can be generated.

[0057] The generated calibration frame is then applied to the generated frame to display the virtual content according to the refresh rate (740). The calibration frame can be inserted anywhere in the frame that includes the virtual content. If desired, multiple calibration frames can be generated and inserted within the frame that includes the virtual content. Multiple calibration frames can be inserted randomly into a set of frames that include the virtual content, or according to a predetermined criteria set by a VAR system designer.

[0058] Based on the calibration frame(s), the processor can determine values for one or more calibration parameters to improve the display of the virtual content as desired on the left and right eyepieces 110L, 110R (750). For example, the processor can determine translation and / or rotation parameters for the eyepieces 110L, 110R (e.g., adjustments to pitch, roll, and / or yaw axes) and translation and / or rotation parameters for the left and right projection subsystems 108L, 108R that will result in the display of the virtual content in a desired manner.

[0059] After determining the calibration parameters, the VAR system can display the virtual content after adjusting its display subsystem 104 using the calibration parameters (760). For example, the projection of the virtual content or the position of the eyepieces 110L / 110R can be adjusted according to the calibration parameters to improve the display of the virtual content. If additional calibration is needed, the VAR system can repeat the calibration method described above. In some embodiments, if the virtual content display is not rendered as desired when a single calibration frame is used, the processor can generate multiple calibration frames and insert the multiple calibration frames within multiple frames of the generated frames that include the virtual content.

[0060] The calibration technique described above is advantageous because it calibrates the display only at the time and place where virtual content is shown to the user. This calibration method performs very well if the virtual content has strong contextual coherence and spatial distribution. Other benefits include low computational burden, low dependency on eye tracking, and low risk of degradation. Such benefits allow the calibration technique to perform well in high frequency displays.

[0061] The described systems, methods, and techniques can be implemented in digital electronic circuitry, in computer hardware, firmware, software, or in combinations of them. Apparatus implementing these techniques can include appropriate input and output devices, a computer processor, and a tangible computer program product tangibly embodied in a machine-readable storage medium for execution by a programmable processor. Processes implementing these techniques can be performed by a programmable processor executing a program of instructions to perform desired functions by operating on input data and generating appropriate output. The techniques can be implemented using one or more computer programs or non-transitory computer-readable storage media, including instructions that are executable by a programmable system to perform desired functions. The described systems, methods, and techniques can be implemented using one or more computer programs or non-transitory computer-readable storage media, including instructions that are executable by a programmable system to perform desired functions.

[0062] Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; in any case, the language can be a compiled or interpreted language. Suitable processors include general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and / or a random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, specially- designed ASICs (application-specific integrated circuits).

[0063] The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine- readable propagated signal, or a combination of one or more of them. The term“data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to a hardware processor, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0064] A computer program (also known as a program, software, software application, script, plug-in, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files stored in multiple locations. A computer program can be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0065] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0066] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The

[0067] The elements of a computer can include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, receive data from or transfer data to, or both, one or more mass storage devices for storing data (e.g., magnetic, magneto-optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a VAR system. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0068] Although this description contains many specifics, these should not be construed as limiting the scope of the disclosure or of the patentable scope in any way. Some features that are described in the context of separate embodiments can also be implemented in combination with each other. Conversely, various features that are described in the context of a single embodiment can also be implemented or practiced separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination. For example, although mapping operations are described as a series of discrete operations, the various operations can be divided into additional operations, combined into fewer operations, altered in the order of execution, or eliminated, depending on the desired implementation.

[0069] Similarly, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments. It should be understood that the described program components and systems typically can be integrated in a single software product or packaged into multiple software products. For example, although some operations are described as being performed by a processing server, one or more of the operations can be performed by a smart meter or other network component.

[0070] As used herein and particularly in the appended claims (e.g., the text of the appended claims), the term "comprising" is intended to be interpreted as "including" or "including but not limited to." The term "comprising" is used in the sense of "including" and means that other features can also be present. Similarly, the terms "comprises", "comprised of", "comprising", and the like can have the meaning ascribed to them in U.S. Patent law and can mean "including", "including but not limited to", and the like.

[0071] Additionally, if a particular number of introduced claim recitations is intended, such intention will be explicitly recited in the claim, and otherwise there will be no such limitation. For example, as a help to understanding, the following appended claims can contain use of the introductory phrases "at least one" and "one or more" to introduce a claim recitation. However, the use of such phrases should not be construed to mean that the scope of the application is limited to mat only contains one of the recitations given by such introductory phrases i.e. that the scope of the application is limited to mat contains only one example of the recitations given by such introductory phrases. It will be noted that mat the use of a transitional phrase, such as "comprising" or "including," followed by a recitation of a step or steps of a method or of elements of a composition, does not, per se, preclude the claim from being interpreted as a step-plus-claim limitation or as a means-plus-claim limitation. The mere use of the introductory phrases "comprising" and "including" followed by a listing of steps or elements does not suggest that mat any recited step or element is essential. The indefinite articles "a" and "an," as used herein in the specification (contd.)

[0072] Moreover, even if a particular number of introduced claim recitations is explicitly recited, one of ordinary skill in the art will recognize that mat such recitation should be interpreted to mean at least the recited number (e.g., "two times recitation" of a bare recitation of "two" without further modifier means at least two recitations or two or more recitations). Moreover, when a phrase is used such as "at least one of A and B" or "one or more of A, B, and C," it is intended to mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together. The term "and / or" is also intended to encompass this meaning.

[0073] The use of the terms "first," "second," "third," etc. herein does not necessarily indicate a particular order or a particular numbering of elements. Generally, the terms "first," "second," "third," etc. are used as generic identifiers to distinguish between different elements. These terms should not be interpreted to indicate a particular order or a particular numbering of elements unless a particular order or a particular numbering of elements is shown to be indicated by the terms "first," "second," "third," etc. Moreover, these terms should not be interpreted to mean a particular number of elements unless a particular number of elements is shown to be indicated by the terms "first," "second," "third," etc. For example, a first widget can be described as having a first face, while a second widget can be described as having a second face. The use of the term "second face" in relation to the second widget can be to distinguish this face of the second widget from the "first face" of the first widget, and not to imply that the second widget has two faces.

Claims

1. A computer-implemented method, comprising: Frames containing virtual content are generated by displaying virtual monocular images through the left and right eyepieces of a virtual or augmented reality (VAR) system according to the refresh rate. The position of the virtual content within the frame is determined by selecting one frame from the frame as a representative frame that includes the virtual content. A calibration frame is generated by convolving the virtual content in the frame with a test pattern, wherein the image characteristics of the calibration frame are used to disguise the perceptibility of the test pattern by using textures and colors similar to those of the virtual content for the calibration frame. as well as The calibration frame is applied to the frame containing the virtual content by randomly inserting the calibration frame into the frame containing the virtual content.

2. The computer-implemented method according to claim 1, wherein, Determining the position of the virtual content within the frame includes: One or more of edge detection or color pattern detection are applied to identify regions in the frame that include the virtual content; Identify the pixel positions and pixel values ​​within the region; and The data indicating the pixel position and the pixel value is stored in the memory.

3. The computer-implemented method according to claim 1, wherein, Convolving the virtual content in the frame with the test pattern includes: Select representative virtual content frames; and The virtual content in the frame is convolved with the test pattern in the representative virtual content frame.

4. The computer-implemented method according to claim 1 further includes: Determine the refresh rate of the display of the virtual or augmented reality system. The number of frames including the virtual content is based on the refresh rate.

5. The computer-implemented method according to claim 1, wherein, Applying the calibration frame to the frame that includes the virtual content includes: The calibration frame is inserted between two randomly selected frames that include the virtual content.

6. The computer-implemented method according to claim 1, wherein, Generating the calibration frame includes: Determine the intensity level of the virtual content in the frame; Determining that the intensity level of the virtual content in the frame is less than a minimum threshold intensity level; and Generate the calibration frame with an intensity level that matches the minimum threshold intensity level.

7. The computer-implemented method according to claim 1, comprising: In response to applying the calibration frame to the frame, one or more calibration parameters are determined; as well as The virtual content is displayed on the monitor using the calibration parameters.

8. The computer-implemented method according to claim 1, wherein, Generating the calibration frame includes: The test image is hidden behind the edges of the virtual content in the calibration frame; and The calibration frame is generated using the test pattern which has the same color as the virtual content.

9. A virtual or augmented reality system, comprising: A framework structure, which includes a display device configured to display data; as well as A processor, coupled to the frame structure, is configured to: Frames containing virtual content are generated by displaying virtual monocular images through the left and right eyepieces of a virtual or augmented reality (VAR) system according to the refresh rate. The position of the virtual content within the frame is determined by selecting one frame from the frame as a representative frame that includes the virtual content. A calibration frame is generated by convolving the virtual content in the frame with a test pattern, wherein the image characteristics of the calibration frame are used to disguise the perceptibility of the test pattern by employing textures and colors similar to the virtual content for the calibration frame; and The calibration frame and the frame including the virtual content are sent to the display device. In response to receiving the calibration frame and the frame including the virtual content from the processor, the display device is configured to display the calibration frame and the frame including the virtual content, which are randomly inserted into the frame including the virtual content.

10. The virtual or augmented reality system according to claim 9, wherein, The processor configured to determine the position of the virtual content in the frame includes a processor configured to: One or more of edge detection or color pattern detection are applied to identify regions in the frame that include the virtual content; Identify the pixel positions and pixel values ​​in the region; as well as The data indicating the pixel position and the pixel value is stored in the memory.

11. The virtual or augmented reality system according to claim 9, wherein, The processor is configured to convolve the virtual content in the frame with the test pattern, including that the processor is configured to: Select representative virtual content frames; as well as The virtual content in the frame is convolved with the test pattern in the representative virtual content frame.

12. The virtual or augmented reality system according to claim 9, wherein: The processor is configured to determine the refresh rate of the display of the virtual or augmented reality system; and The number of frames including the virtual content is based on the refresh rate.

13. The virtual or augmented reality system according to claim 9, wherein, The processor is configured to apply the calibration frame to the frame including the virtual content, wherein the processor is configured to: The calibration frame is inserted between two randomly selected frames that include the virtual content.

14. The virtual or augmented reality system according to claim 9, wherein: The processor is configured to determine one or more calibration parameters in response to applying the calibration frame to the frame; and The display device is configured to display the virtual content using the calibration parameters.

15. A non-transitory computer-readable storage medium including instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations including: Frames containing virtual content are generated by displaying virtual monocular images through the left and right eyepieces of a virtual or augmented reality (VAR) system according to the refresh rate. The position of the virtual content within the frame is determined by selecting one frame from the frame as a representative frame that includes the virtual content. A calibration frame is generated by convolving the virtual content in the frame with the test pattern, wherein, The image characteristics of the calibration frame are used to disguise the perceptibility of the test pattern by using textures and colors similar to those of the virtual content for the calibration frame. as well as The calibration frame is applied to the frame containing the virtual content by randomly inserting the calibration frame into the frame containing the virtual content.

16. The non-transitory computer-readable storage medium according to claim 15, wherein, Determining the position of the virtual content within the frame includes: One or more of edge detection or color pattern detection are applied to identify regions in the frame that include the virtual content; Identify the pixel positions and pixel values ​​within the region; and The data indicating the pixel position and the pixel value is stored in the memory.

17. The non-transitory computer-readable storage medium according to claim 15, wherein, Convolving the virtual content in the frame with the test pattern includes: Select representative virtual content frames; and The virtual content in the frame is convolved with the test pattern in the representative virtual content frame.

18. The non-transitory computer-readable storage medium according to claim 15, further comprising: Determine the refresh rate of the display of the virtual or augmented reality system. The number of frames including the virtual content is based on the refresh rate.

19. The non-transitory computer-readable storage medium according to claim 15, wherein, Applying the calibration frame to the frame that includes the virtual content includes: The calibration frame is inserted between two randomly selected frames that include the virtual content.

20. The non-transitory computer-readable storage medium according to claim 15, wherein, The operation also includes: In response to applying the calibration frame to the frame, one or more calibration parameters are determined; and The virtual content is displayed on the monitor using the calibration parameters.

Images