Flicker-free and marker-free biphase display calibration

Abstract

Techniques for separating an image into a forward swept image and a reverse swept image are disclosed. A lookup table maps MEMS projection positions on a display with corresponding pixel positions in an image generated by a camera facing the display. A lookup table is used to associate a first set of pixel positions in the image with a forward scan sweep of the MEMS system. The lookup table is used to associate a second set of pixel positions in the image with a reverse scan sweep of the MEMS system. The first set of pixel positions and the second set of pixel positions are used to generate a forward swept image and a reverse swept image, respectively. These images may then be used to calibrate the MEMS system to compensate for biphase.

Description

Background Art

[0001] Microelectromechanical systems (MEMS) are miniaturized mechanical and electromechanical components, typically manufactured using micromachining techniques. The physical size of MEMS devices can range from well below a micron to several millimeters.

[0002] One application of MEMS is related to optical switches and micromirrors for redirecting or modulating light beams, particularly for mixed reality (MR) systems (e.g., systems that may include augmented reality or virtual reality). Micromirrors can be precisely controlled by MEMS elements to swing back and forth at a given frequency. One or more laser beams of different intensities (e.g., red, green, and blue beams) can be combined into a single beam, and the single beam can be relayed to various MEMS scanning mirrors. The MEMS scanning mirrors then swing back and forth, scanning the beams in a raster pattern to project an image onto the display of the MR system. As long as the scanning rate is fast enough, the raster scanning laser beam can produce the impression of a stable image using a single scanning point. These images can produce a dynamic impression. Such a projected image or dynamic picture can be created by synchronously adjusting the position of the modulated laser and the scanning laser beams.

[0003] Each MEMS device may have a specific set of temperature characteristics, such that each MEMS device may behave slightly differently even when performing at the same temperature. For example, a MEMS scanning device may include one or more MEMS scanning mirrors, each of which may have its own temperature characteristics that may cause the projected image to be distorted or out of specification at high or low temperatures. Furthermore, certain temperature characteristics may gradually change over the life of the MEMS scanning device, causing the performance of the MEMS device to deteriorate over time.

[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in the above-described environment. Rather, this background is merely provided to illustrate one exemplary technology area in which some embodiments described herein may be practiced. Summary of the invention

[0005] Embodiments disclosed herein relate to systems, devices (e.g., wearable head-mounted mixed reality (MR) devices, hardware storage devices, etc.) and methods for separating an image into a forward sweep image and a reverse sweep image. The image is captured by a camera facing a display, and a micro-electromechanical scanning (MEMS) system is projecting rendered content on the display. The forward and reverse sweep images can be used to calibrate the MEMS system, for example, to compensate for bi-phase offsets that may occur due to use over time or temperature effects.

[0006] Some embodiments access a lookup table. The lookup table maps (i) a MEMS projection position on a display (wherein the MEMS projection position corresponds to the position of the MEMS system projected content in the display) to (ii) a corresponding pixel position in an image generated by a camera facing the display. The embodiment accesses an image, which is generated by a camera. Then, the lookup table is used to associate a first set of pixel positions in the image with a forward scanning sweep of the MEMS system. The lookup table is also used to associate a second set of pixel positions in the image with a reverse scanning sweep of the MEMS system. The embodiment uses the first pixel position set to generate a forward sweep image. The process of generating a forward sweep image is performed by extracting a first pixel content at each pixel position in the image in the first pixel position set and including the extracted first pixel content in the forward sweep image. Similarly, the embodiment uses the second pixel position set to generate a reverse sweep image. The process of generating a reverse sweep image is performed by extracting a second pixel content at each pixel position in the image in the second pixel position set and including the extracted second pixel content in the reverse sweep image.

[0007] Optionally, some embodiments perform a vertical alignment operation to vertically align the reverse sweep image and the forward sweep image. Embodiments may also perform a horizontal alignment operation to horizontally align the reverse sweep image and the forward sweep image. Then, based on the offset determined as a result of performing the horizontal alignment operation, embodiments may calibrate the MEMS system.

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0009] Additional features and advantages will be set forth in the following description, and in part will be apparent from the description, or may be learned through practice of the teachings herein. The features and advantages of the present invention may be realized and obtained by the instruments and combinations particularly pointed out in the appended claims. The features of the present invention will become more apparent from the following description and the appended claims, or may be learned through practice of the present invention as described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to describe the manner in which the above and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments and are therefore not to be considered limiting in scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1A , 1B, 1C and 1D illustrate aspects related to MEMS scanning devices.

[0012] Figure 2A and 2B Various scenarios involving bi-phase offset correlation are described.

[0013] Figure 3 The lines displayed to determine biphasicity are also described.

[0014] Figure 4A and 4B The diagram illustrates interleaving images generated from a camera on the HMD to generate forward and reverse images.

[0015] Figure 5A and 5B Various techniques for generating forward images (also called forward sweep images) and reverse images (or reverse sweep images) are illustrated.

[0016] Figure 6 The diagram shows an even-numbered sweep (ie, forward sweep) and an odd-numbered sweep (ie, reverse sweep) of the MEMS scanning system.

[0017] Figure 7 Various functional diagrams are illustrated.

[0018] Figure 8 Various functional diagrams are illustrated.

[0019] Fig. 9A and 9B A flow chart of an example method for generating forward-sweep images and reverse-sweep images is illustrated.

[0020] Fig.10 An example computer system is illustrated that may be configured to perform any of the disclosed operations. DETAILED DESCRIPTION

[0021] Embodiments disclosed herein relate to techniques for separating an image into a forward sweep image and a reverse sweep image. The original image is captured by a camera facing a display, and the MEMS system is projecting rendered content on the display. The forward sweep image and the reverse sweep image are then used to facilitate calibration of the MEMS system.

[0022] Some embodiments access a lookup table that maps MEMS projection positions on a display to corresponding pixel positions in an image generated by a camera facing the display. The lookup table associates a first set of pixel positions in the image with a forward scanning sweep of the MEMS system, and associates a second set of pixel positions in the image with a reverse scanning sweep of the MEMS system. The first and second sets of pixel positions are used to generate a forward swept image and a reverse swept image, respectively.

[0023] Some embodiments align the reverse sweep image and the forward sweep image vertically. These embodiments also align the reverse sweep image and the forward sweep image horizontally. Then, based on the offset (e.g., bi-phase offset) determined as a result of performing the horizontal alignment operation, the embodiments can calibrate the MEMS system.

[0024] Examples of technical advantages, improvements and practical applications

[0025] The following section summarizes some example improvements and practical applications provided by the disclosed embodiments. However, it should be understood that these are only examples and the embodiments are not limited to these improvements.

[0026] The disclosed embodiments beneficially provide a flicker-free and marker-free technique for bi-phase display calibration of an MR system. Use of this technique allows an MR system to continuously render time-critical content in the full field of view of the bi-phase display of the MR system. Traditional calibration techniques require the use of blank areas with reference patterns for online bi-phase and geometric calibration of the MR system's display. Blanking out the screen has proven to be quite cumbersome for the user experience. For real-time applications, it is critical to continuously render time-critical content on the display. The disclosed flicker-free and marker-free bi-phase display calibration technique is able to reliably calculate bi-phase correction values ​​while enabling the display to render time-critical content in its full field of view.

[0027] As mentioned above, the traditional method of dual-phase display calibration requires a blank area with a reference pattern. This traditional technique causes inevitable disadvantages to the user experience. For example, the blank area not only reduces the field of view of the display, but the traditional calibration technique also requires periodic flickering, which often causes discomfort to the user. These traditional limitations often cause users to miss information at various critical moments. They also cause visual fatigue.

[0028] The disclosed embodiments effectively solve these problems by providing a flicker-free and mark-free two-phase calibration solution. By following the disclosed principles, the user experience will be significantly improved. In addition, the visual display of information will also be improved because it will no longer be periodically corrupted when a calibration event is triggered.

[0029] Embodiments also provide energy savings because the camera generates fewer images than conventional techniques, which will be discussed in more detail later. Because embodiments also reduce computation time and proximity access to memory, additional energy savings are achieved. Accordingly, these and many other benefits will now be described in more detail in the remainder of this disclosure.

[0030] MEMS Devices

[0031] Having just described some of the various advantages, we now turn our attention to Figure 1A , Figure 1A The figure shows an example architecture of a MEMS scanning device 100 (hereinafter also referred to as “device”). The device 100 also includes one or more laser projectors 148, one or more MEMS scanning mirrors 136 and / or 138, a display 160, and a thermometer 150.

[0032] The laser projector 148 is controlled by the laser driver 140 to project a laser beam onto the MEMS scanning mirrors 136 and / or 138. In some embodiments, only one laser projector is implemented to project (multiple) grayscale images. In some embodiments, the laser projector 148 includes multiple laser projectors, each laser projector projects a laser beam of a different color. The multi-color laser beams are merged into a monochromatic beam for projecting a color image. In some cases, an infrared (IR) laser can also be used to project IR light onto the MEMS mirrors 136 / 138.

[0033] The laser projector 148 includes a red laser projector 148R, a green laser projector 148G, and a blue laser projector 148B. The red laser projector 148R projects a red laser beam, the green laser projector 148G projects a green laser beam, and the blue laser projector 148B projects a blue laser beam. The laser driver 140 uses three different control signals or parameters Vr 142, Vg 144, and Vb 146 to control the intensity (i.e., brightness) of each of the laser projectors 148R, 148G, and 148B at different times. The three different colored laser beams (each of which is projected at a specific intensity) are combined into a single laser beam to produce the desired color. For example, if each of the red, green, and blue laser projectors 148R, 148G, and 148B has 256 intensity levels, a total of 16777216 (=256×256×256) colors can be produced.

[0034] A single laser beam is projected onto a MEMS scanning mirror 136 and / or 138. The MEMS scanning mirror(s) 136 and / or 138 are controlled by a MEMS driver 130. The MEMS scanning driver 130 causes each of the MEMS scanning mirrors 136 or 138 to deflect back and forth at a certain amplitude and frequency. The amplitude and frequency of each MEMS scanning mirror 136 or 138 is controlled by control signals or parameters V1 132 and V2 134. The deflection of the MEMS scanning mirrors 136 or 138 causes the single laser beam to project a raster scan on the display 160.

[0035] In some embodiments, each of the MEMS mirrors 136 or 138 is a single-dimensional scanning mirror that scans in a single dimension. The MEMS mirror 136 is configured to scan in a first dimension (e.g., horizontally), and the MEMS mirror 138 is configured to scan in a second dimension (e.g., vertically). The first dimension and the second dimension are staggered (e.g., orthogonal). A single laser beam is first projected onto one of the MEMS mirrors 136 or 138. The laser beam is then reflected from one of the MEMS mirrors 136 or 138 to the other MEMS mirror 136 or 138. The laser beam is then reflected from the other MEMS mirror 136 or 138 to the display 160 to generate a two-dimensional raster scan.

[0036] Figure 1B Also illustrated is an example MEMS scanning device 100B that includes two single-dimensional scanning mirrors 136 and 138. These mirrors are configured to project a raster scan onto a display 160. Figure 1A (shown)

[0037] like Figure 1B As shown, the laser projector 148 first projects a single laser beam to the MEMS mirror 136. The MEMS mirror 136 deflects around the axis 133 to scan a single-dimensional line. The single-dimensional line is reflected from the MEMS mirror 136 to the MEMS mirror 138 deflected around the axis 135. The axis 133 and the axis 135 are orthogonal to each other, so that the single-dimensional line received by the MEMS mirror 138 is scanned into a two-dimensional raster image, which is finally projected onto the display 160.

[0038] Figure 1C An example sequence 100C of raster scanning is shown. Figure 1C As shown, first, the first mirror (eg Figure 1B The first mirror 136 in FIG. 1 is scanned forward and backward in the first dimension to draw the first period 102C at the top of the display 160. At the same time, the second mirror (eg Figure 1B The second mirror 138 in the first cycle 102C is gradually deflected so that the line scanned by the first mirror is gradually shifted in the second dimension. Similarly, the second cycle 104C and the third cycle 106C are continuously drawn by the first mirror below the first cycle 102C and the gradually deflected second mirror, and so on, until the last cycle or line 108C is drawn on the display. Then, the second mirror is deflected all the way back to the starting position of the cycle 102C to restart drawing the next frame of the image. As shown in Figure 1C As shown, the first mirror scans (horizontally) at a much faster scanning frequency than the second mirror (vertically), because the second mirror completes only the first cycle after the first mirror has scanned all cycles 102C-108C. Therefore, the first mirror is also referred to as a fast scanning mirror, and the second mirror is also referred to as a slow scanning mirror.

[0039] refer to Figure 1C When the MEMS scanning system scans from the left side to the right side, this scanning operation is the forward scanning of the MEMS system. When the MEMS scanning system scans from the right side to the left side, this scanning operation is the reverse scanning of the MEMS system.

[0040] A MEMS mirror typically has a resonant frequency that is determined by its mass, structure, and spring constant. In some embodiments, the deflection frequency of the fast mirror is controlled to be close to the resonant frequency of the mirror to obtain a large mirror deflection angle with a small current.

[0041] Figure 1D The figure shows an example control signal V1 132 for the fast scan mirror and an example control signal V2 134 for the slow scan mirror. Figure 1D As shown, the fast scanning mirror can be controlled by a basic sinusoidal signal of a first frequency, and the slow scanning mirror can be controlled by a basic sawtooth signal of a second frequency. The first frequency is much faster than the second frequency. In some embodiments, the fast scanning mirror can also be a two-phase scanning mirror that scans in both directions to project lines, so as to draw an image by lines projected in two directions. The slow scanning mirror is typically a single-phase scanning mirror that scans in only one direction, so that the laser beam is projected only during the first edge 102D of the sawtooth signal 134. During the second edge 104D of the sawtooth wave signal 134, the laser projector 148 can be turned off so that no laser beam is projected onto the display 160. When the MEMS system completes a set of scanning operations, a single display frame (such as Figure 1C That is, when the MEMS mirror is reset and returns to pointing Figure 1C , the single display frame is completed.

[0042] In some embodiments, two-dimensional raster scanning can also be achieved by a single two-dimensional scanning mirror. The two-dimensional scanning mirror scans in two dimensions (i.e., the first dimension and the second dimension). The first dimension intersects or is orthogonal to the second dimension. The MEMS driver 130 can use a first control signal or parameter V1 132 to control the deflection of the mirror in the first dimension, and use a second control signal V2 134 to control the deflection in the second dimension in the mirror. In this case, a single laser beam is projected onto a single two-dimensional scanning MEMS mirror, and the single MEMS mirror itself can reflect the laser beam onto the display 160 to generate a two-dimensional raster scan. The principles described herein are applicable to embodiments that implement two single-dimensional mirrors and / or embodiments that implement a single two-dimensional mirror.

[0043] It is desirable that the laser projector 148 and the MEMS mirrors 136 and / or 138 be substantially synchronized at all times to project a sharp image of the raster scan. In other words, each of the control parameter(s) V1 132, V2 134, Vr 142, Vg 144, Vb 146 should be substantially synchronized at all times. However, because each MEMS mirror 136 or 138 and / or certain other component(s) of the device 100 may have various unique temperature characteristics, the MEMS mirrors 136 or 138 and / or other components(s) may behave slightly differently when the temperature changes. Such differences may cause the raster scan to be distorted, or may cause the laser projector 148 and the MEMS mirrors 136 and / or 138 to be out of sync. Such distortion is a result of the bi-phase difference / offset that exists between the forward and reverse sweeps of the MEMS system.

[0044] Return to reference Figure 1A , the device 100 further includes a thermometer 150. The thermometer 150 is configured to detect the current temperature of the device 100. The thermometer 150 may be installed next to the MEMS scanning mirror 136 or 138, or installed at any position in the housing of the device 100. In some embodiments, a different thermometer 150 may be implemented for each MEMS scanning mirror 136 or 138, so that the current temperature of the corresponding MEMS scanning mirror 136 or 138 can be obtained more accurately.

[0045] like Figure 1A As shown, in some embodiments, device 100 also includes a display viewing camera 170 configured to capture (a plurality of) images of a predetermined area of ​​display 160. In some cases, the predetermined area captured by display viewing camera 170 is less than the entire field of view of display 160.

[0046] The display viewing camera 170 captures one or more images of whatever is displayed on the display 160. Notably, embodiments may operate without necessarily blanking any portion of the display 160, and may operate without necessarily displaying a predetermined pattern on the display 160. Alternatively, the display viewing camera 170 captures an image of whatever is currently being displayed during normal operation of the system. As will be described in more detail later, embodiments are capable of determining the level of bi-phase offset present. In response to identifying such a bi-phase offset, the processor 110 of the device 100 may cause the MEMS driver 130 and / or the laser driver 140 to adjust the control parameter(s) V1 132, V2 134, Vr 142, Vg 144, and / or Vb 146 to mitigate the difference during the calibration event.

[0047] Figure 2A and 2BThe concept of bi-phase shift is better explained. The examples are provided using scenarios involving dot projection. However, one will understand how this can be used with any type of image content, not just dots.

[0048] Figure 2A An example diagram showing a bi-phase scanning cycle 200A is shown. The vertical axis represents the amplitude of the fast scanning mirror (e.g., mirror 136); the horizontal axis represents time. During the bi-phase scanning cycle, the bi-phase scanning mirror scans forward and backward in the horizontal direction. Line 230 can be connected to Figure 1C In this example, any point on line 230 corresponds to two points drawn in the same cycle: (1) one drawn during the forward scan and (2) one drawn during the reverse scan.

[0049] Although two corresponding points should be considered distinct and can usually be plotted using different color codes (i.e. different intensities of the laser), if the biphasing is correct, i.e. the biphasic scanning cycles are synchronized (i.e. there is no offset), the two corresponding points should coincide on the horizontal axis but will still be separated along the other axis of the other mirror. For example, two corresponding points 232 are plotted on Figure 2A When the bi-phase is incorrect (i.e., there is a bi-phase offset), such corresponding points will be separated along the two axes. For example, the second corresponding points 234 and 236 are plotted at Figure 2B time T1 and T2.

[0050] Ideally, the amplitude of the scanning mirror corresponds closely to the control signal, so that when a sinusoidal control signal is applied to the scanning mirror, a corresponding sinusoidal amplitude is generated. However, in practice, there is usually a phase shift or delay between the control signal and the amplitude of the scanning mirror. The amount of phase shift is temperature dependent. Generally speaking, the higher the temperature, the greater the phase delay may be. The laser driver 140 must be synchronized with the phase of the mirror so that the projected image is sharp and not distorted.

[0051] Figure 2B An example of a phase shift of a MEMS scanning mirror is shown (i.e., there is a double phase offset here). Figure 2BAs shown, dashed line 250 illustrates the amplitude of the MEMS mirror, which is shifted from amplitude 240 (drawn in solid line). Assuming that the laser driver 140 projects a laser beam based on amplitude 240 to draw a corresponding dot at position or point 232, the laser beam is projected at time T1 and T2. However, if, in fact, the amplitude has shifted to dashed line 250, then at time T1, the first dot will be drawn at point 234; and at time T2, the second dot will be drawn at 236. In this exaggerated example, the two points or points 234 and 236 (which should coincide at the vertical axis 232) are far apart. This will significantly affect the final image projected on the display. When the temperature changes and there is a slight phase shift, the same principle applies, and even with a slight phase shift, the dual-phase scanning may cause two corresponding points to be projected at two slightly different positions in the dual-phase scanning direction, resulting in blurred and / or distorted images (e.g., double vision).

[0052] With traditional calibration techniques, the system blanks a portion of the screen, displays a dot pattern, and then captures an image of the dot pattern, such as Figure 3 As shown in image 300 in . It is worth noting that the MEMS system will project a point on both the forward sweep and the reverse sweep. In some cases, there may be some uncertainty as to which point was generated during the scanning process of the MEMS system. Therefore, the MEMS system will typically repeat this projection process multiple times to determine which point was displayed during the forward sweep and which point was displayed during the reverse sweep. The MEMS system can then compare the multiple images with the calibration image to determine the degree of out-of-phase of the MEMS (i.e., determine the bi-phase offset). Typically, this image capture and comparison event occurs every 10-20 seconds. In addition, the raster / projection event is typically performed using at least three frames, which can also disrupt the user experience. Users find this frequent screen blanking problematic.

[0053] Improved calibration techniques

[0054] The disclosed embodiments provide an improved technique for calibrating a MEMS system without requiring a blanking event, and also without requiring the display of a predetermined pattern.

[0055] An embodiment initially determines display pixel coordinates for a display. That is, for each pixel displayed on the display, an embodiment determines a corresponding set of coordinates for that pixel. To further clarify, an embodiment is capable of capturing an image of content displayed on the screen. The display viewing camera 170 generates or captures the image.

[0056] Based on the factory calibration of the MR system / MEMS system, an embodiment is able to correlate the pixel positions in the image with the pixel positions displayed on the screen. In other words, an embodiment has a calibration parameter set that is able to map the pixel coordinates of the pixels in the image generated by the display viewing camera 170 and the pixel coordinates of the pixels actually displayed on the display 160 (or more precisely, the MEMS projected position). Figure 4A is illustrative.

[0057] Figure 4A An example of a head mounted device of an MR system 400 is shown. The MR system 400 may include or may represent Figure 1A A MEMS scanning device 100 is provided.

[0058] MR system 400 is shown to include a display including a left-eye display 405 and a right-eye display 410. Either the left-eye display 405 or the right-eye display 410 represents Figure 1A The display 160 in FIG. It will be appreciated how each display may include Figure 1A Its own corresponding set of components is shown.

[0059] Figure 4A An exaggerated blow-up of various pixels (or more precisely, MEMS projected locations) are illustrated, labeled "a," "b," "c," "d," "e," "f," "g," and "h." Pixel "a" is shown on the left eye display 405 as having a pixel location 415 or pixel coordinates. During operation, the MEMS system will illuminate that area of ​​the display with a laser. The MEMS system will then illuminate the next area, and so on. If the MEMS system scans fast enough, an entire display frame can be generated by the MEMS system scanning or illuminating corresponding locations on the display alone.

[0060] Figure 1A The display viewing camera 170 in the embodiment can capture an image 420 of the left eye display 405 (another display viewing camera can capture an image of the right eye display 410). An embodiment has an initial set of calibration parameters that can map pixel locations in the image 420 to pixel locations on the left eye display.

[0061] For example, various pixels are labeled in image 420, such as pixels "a", "b", "c", "d", "e", "f", "g", and "h" shown in image 420. Notably, pixel "a" has pixel position 425. Pixel position 425 of pixel "a" in image 420 corresponds to pixel position 415 of pixel "a" in left eye display 405. Thus, an embodiment is able to generate or access an initial set of calibration parameters that map pixel positions from a display (i.e., MEMS projected positions) to pixel positions of display content captured in a generated image. The mapping between the MEMS projected positions on the display (e.g., pixel position 415) and the corresponding pixel positions in the image (e.g., pixel position 425) can be maintained in a lookup table 425A.

[0062] During normal and non-suspended use of the MR system, and without blanking the screen, embodiments generate Figure 4B 4. A frame of what may be referred to as an interlaced image 430 is shown. The interlaced image 430 is generated during normal use of the MR system, which means that the frame may include various levels of holographic content. The interlaced image 430 may represent Figure 4A Image 420 in.

[0063] Embodiments are able to parse, isolate, extract, or otherwise separate the interlaced image 430 into two separate images, namely a forward image 435 (also referred to as a forward sweep image) and a reverse image 440 (also referred to as a reverse sweep image). Because embodiments include information detailing the different pixel locations of the image as they relate to the display, and because embodiments also include detailing which pixels correspond to a forward sweep of the MEMS system and which pixels correspond to a reverse sweep of the MEMS system, embodiments generate the forward image 435 by parsing all pixels generated during a forward sweep of the MEMS from the interlaced image 430. If the first row of pixels in the image is labeled row 0, then the forward sweep will include row 0 pixels as well as pixels in even-numbered rows.

[0064] An embodiment generates reverse image 440 by parsing, isolating, extracting, or otherwise separating all pixels generated during a reverse sweep of the MEMS system from interlaced image 430. If the second row of pixels in the image is labeled row 1, the reverse sweep will include pixels in odd rows.

[0065] Figure 4BAlso shown are texture 445 and texture threshold 450. Further details on these aspects will be provided later, but by way of a quick introduction, the interlaced image 430 (and optionally the forward image 435 and reverse image 440) will need to have a sufficient amount of texture 445 (i.e., a texture level that exceeds the texture threshold 450) in order to perform the disclosed operations. Texture 445 refers to the amount of spatial arrangement of colors that an image has. In other words, it describes the pixel intensities of the image.

[0066] Figure 4B A calibration scenario is shown where the MEMS system is properly calibrated and there is no bi-phase offset (e.g., a timing offset between the forward sweep of the MEMS and the reverse sweep of the MEMS). Figure 5A An uncalibrated scene is shown involving interlaced image 430, resulting forward image 505, and reverse image 510. Note, however, that there is an offset or "double vision" in interlaced image 500, where the offset is a result of dual phase offset 515.

[0067] By separating the forward and reverse images from the interlaced images, embodiments are able to calculate the amount or level of bi-phase difference that exists (i.e., bi-phase offset or bi-phase value). Once the bi-phase value is determined, embodiments can then trigger a calibration event to determine the bi-phase offset. Compared to conventional techniques (e.g., dot pattern techniques that require the display to be blank and multiple images to be collected), embodiments are able to immediately obtain observations that distinguish the marks of the forward and reverse sweeps of the MEMS in a single frame. On the other hand, conventional systems require the use of multiple frames to determine which pixels correspond to the forward sweep and which correspond to the reverse sweep. Because the disclosed embodiments require the use of fewer frames, these embodiments improve the efficiency of the computer system when performing various calibration actions.

[0068] When calculating the bi-phase shift, an embodiment identifies a pixel, point, or region in the forward image 505, and then attempts to identify a corresponding pixel, point, or region in the reverse image 510. In some implementations, pixels are selected based on pixel coordinates that are adjacent to each other. In some implementations, pixels are selected using machine learning, where machine learning attempts to select pixels that represent the same or similar content. In some implementations, pixels are selected based on their color values ​​(e.g., pixels with the same or similar color values).

[0069] For example, Figure 5A Selected pixels in the forward image 505 are shown, as labeled forward scan pixel 520. This pixel is generated when the MEMS system is scanning the projected pixel in the forward direction. Figure 5AAlso shown are selected pixels in reverse image 510, as shown in reverse scan pixel 525. These two pixels are very close to each other. For the purpose of explanation, forward scan pixel 520 is located at the end of the forward sweep of the MEMS system, and reverse scan pixel 525 is located at the starting end of the next reverse sweep of the MEMS system. Selecting these pixels causes the minimum distance between the selected pixels in the forward image 505 and the selected pixels in the reverse image 510. For clarification, in this scenario, select the first pixel from a scan line generated during the forward sweep, and select the second pixel from the different scan lines generated during the reverse sweep. In this case, select the last pixel from the forward sweep, and select the first pixel from the reverse sweep. Although these two pixels are located on different lines, they should be adjacent to each other. Therefore, these two pixels should be highly close to each other. By selecting adjacent pixels, subsequent vertical and / or horizontal alignment processes can be reduced to a minimum.

[0070] By calculating the horizontal distance between the forward scanned pixels 520 and the reverse scanned pixels 525, the bi-phase value / offset can be selectively selected. In some cases, multiple bi-phase values ​​can be calculated. For example, multiple pixels from multiple lines can be selected from the forward image 505, and multiple corresponding pixels can be selected from the reverse image 510. Then, embodiments can use these pixels to calculate multiple bi-phase values. In some instances, these multiple bi-phase values ​​can be different relative to each other, so that for different rows of the forward and reverse images, the bi-phase values ​​may be different.

[0071] Some embodiments perform a vertical alignment 530 operation. For example, observe Figure 5A , it can be easily seen that the forward scanned pixel 520 has a slight vertical offset relative to the back scanned pixel 525. In this particular case, the vertical offset is equivalent to the distance of one line. For this vertical alignment 530, different techniques can be implemented.

[0072] In some cases, embodiments perform a vertical alignment operation by shifting either forward-scan pixel 520 or reverse-scan pixel 525 (or a row that may include these pixels) a distance (eg, perhaps one row) to compensate for any offset that exists between the two pixels.

[0073] Optionally, the amount of vertical alignment 530 can be set so that the level of the amount of correlation between the two pixels is adjusted to the level of maximum correlation. That is, based on various statistical characteristics of the two pixels, there will be a certain degree of vertical offset adjustment that results in the two pixels having the highest correlation relative to each other. Embodiments may choose to adjust to achieve this highest level of correlation.

[0074] Other techniques may be performed for the vertical alignment 530 operation. For example, some embodiments may optionally "fill in" or "restore" missing points in one or both of the forward image 505 or the reverse image 510 so that the horizontal bi-phase can be subsequently calculated based on verified pixel correspondences.

[0075] One example embodiment replicates the nearest neighboring pixels to "fill in" the missing pixels. Figure 5A , reverse scan pixel 525 is shown to be slightly lower than forward scan pixel 520. Embodiment can correct this vertical offset by selecting at least one of the nearest adjacent pixels in forward scan pixel 520 or reverse scan pixel 525. Select this nearest adjacent pixel to compensate for vertical offset. Once compensate (that is, remove) this vertical offset by selecting the nearest adjacent pixel, then embodiment can use the same selected pixel to calculate horizontal offset, as will be discussed later. That is to say, the selected pixel will maintain the horizontal offset between forward scan pixel 520 and reverse scan pixel 525, but it will be selected to eliminate the vertical offset between forward scan pixel 520 and reverse scan pixel 525.

[0076] Another way to "restore" missing points is through the process of interpolation. Figure 5B A representative example interpolation method 535 is shown. Here, two pixels are identified, as shown by pixel 540 and pixel 550. It should be noted that more than two pixels may be optionally used. In fact, the interpolation may be a weighted average of all pixels to recover the pixel value at any point in the image. The disclosed principles are not necessarily limited to two pixels in their neighborhood. Therefore, this scenario is provided only as an example and should not be considered limiting.

[0077] These pixels can be in either the forward image or the reverse image. It is desirable to calculate an interpolated pixel 545 that exists between pixel 540 and pixel 550 in order to remove the vertical offset. In order to calculate the interpolated pixel 545, some embodiments calculate a linear average, a convex combination, or even a weighted average of the intensity values ​​of pixels 540 and 550. Some embodiments calculate a bilinear interpolation of these two pixels, wherein the bilinear interpolation may include a bilinear interpolation of four adjacent pixels for a particular point / pixel. Such a process can be performed to fill in the missing pixel context currently present at a position between pixels 540 and 550. It is beneficial to have an accurate vertical correspondence between the forward image and the reverse image in order to subsequently calculate horizontal disparity / offset (i.e., bi-phase offset).

[0078] With reference to the various techniques described above, the line shifting technique can be viewed as a simple or naive approach. The nearest neighbor and interpolation techniques can be viewed as more complex approaches. One potential risk of performing a line shifting approach is that if the shifted pixel represents completely different content (even though it is only a line away), then it may distort the calculation. More complex approaches may better address such scenarios (i.e., pixels that are a line away represent completely different content, such as in the case of a boundary area of ​​content). Machine learning can also be implemented to optimize any of the above methods (e.g., smooth pixel values).

[0079] After the vertical alignment is performed, some embodiments then select a single pixel from the forward image and a corresponding single pixel from the reverse image. The horizontal difference (i.e., bi-phase offset) can then be calculated based on the two selected pixels. Thus, a single pair of pixels can optionally be used to calculate the bi-phase offset for an entire region (e.g., where the region is the entire image). This single pixel approach may reduce accuracy, but also improves (e.g., reduces) calculation time. Therefore, after the vertical offset adjustment (i.e., vertical alignment 530), the calculation of the horizontal offset is calculated.

[0080] On the other hand, some embodiments calculate multiple horizontal differences by selecting pixels from multiple lines of an image and then calculating multiple horizontal differences. In some cases, a pixel is selected for each line in the image. In some embodiments, pixels are selected for some, but not all, lines in the image. Therefore, multiple pairs of pixels can be optionally used to calculate the bi-phase offset of the entire area (for example, the area is the entire image). This method can improve accuracy, but will also increase calculation time. Therefore, an embodiment can select an area in the forward image and select a corresponding area in the reverse image. These selected areas can be as small as one pixel. In some cases, the selected area can include multiple lines. Then, the embodiment selects one or more pixels from the selected area and calculates the bi-phase offset by calculating the horizontal offset between the selected pixels in the two images. Optionally, a single line can be selected. Optionally, multiple lines can be selected.

[0081] In some cases, it may be chosen to calculate a horizontal offset by overlaying the reverse image onto the forward image. An embodiment then horizontally shifts the reverse image (or, alternatively, the forward image) in an attempt to identify a scene in which the two images become most aligned relative to each other. This shift is then set as a horizontal offset, which represents the presence of a bi-phase offset. Some embodiments impose various size restrictions on the forward and reverse images. For example, when overlaying two images, some embodiments will reduce the size of these images based on specific regions of interest selected to determine correlation. These sizes can be reduced based on a selected reduction in the number of pixels to be included in the image. These sizes can be reduced by cropping or trimming a selected amount or percentage of the image. In some cases, the sides of the image can be cropped so that the image is horizontally shorter than before.

[0082] Therefore, to calculate the normalized correlation between the images, the embodiments attempt to create a perfect alignment between these images by performing any of the vertical alignment processes mentioned earlier as well as the horizontal alignment process. The disclosed embodiments utilize an error-based metric (e.g., a horizontal shift process that detects a "most aligned" condition) to calculate the bi-phase offset. To further clarify, a process of interpolating cross-correlations can be performed to find the bi-phase differences. The embodiments can also calculate the cross-correlations between the interpolated forward and reverse images. Optionally, Fourier transforms can be used to recover missing values ​​in the forward and reverse images. In addition, optimal estimation algorithms (e.g., Kalman and particle filters) can be used to take into account uncertainty information for the offset parameters during the calibration event. More details of the Kalman filter will be provided shortly.

[0083] In order to perform the above process, the forward image and the reverse image need to have a sufficient amount of texture. That is, the texture level in the image should meet or exceed the minimum texture requirement. If the texture level of the detected image is below a threshold, it may be difficult to identify enough pixels to perform vertical and horizontal alignment. Therefore, if the texture level is below a threshold, some embodiments revert to the traditional technique of blanking the screen and projecting a dot pattern to perform calibration. Figure 4B The texture threshold 450 is illustrated.

[0084] Some embodiments calculate a confidence indicator to indicate the confidence of the embodiment in calculating the horizontal offset value. For example, the variance of the horizontal offset value. The confidence indicator can be provided to the MR system to determine whether the MR system should continue to use the calculated offset to perform calibration, or alternatively, whether it should continue to use the traditional method. That is, if the confidence indicator meets or exceeds a predefined threshold, the MR system can choose to use the calculated horizontal offset (dual phase offset) to perform its calibration. If the confidence indicator is below the threshold, the MR system can choose to use the traditional blanking method.

[0085] In some implementations, a Kalman filter or an extended Kalman filter can be used to take into account the dynamic changes of the bi-phase correction. It is particularly powerful in scenarios involving low or poor confidence indicators of bi-phase offset. For example, when calculating new bi-phase and / or calibration parameters, an embodiment can utilize a Kalman filter or an extended Kalman filter to consider past bi-phase offset estimates and their variances. Doing so enables an embodiment to improve the robustness of display calibration. An embodiment can be triggered to utilize a Kalman filter or an extended Kalman filter to fundamentally reject the current, poorly performing estimate of the bi-phase offset, and using historical data, generate a new bi-phase offset estimate based on a Kalman filter or an extended Kalman filter. In some implementations, more recent historical data can be attributed to a higher relevance rating, or may have a higher priority when calculating a bi-phase offset estimate.

[0086] In the case where the texture of the image visible in the display is below a texture threshold, some embodiments use an information camouflage method to add texture, which is a technique for obscuring or hiding a secret data set in non-secret data. For example, an embodiment may optionally apply a watermark type or static image to the display. Using infrared light (IR), the static image can be displayed so that it is invisible to a human user. The display viewing camera 170 can be configured to detect IR light and visible light. Therefore, the display viewing camera 170 will be able to detect the static image. When the detected texture level is too low, the introduction of infrared light can be performed to increase the texture.

[0087] In some cases, the static image may be visible light, but may be small. For example, the size of the static image may be only a single pixel, where the single pixel is an added pixel that would not normally be included in the image. In some cases, the static image may be one or a group of multiple pixels that would normally be included in the image, but the intensity of these pixels may be slightly modified. Such operation beneficially adds texture to the resulting image generated by the display viewing camera 170.

[0088] Some embodiments perform a sub-pixel refinement process to determine a finer granularity or more accurate determination of the dual phase offset. Sub-pixel refinement is performed subsequent to a previous correlation or poor correlation operation. More details on the cross-correlation operation are provided below. Sub-pixel refinement will then be discussed.

[0089] For continuous real-valued functions f and g, their cross-correlation is defined as:

[0090]

[0091] where τ is called displacement or hysteresis. *The highly correlated f and g have the maximum cross-correlation at t, and the characteristics of f at t are also later at g τ * So g can be described as lagging f by τ * :

[0092]

[0093] Similarly, for discrete functions, the cross-correlation is defined as:

[0094]

[0095] Here n is called displacement or hysteresis. * The highly correlated f and g have the maximum cross-correlation at position m. The feature of f at position m also appears later in g at position n, so g can be described as lagging f by n. * :

[0096]

[0097] like Figure 6 As shown, for an interlaced image, even (eg, forward sweeps) and odd (eg, reverse sweeps) sweeps are to be correlated. Figure 6 A graph for an even sweep 600 and a graph for an odd sweep 605 are shown. Let f and t be the even and odd sweeps, respectively. The maximization of the discrete cross-correlation is a bi-phase value with discrete precision.

[0098] In mathematics, linear interpolation is a curve fitting method that uses a linear polynomial to construct new data points within a discrete set of known data points. 0 ,y 0 ) and (x 1 ,y 1 ), then the linear interpolation is the straight line between these points. 0 , x 1 ), the value x along the line is given by Figure 7 The slope equation of the linear function 700 is shown. The linear interpolation on the data set is composed of multiple segments of linear interpolation, such as Figure 7 The piecewise linear function 705 is shown as follows.

[0099] After cross-correlating the two discrete functions, subpixel refinement uses their interpolation function to obtain bi-phase values ​​with decimal precision. A continuous function of the intensity distribution in the image can be calculated. Interpolation tries to fill this value from sparse data. Linear interpolation is used for its simplicity, while cubic splines can also be used for the smoothness of their second-order derivatives. Figure 8An example implementation showing sub-pixel refinement is shown with even sweeps 800 and odd sweeps 805 .

[0100] It is possible to obtain discrete biphase values ​​to maximize the cross-correlation. Let n * is a discrete bi-phase value. Assume that the bi-phase value τ of the odd sweep relative to the even sweep * , belongs to n with radius 1 * In the neighborhood, that is, τ * ∈(n * -1,n * +1).

[0101] Then, the cross-correlation of the second interval may be determined.Using these different intervals and cross-correlations, embodiments can provide improved granularity and accuracy in determining the amount of bi-phase and offset.

[0102] Example Method

[0103] The following discussion now involves a number of methods and method actions that can be performed. Although method actions may be discussed in a particular order or shown in a flowchart as occurring in a particular order, no particular order is required unless specifically stated or required because one action depends on another action being completed before the action is performed.

[0104] Now focus on Fig. 9A and 9B , they illustrate a flow chart of an example method 900 for separating an image into a forward sweep image and a reverse sweep image. These images can then be used to determine the bi-phase offset of the MEMS system. The system can then self-calibrate using the known bi-phase offset. The image is captured by a camera facing the display (e.g., Figure 1A A display viewing camera 170 in FIG. 1 is shown, and a micro-electromechanical scanning (MEMS) system is projecting rendered content on the display.

[0105] Optionally, method 900 can be performed by a MEMS scanning device 100, which can be an example of a mixed reality (MR) device. In some cases, the MR device can be a head-mounted device (HMD). Therefore, method 900 can be optionally performed by a wearable MR system, for example in the form of an HMD, and so that a computer system, or more precisely, a wearable MR system, includes a display and a MEMS system. In some cases, method 900 can be optionally performed by a cloud service operating in a cloud environment. The above-mentioned display can be one of a left eye display or a right eye display for the MR system.

[0106] There are various benefits to performing method 900. Some benefits (of course not all) are that, compared to conventional methods, method 900 can be performed without blanking (or avoiding blanking) any portion of a display. In addition, method 900 can be performed without displaying (or not displaying) a predetermined pattern on the display. Instead, method 900 can be performed using any content that has been projected onto a display without interrupting the projected scene or hologram.

[0107] like Fig. 9A As shown, method 900 includes accessing a lookup table (e.g., Figure 4A The lookup table maps (i) MEMS projected positions on the display corresponding to positions in the display where the MEMS system projects content to (ii) corresponding pixel positions in an image generated by a camera facing the display. The lookup table is typically generated during an initial calibration event for the MEMS system, such as a factory calibration event.

[0108] Action 910 includes accessing an image, the image being generated by a camera. For example, the camera may be from Figure 1A The display viewing camera 170 (e.g., a display alignment tracker (DAT) camera) is used to view the display. It is important to note that the image is generated after the MEMS system projects a complete image frame onto the display. That is, the MEMS system has completed all projections for that particular frame and is resetting itself to start projection of a new frame. The camera generates an image of the display during the reset transition phase or after the last scan is completed.

[0109] Embodiments can determine the texture level of the image. In addition, embodiments can determine whether the texture level of the image meets or exceeds a minimum texture level requirement. If the texture level does not meet or exceed the minimum texture level requirement, the MR system can fall back to traditional methods of performing calibration, such as by blanking the screen and projecting a known pattern.

[0110] Alternatively, the MR system may attempt to add texture to the display. For example, an embodiment may determine the amount of texture in the image, and then determine that the amount of texture in the image is below a texture threshold. An embodiment may cause the MEMS system to add texture to a subsequent projection of a subsequent display frame. An embodiment may then cause the camera to generate a subsequent image of the subsequent display frame. An embodiment may then generate a new forward sweep image and a new reverse sweep image based on the subsequent image.

[0111] Action 915 then includes associating a first set of pixel positions in the image with a forward scanning sweep of the MEMS system using a lookup table. In parallel, action 920 includes associating a second set of pixel positions in the image with a reverse scanning sweep of the MEMS system using a lookup table. For example, an image typically includes multiple rows of pixels. For illustration, Figure 4A 4 shows how image 420 includes at least one row of pixels "ah". Typically, there will be multiple rows of pixels in an image. For simplicity, Figure 4A Only one row is shown.

[0112] One row in the image will correspond to the set of content displayed when the MEMS system performs a forward sweep. The next row in the image will correspond to the set of content displayed when the MEMS system performs a reverse sweep. Embodiments can establish a correlation between the movement of the MEMS system and the position of the pixels in the image.

[0113] Action 925 includes generating a forward sweep image using a first pixel position set. It is noteworthy that the forward sweep image is generated by extracting a first pixel content at each pixel position of the image in the first pixel position set and including the extracted first pixel content in the forward sweep image. Meanwhile, action 930 includes generating a reverse sweep image using a second pixel position set. The process of generating a reverse sweep image is performed by extracting a second pixel content at each pixel position of the image in the second pixel position set and including the extracted second pixel content in the reverse sweep image.

[0114] In some cases, the first pixel content is obtained from all even-numbered rows of pixels of the image. Similarly, in some cases, the second pixel content is obtained from all odd-numbered rows of pixels of the image.

[0115] like Fig. 9B As shown, method 900 can continue. For example, after generating forward and reverse sweep images, method 900 may include performing a vertical alignment operation to vertically align the reverse sweep image and the forward sweep image (action 935). Various different options for performing this vertical alignment have been mentioned above. For illustration, one of the following items of the vertical alignment operation is selected: a line shift operation, a nearest neighbor operation, or an interpolation operation. The vertical alignment operation restores the missing content of one or both of the forward sweep image or the reverse sweep image. Missing content exists in each image because each image only includes the pixel content of half of the original image generated by the camera. However, it is desirable to provide perfect alignment between the two images, or at least an alignment that satisfies the alignment threshold, to accurately perform calibration.

[0116] Action 940 includes performing a horizontal alignment operation to horizontally align the reverse sweep image and the forward sweep image. The horizontal alignment operation is performed after the vertical alignment operation. Based on the horizontal alignment operation, an embodiment may be present in an offset level (i.e., a bi-phase offset) between the forward sweep image and the reverse sweep image. An embodiment may set the offset level to a parameter of the bi-phase offset that exists between the forward sweep image and the reverse sweep image. In some cases, a confidence index is calculated to indicate the alignment degree of the reverse sweep image and the forward sweep image. For example, the variance of the bi-phase estimate.

[0117] Based on the offset (eg, bi-phase) determined as a result of performing the horizontal alignment operation, act 945 includes calibrating the MEMS system. That is, embodiments may trigger a calibration event for the MEMS system to compensate for the bi-phase offset.

[0118] Thus, the disclosed embodiments relate to new and improved techniques for facilitating calibration events of MEMS systems to compensate for bi-phase offsets.The embodiments reduce power consumption and improve computational efficiency.

[0119] Example Computer / Computer System

[0120] Now turn your attention to Fig.10 , Fig.10 The figure shows an example computer system 1000, which may include and / or be used to perform any of the operations described herein. For example, the computer system 1000 may be the MR system mentioned above, or the computer system 1000 may implement the cloud service mentioned above. The computer system 1000 may take a variety of different forms. For example, the computer system 1000 may be embodied as a tablet computer, a desktop computer, a laptop computer, a mobile device, or a stand-alone device, such as those described in the present disclosure. The computer system 1000 may also be a distributed system, which includes one or more connected computing components / devices that communicate with the computer system 1000.

[0121] In its most basic configuration, computer system 1000 includes a variety of different components. Fig.10 Computer system 1000 is shown including one or more processors 1005 (also referred to as “hardware processing units”) and memory 1010 .

[0122] With respect to processor 1005, it should be understood that the functionality described herein may be performed, at least in part, by one or more hardware logic components (e.g., processor 1005). For example, and without limitation, exemplary types of hardware logic components / processors that may be used include field programmable gate arrays (“FPGAs”), program-specific or application-specific integrated circuits (“ASICs”), program-specific standard products (“ASSPs”), systems on chips (“SOCs”), complex programmable logic devices (“CPLDs”), central processing units (“CPUs”), graphics processing units (“GPUs”), or any other type of programmable hardware.

[0123] As used herein, the terms "executable module," "executable component," "component," "module," or "engine" may refer to a hardware processing unit or a software object, routine, or method that may be executed on the computer system 1000. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on the computer system 1000 (e.g., as separate threads).

[0124] Memory 1010 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage, such as physical storage media. If computer system 1000 is distributed, processing, memory, and / or storage capabilities may also be distributed.

[0125] Memory 1010 is shown as including executable instructions 1015. Executable instructions 1015 represent instructions executable by processor 1005 of computer system 1000 to perform disclosed operations, such as those described in various methods.

[0126] The disclosed embodiments may include or utilize a special-purpose or general-purpose computer including computer hardware, such as one or more processors (such as processor 1005) and system memory (such as memory 1010), as discussed in more detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are "physical computer storage media" or "hardware storage devices." In addition, computer-readable storage media include physical computer storage media and hardware storage devices, and do not include signals, carriers, and propagation signals. On the other hand, computer-readable media that carry computer-executable instructions are "transmission media," including signals, carriers, and propagation signals. Therefore, by way of example and not limitation, the current embodiments may include at least two distinct computer-readable media: computer storage media and transmission media.

[0127] Computer storage media (also known as "hardware storage devices") are computer-readable hardware storage devices such as RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") based on RAM, flash memory, phase change memory ("PCM") or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code in the form of computer-executable instructions, data or data structures and that can be accessed by a general purpose or special purpose computer.

[0128] The computer system 1000 may also be connected (via wired or wireless connections) to external sensors (e.g., one or more remote cameras) or devices via the network 1020. For example, the computer system 1000 may communicate with any number of devices or cloud services to acquire or process data. In some cases, the network 1020 may itself be a cloud network. In addition, the computer system 1000 may also be connected to remote / separate computer systems via one or more wired or wireless networks that are configured to perform any of the processing described with respect to the computer system 1000.

[0129] A "network," like network 1020, is defined as one or more data links and / or data switches for transmitting electronic data between computer systems, modules, and / or other electronic devices. When information is transmitted or provided to a computer over a network (hardwired, wireless, or a combination of hardwired and wireless), the computer properly considers the connection to be a transmission medium. Computer system 1000 will include one or more communication channels for communicating with network 1020. Transmission media include networks that can be used to carry data or desired program code devices in the form of computer executable instructions or in the form of data structures. In addition, these computer executable instructions can be accessed by general or special purpose computers. Combinations of the above should also be included within the scope of computer readable media.

[0130] Upon reaching various computer system components, program code in the form of computer executable instructions or data structures may be automatically transferred from transmission media to computer storage media (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a network interface card or "NIC") and then ultimately transferred to computer system RAM and / or less volatile computer storage media within the computer system. Thus, it should be understood that computer storage media may be included in computer system components that also (or even primarily) utilize transmission media.

[0131] Computer executable (or computer interpretable) instructions include, for example, instructions that cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform a specific function or group of functions. Computer executable instructions can be, for example, intermediate format instructions such as binary files, assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Instead, the described features and actions are disclosed as example forms of implementing the claims.

[0132] Those skilled in the art will appreciate that the embodiments can be implemented in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, etc. The embodiments can also be implemented in a distributed system environment, where local and remote computer systems linked by a network (by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) each perform tasks (e.g., cloud computing, cloud services, etc.). In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0133] The present invention may be embodied in other specific forms without departing from its characteristics. The described embodiments are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the above description. All changes within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A method for separating an image into a forward-sweep image and a reverse-sweep image, the image being acquired by a camera facing a display on which a micro-electromechanical scanning (MEMS) system is projecting rendered content, the method comprising: accessing a lookup table that maps (i) a MEMS projection position on the display to (ii) a corresponding pixel position in an image generated by a camera facing the display, wherein the MEMS projection position corresponds to a position on the display where the MEMS system projects content; accessing the image, the image generated by the camera; associating a first set of pixel positions in the image with a forward scanning sweep of the MEMS system using the lookup table; associating a second set of pixel positions in the image with a reverse scanning sweep of the MEMS system using the lookup table; generating the forward-swept image using the first set of pixel positions, wherein generating the forward-swept image is performed by extracting first pixel content from each pixel position of the image in the first set of pixel positions and including the extracted first pixel content in the forward-swept image; as well as The reverse-swept image is generated using the second set of pixel positions, wherein generating the reverse-swept image is performed by extracting second pixel content from each pixel position of the image in the second set of pixel positions and including the extracted second pixel content in the reverse-swept image.

2. The method of claim 1, wherein the method is performed by a wearable mixed reality (MR) system.

3. The method of claim 1, wherein the method is performed without blanking any portion of the display. The method of claim 1 , wherein the method is performed without displaying a predetermined pattern on the display. The method of claim 1 , wherein the texture level of the image is determined to meet or exceed a minimum texture level requirement.

6. The method of claim 1, wherein the image is generated after a complete image frame is projected by the MEMS system onto the display. The method of claim 1 , wherein the lookup table is generated during an initial calibration event of the MEMS system.

8. The method of claim 1, wherein the image comprises a plurality of rows of pixels, wherein the first pixel content is obtained from all even-numbered rows of pixels in the image, and wherein the second pixel content is obtained from all odd-numbered rows of pixels in the image.

9. The method according to claim 1, wherein the method further comprises: After the generation of the forward-sweep image and the reverse-sweep image, a vertical alignment operation is performed to vertically align the forward-sweep image and the reverse-sweep image.

10. The method according to claim 9, wherein the method further comprises: After performing the vertical alignment operation, a horizontal alignment operation is performed to horizontally align the forward-scanned image and the reverse-scanned image.

11. The method according to claim 10, wherein the method further comprises: Based on the horizontal alignment operation, determining an offset level existing between the forward-sweep image and the reverse-sweep image; setting the offset level as a parameter for a dual phase, the dual phase existing between the forward sweep image and the reverse sweep image; as well as A calibration event for the MEMS system is triggered to compensate for the dual phase.

12. The method of claim 1, wherein the display is one of a left-eye display or a right-eye display for a mixed reality (MR) system.

13. The method according to claim 1, wherein the method further comprises: determining an amount of texture in the image; determining that the amount of texture in the image is below a texture threshold; causing the MEMS system to add the texture to a subsequent projection of a subsequent display frame; causing the camera to generate a subsequent image of the subsequent display frame; and Based on the subsequent images, a new forward-sweep image and a new reverse-sweep image are generated.

14. The method of claim 1, wherein: After the generation of the forward-sweep image and the reverse-sweep image, the method includes performing a vertical alignment operation to vertically align the forward-sweep image and the reverse-sweep image, The vertical alignment operation is selected from one of: a line shift operation, a nearest neighbor operation, or an interpolation operation, and The vertical alignment operation restores missing content in one or both of the forward-scanned image or the reverse-scanned image.

15. The method of claim 1, wherein: The method further includes performing a horizontal alignment operation to horizontally align the forward-sweep image and the reverse-sweep image, and A confidence index is calculated to indicate how closely the back-sweep image is aligned with the forward-sweep image.

Images