Image size triggered disambiguation to maintain image clarity

Abstract

Viewing systems and methods for viewing content SOLUTION: Disambiguation triggered by image size to maintain image sharpness is described. The present invention relates to a viewing system and a method of viewing content. The application engine renders an image based on image data from a content source. The display engine displays images to a user. The image is adjustable from a first size to a second size. A detection system detects the size adjustment. The application performs a disambiguation routine to maintain the sharpness of the image.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 768,705, filed November 16, 2018, and U.S. Provisional Patent Application No. 62 / 842,844, filed May 3, 2019, all of which are incorporated herein by reference in their entireties.

[0002] The present invention relates to a viewing system and a method for viewing content. [Background technology]

[0003] Personal computers typically have a standalone screen, in the form of a laptop screen or a separate screen, connected to a central processing unit. Applications that display images, such as browsers that display content in browser windows and web pages, word processing applications that display windows with text within the window, or the like, are designed to display such content so that a user can comfortably read the content at a distance of typically 0.25 meters to 1 meter. As the user moves further away from the display, the content simply becomes smaller in the user's view.

[0004] Three-dimensional viewing systems are also sometimes used to view content. The three-dimensional viewing system may have a head-mountable frame and left and right displays in front of the user's left and right eyes. The displays may generate images or projected images that are slightly different from each other to give the user the impression of depth. Alternatively, or in addition, the individual displays may have focal lengths adjustable to different depths.

[0005] The user may receive a browser window with a web page at a distance of, for example, two meters from the user, which may then be moved toward or away from the user, followed by user interaction with sensors, typically located on the head-mountable unit, to track the movement of the user's head and / or body parts. Summary of the Invention [Means for solving the problem]

[0006] The present invention provides a viewing system including: a content source for holding image data; an application engine forming part of an application, the application engine being communicatively connected to the content source, receiving the image data, and rendering an image based on the image data; a display engine being communicatively connected to the application engine for displaying the image to a user, the image having a first size, the image being adjustable to a second size smaller than the first size; and a detection system that detects measurements indicative of the adjustment of the image from the first size to the second size, the application performing a disambiguation routine in response to the detection to maintain image clarity.

[0007] The present invention also provides a method of viewing content, the method including the steps of: using an application engine, rendering an image based on image data from a content source; using a display engine, displaying the image to a user, the image having a first size, and the image being adjustable to a second size smaller than the first size; using a detection system, detecting measurements indicative of the adjustment of the image from the first size to the second size; and using an application, based on the detection, performing a clarity routine to maintain image clarity. The present invention provides, for example, the following items. (Item 1) 1. A vision system comprising: a content source for holding image data; an application engine forming part of an application, the application engine communicatively coupled to the content source, the application engine receiving the image data, and rendering an image based on the image data; a display engine communicatively connected to the application engine for displaying the image to a user, the image having a first size and the image being adjustable to a second size smaller than the first size; a detection system that detects measurements indicative of an adjustment of the image from the first size to the second size, the application, in response to the detection, performing a disambiguation routine to maintain clarity of the image; and A vision system comprising: (Item 2) 2. The vision system of claim 1, wherein the display engine displays the image to the user at a first virtual distance relative to the user, the image having the first size, the image being movable to a second virtual distance relative to the user, the image having a second size smaller than the first size, and the detection system is a movement detection system that detects movement of the image to the second virtual distance relative to the user. (Item 3) The application a frame size determiner that determines a selected frame size corresponding to the second virtual distance, the disambiguation routine being based on the frame size; Item 3. The vision system of item 2, comprising: (Item 4) The application a lookup table having a plurality of distances and a plurality of frame sizes corresponding to the respective distances; a lookup algorithm that determines a selected distance based on the second virtual distance from the lookup table, and determines that the selected frame size is a frame size corresponding to the selected distance; and Item 4. The vision system of item 3, comprising: (Item 5) 3. The vision system of claim 2, wherein the application executes the disambiguation routine only if the movement from the first virtual distance to the second virtual distance exceeds a predetermined minimum threshold distance. (Item 6) Item 6. The vision system of item 5, wherein the application executes the disambiguation routine only if a predetermined amount of time has elapsed after the movement from the first virtual distance to the second virtual distance exceeds a predetermined minimum threshold distance. (Item 7) 3. The vision system of claim 2, wherein the application executes the disambiguation routine only if the movement from the first virtual distance to the second virtual distance is below a predetermined maximum threshold distance. (Item 8) Item 10. The viewing system of item 1, wherein the image is adjustable from the first size to the second size by user interaction with the display engine. (Item 9) The application an application compositor, the application compositor receiving the selected frame size from the frame size determiner and the image from the application engine, and scaling the image based on the frame size; Item 4. The vision system of item 3, comprising: (Item 10) The application an application buffer that receives the image from the application engine, wherein an application compositor draws a subset of the image into the application buffer, the subset having a sample interval based on the selected frame size and containing fewer pixels than if the buffer were fully written; Item 10. The vision system of item 9, comprising: (Item 11) Item 11. The vision system of item 10, wherein sampling the subset requires fewer processing cycles and consumes less power than if the buffer were completely sampled. (Item 12) Item 11. The vision system of item 10, wherein the application buffer has a size that remains unchanged when the image is moved from the first virtual distance to the second virtual distance. (Item 13) Item 13. The viewing system of item 12, wherein the display engine performs a stretching operation in which the image is gradually adjusted from the first size to the second size in a series of cycles, with a smaller section of the buffer being drawn each cycle. (Item 14) 4. The viewing system of claim 3, wherein the application inputs the frame size into the application engine, and the application engine performs a re-layout of window area based on the frame size to maintain clarity of the image. (Item 15) Item 15. The viewing system of item 14, further comprising a stereoscopic viewer connected to the display engine and displaying the image to the user. (Item 16) Item 1. The vision system of item 1, wherein the image includes text. (Item 17) Item 10. The viewing system of item 1, wherein the display engine provides a screen size to the application engine, and a change in the screen size is the measurement indicative of an adjustment of the image. (Item 18) A method for viewing content, comprising: Rendering an image based on image data from a content source using an application engine; displaying the image to a user using a display engine, the image having a first size and the image being adjustable to a second size smaller than the first size; detecting, with a detection system, measurements indicative of an adjustment of the image from the first size to the second size; and using an application to perform a clarifying routine in response to said detection to maintain clarity of said image. A method comprising: (Item 19) Item 19. The method of item 18, wherein the display engine displays the image to the user at a first virtual distance relative to the user, the image having the first size, the image being movable to a second virtual distance relative to the user, the image having a second size smaller than the first size, and the detection system is a movement detection system that detects movement of the image to the second virtual distance relative to the user. (Item 20) The application determining a selected frame size corresponding to the second virtual distance using a frame size determiner of the application, and the disambiguation routine is based on the frame size; 20. The method according to item 19, comprising: (Item 21) The application determining a selected distance based on the second virtual distance from a lookup table using a lookup algorithm, the lookup table having a plurality of distances and a plurality of frame sizes corresponding to the respective distances; determining, using the lookup algorithm, that the selected frame size is the frame size corresponding to the selected distance; Item 21. The method according to Item 20, comprising: (Item 22) 22. The method of claim 21, wherein the application executes the disambiguation routine only if the movement from the first virtual distance to the second virtual distance exceeds a predetermined minimum threshold distance. (Item 23) 23. The method of claim 22, wherein the application executes the disambiguation routine only if a predetermined amount of time has elapsed after the movement from the first virtual distance to the second virtual distance exceeds a predetermined minimum threshold distance. (Item 24) 20. The method of claim 19, wherein the application executes the disambiguation routine only if the movement from the first virtual distance to the second virtual distance is below a predetermined maximum threshold distance. (Item 25) Item 19. The method of item 18, wherein the image is adjustable from the first size to the second size by user interaction with the display engine. (Item 26) The application receiving the selected frame size from the frame size determiner using an application composer of the application; scaling the image based on the frame size using an application compositor of the application; Item 21. The method according to Item 20, comprising: (Item 27) receiving the image from the application engine using an application buffer of the application, wherein the application compositor draws a subset of the image into the application buffer, the subset having a sample interval based on the selected frame size and containing fewer pixels than if the buffer were fully written; 27. The method of claim 26, further comprising: (Item 28) 28. The method of claim 27, wherein sampling the subset requires fewer processing cycles and consumes less power than if the buffer were completely sampled. (Item 29) 28. The method of claim 27, wherein the application buffer has a size that remains unchanged when the image is moved from the first virtual distance to the second virtual distance. (Item 30) 30. The method of claim 29, wherein the display engine performs a stretching operation in which the image is gradually adjusted from the first size to the second size in a series of cycles, with a smaller section of the buffer being drawn each cycle. (Item 31) typing the frame size into the application engine using the application; using the application engine to perform a window area re-layout based on the frame size that maintains the image clarity; 20. The method of claim 19, further comprising: (Item 32) displaying the image to the user using a stereoscopic viewer connected to the display engine; 32. The method of claim 31, further comprising: (Item 33) Item 19. The method of item 18, wherein the image includes text. (Item 34) Item 19. The method of item 18, wherein the display engine provides a screen size to the application engine, and a change in the screen size is the measurement that indicates an adjustment of the image. [Brief explanation of the drawings]

[0008] The invention will now be further described, by way of example only, with reference to the accompanying drawings in which:

[0009] [Figure 1] FIG. 1 is a block diagram of a vision system according to a first embodiment of the present invention.

[0010] [Figure 2A] 2A, 2B, and 2C are flow charts illustrating the functionality of the vision system of FIG. [Figure 2B] 2A, 2B, and 2C are flow charts illustrating the functionality of the vision system of FIG. [Figure 2C] 2A, 2B, and 2C are flow charts illustrating the functionality of the vision system of FIG.

[0011] [Figure 3A] FIG. 3A illustrates the image as it appears before being resized.

[0012] [Figure 3B] FIG. 3B shows the image after being resized and maintaining its sharpness.

[0013] [Figure 3C] FIG. 3C illustrates an image that has been resized without maintaining its clarity.

[0014] [Figure 4] 4A-4F illustrate rescaling aspects within the context of a browser window.

[0015] [Figure 5] FIG. 5 is a flow chart illustrating the steps that make the first embodiment work at a high level.

[0016] [Figure 6] FIG. 6 is a block diagram of a vision system according to a second embodiment of the present invention.

[0017] [Figure 7A]7A and 7B are flow charts illustrating the functionality of the vision system of FIG. [Figure 7B] 7A and 7B are flow charts illustrating the functionality of the vision system of FIG.

[0018] [Figure 8] FIG. 8 is a block diagram of a machine in the form of a computer that may find use in the system of the present invention, in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] 1 of the accompanying drawings illustrates a viewing system 10 according to one embodiment of the present invention, including a content source 12, a display engine 16, a stereoscopic viewer 18, and an application 20. The viewing system 10 may use a rescaling routine to maintain image clarity.

[0020] The display engine 16 and the application 20 are software systems that reside on computer-readable storage media and are executable by the processor of the vision system 10. The display engine 16 and the application 20 are communicatively coupled to each other through calls and subroutines as would be commonly understood by one of ordinary skill in the art.

[0021] The stereoscopic viewer 18 is a hardware component that is connected to the processor via a physical or wireless connection. The stereoscopic viewer 18 is also communicatively connected to the display engine 16.

[0022] The application 20 includes an application engine 14 , an application composer 30 , and a frame size determiner 32 .

[0023] The content source 12 is a hardware component connected to the processor. The content source 12 is also communicatively connected to the application engine 14. The content source 12 may contain image data. Such image data may be, for example, a still image such as a photograph or a document. The image data may alternatively be moving image data, such as a document or web page that moves as a user scrolls through the document or web page, or a series of frames of images that form a movie.

[0024] The application engine 14 may be, for example, an image viewer, a document editing application, a browser, a movie viewer, or the like, capable of rendering an image based on image data.

[0025] The display engine 16 determines a height (H) and width (W) of a field of view 22 that is specific to the stereoscopic viewer 18. The field of view 22 has a buffer texture 24 that has a frame size. "Frame size" is a term used in the art to generally refer to a combination of the number of horizontal pixels (X') and the number of vertical pixels (Y').

[0026] The application compositor 30 has an application buffer 28. The application compositor 30 draws images into the application buffer 28. The display engine 16 uses the coordinates of the buffer texture 24 to stretch the image in the application buffer 28 and then displays the image to the user.

[0027] Stereoscopic viewer 18 may have a frame mountable on the viewer's head with left and right displays in front of the viewer's left and right eyes. Depending on the configuration, the displays in front of the eyes may appear as a single display or may be configured to project two different images, one per eye. Stereoscopic viewer 18 receives images from display engine 16 and displays them to the viewer within field of view 22. Display engine 16 may display slightly different images to the user's left and right eyes to give the user the perception of depth. Display engine 16 may display each image in an adjustable depth field, adapting to vision-related enhancements such as "vergence-divergence accommodation," while also giving the user the perception of depth. Images displayed within buffer texture 24 are therefore perceived by the user as being at a specific virtual distance from the user.

[0028] The frame size determiner 32 includes the texture size calculator 26, a lookup table 34, and a lookup algorithm 36. The lookup table 34 has a plurality of distances in an X column and a plurality of frame sizes in a Y column, with each frame size corresponding to a distinct distance. The frame size may be represented, for example, by the number of pixels in a single direction (e.g., frame width, or frame height, or diagonal). The frame size may then be simply calculated, for example, by applying a fixed aspect ratio. The frame size decreases as the distance between the user and the virtual content increases. The lookup algorithm 36 receives as input the virtual distance between the user and the virtual content and determines a selected distance from the distances in the X column that corresponds to the virtual distance. The lookup algorithm 36 then determines a frame size that corresponds to the selected distance.

[0029] The lookup table 34 may have been generated by first displaying an image on the vision system 10 and then measuring the number of pixels the image occupies on the stereoscopic viewer 18. The image may be gradually moved away from the user, and the new frame size may be measured. The distances and corresponding pixel counts may be aggregated and placed in a table to form the lookup table 34. Other suitable methods of generating a lookup table may also be used. The lookup table 34 and lookup algorithm 36 may alternatively be replaced with an interpolation algorithm that may perform the same function as the lookup table 34 and lookup algorithm 36, but may use one or more interpolation algorithms, such as a spline interpolation algorithm or the like.

[0030] 2A, 2B, and 2C show the functionality of the vision system 10 in more detail. At 50, a lookup table 34 is stored in memory. As described above, the lookup table 34 has a plurality of distances and a plurality of frame sizes corresponding to the respective distances. Reference numeral 52 indicates that image data is maintained on the content source 12. At 54, the application engine 14 renders an image based on the image data.

[0031] At 56, the application buffer 28 receives the image from the application engine 14. The application buffer 28 has a predetermined size, and the image from the application engine 14 fills the application buffer 28.

[0032] At 60, display engine 16 displays an image to the user at a first virtual distance relative to the user, the image having a first size. Display engine 16 may, for example, display a browser window or document viewer window with a preset maximum surface area in a default location. The default location may be a preselected, user-friendly location that makes it easy for the user to view the image. The image may include text, images, and / or other visual content. If the image includes text, the image is preferably displayed at a default distance from the user so that the user can easily read the text.

[0033] The maximum surface may be the largest possible texture that the vision system 10 assigns to an image. The stereoscopic viewer 18 may have a resolution of 1,280 x 960 pixels. The vision system 10 may assign a maximum frame size for a given application, window, or type of content (e.g., a browser window). The browser application may be a landscape application (meaning it may be displayed simultaneously with other application windows). The browser application may not be an immersive application (one application window occupies the entire field of view), and therefore it may be assumed that the browser window will never require the entire field of view (e.g., 1,280 x 960); therefore, a value of 1,120 pixels (e.g., with a fixed aspect ratio) may be chosen because it is slightly smaller than 1,280 pixels, which may allow for a boundary of the field of view, or a small portion thereof, that would not contain the browser window. Selecting a maximum surface frame size greater than 1,280 pixels would exceed the resolution of the stereoscopic viewer 18, in which case the viewing system 10 may apply a minimum filter to match the display resolution when displaying content to the user, which may in turn result in unnecessary processing power to display an image of equivalent resolution (i.e., a maximum surface of 1,280 pixels becomes equivalent to 1,120 pixels after a minimum filter is applied to the 1,280 pixels).

[0034] The stereoscopic viewer 18 may have several systems that detect movement of the stereoscopic viewer 18 relative to real-world objects. For example, the stereoscopic viewer 18 may have an inertial measurement unit with one or more accelerometers that detect the acceleration of the stereoscopic viewer 18 and a gyroscope that detects its rotation. In addition, the stereoscopic viewer 18 may have a camera system that can capture images of real-world objects and track their movement within the view of the stereoscopic viewer 18. These systems provide the “intelligence” that enables detection of movement of the stereoscopic viewer 18 relative to real-world objects. As the stereoscopic viewer 18 moves relative to real-world objects, the images displayed by the display engine 16 may also move within the view of the stereoscopic viewer 18. For example, if an image is displayed to a user that gives the user the impression that the image is located on a wall, and the user walks closer to the wall, the depth at which the image is displayed also moves closer to the user, giving the user the impression that the image remains on the wall. The image will also be larger in the user's field of view.

[0035] It is also possible for the user to interact with the display engine 16 and increase the size of the image. The stereoscopic viewer 18 may have one or more cameras that may capture gestures made by the user's hands, for example, and the vision system 10 may interpret such gestures and increase the size of the image. The user may increase the size of the image by moving the image closer to the user, or may increase the size of the image while the image remains at a constant virtual distance relative to the user.

[0036] If the image becomes larger (e.g., the user walks toward the image), the rendered resolution may change (e.g., increase), but the size of the application buffer 28 may remain the same. Conventional systems may use a built-in resizing feature, broadly referred to as "mip-mapping," which may cause image blurring. Mip-mapping may include the use of filters well known in the art, such as "average filters," "Gaussian smoothing filters," "bilinear filters," and the like. For example, if the browser window moves further apart and fewer pixels are available to the browser window, an averaging method may be used in which four adjacent pixels in a first image are displayed as a single pixel in a second image. The single pixel is given an average pixel value between 0 and 255, representing the average of the four pixels originally displayed (in the first image), which may result in blurring in the second image.

[0037] At 62, the image is moved to a second virtual distance relative to the user, where the image has a second size that is smaller than the first size. In a given example, the user may move further away from the wall, and display engine 16 may maintain the impression that the image remains on the wall by moving the image further away from the user in the user's view, also causing the image to become smaller. Fewer pixels are now available to display the image. Traditional mipmapping techniques may take the four originally displayed pixels, average their pixel values, and display the averaged value as a single pixel. Such averaging of pixel values ​​may result in blurring.

[0038] A conventional system may be used for the purpose of displaying images on a stand-alone computer display. For example, a browser window may be displayed on a conventional stand-alone computer display. A user may then use controls to zoom in or out on a web page. Such zooming in and out on a web page may trigger mipmapping filtering techniques. As a user moves away from the stand-alone computer display, the browser window becomes smaller in the user's eye field of view and therefore becomes smaller in the user's view. The window also does not become smaller as a result of the user moving away from the stand-alone computer display, and no zooming occurs in the window. The window remains static relative to the stand-alone computer display and becomes smaller only in the user's eye field of view. As a result, the image in the display does not become blurrier as the user moves away from the stand-alone computer display.

[0039] At 64, the motion detection system of application 20 detects movement of the image to a second virtual distance relative to the user. The detected movement is a measurement that indicates the image is becoming smaller. The change in size (smaller or larger) of the image may require a clarifying routine to maintain image clarity (as opposed to blurring).

[0040] At 66, the user may additionally interact with display engine 16 to change the size of the image from a first size to a second size. At 68, the detection system detects a measurement indicating the adjustment from the first size to the second size. At 70, in response to the detection at 68, a magnification factor may be determined to account for the adjustment at 66. The magnification factor may ultimately be sent to subroutine 82 (optionally via subroutines 64, 74, 76, and / or 78), and the step of determining the frame size may be determined, at least in part, by the magnification factor. The magnification factor may be determined, for example, by calculating the ratio of the image displayed before and after the user manipulation. For example, the image may initially be displayed at 1,280 pixels (e.g., full resolution / full buffer). The image may then move away from the user and thus become 640 pixels (which may be determined by lookup table 34 based on the image's distance from the user). The user may then stretch the image larger by a factor of 1.5 (i.e., the image is now 50% larger than it was before), resulting in a post-adjusted image with 960 pixels at the same distance as the pre-adjusted image with 640 pixels. The user may now walk towards the image. Lookup table 34 may determine, based on the distance between the browser window and the user, that the image should be 750 pixels, and that after the scaling factor is applied, the screen should actually be 1,100 pixels. The scaling factors from lookup table 34 and provided by lookup algorithm 36 are therefore multiplied by the scaling factor. Depending on the particular situation, the image may be smaller at 60, larger at 66, smaller at 60, smaller at 66, remain unchanged at 60, enlarge or become smaller at 66, etc. For purposes of further discussion, the effect of user interaction at 66 is ignored for the detection of changes in virtual distance at 64, where the virtual distance may be the distance between the user and the virtual content / image.

[0041] At 74, a determination is (optionally) made as to whether movement from the first virtual distance to the second virtual distance exceeds a predetermined minimum threshold distance. At 76, a determination is (optionally) made as to whether a predetermined minimum threshold amount of time has elapsed between movement from the first virtual distance and movement of the second virtual distance. The predetermined minimum threshold distance may be, for example, 0.25 meters. The predetermined minimum amount of time may be, for example, 1 second or 1 minute. The thresholds referenced at 74 and 76 may be set to avoid experiencing jitter due to potentially overly frequent disambiguation updates.

[0042] At 78, a determination is (optionally) made as to whether the movement from the first virtual distance to the second virtual distance is below a predetermined maximum threshold distance. The predetermined maximum threshold distance may be, for example, four meters. If the second virtual distance is greater than four meters, there may be no need for a disambiguation routine to be performed because the image may be too small for the user to read. In particular, text on a web page or within a document may be too small for the user to see. Any image adjustment resulting from traditional mipmapping techniques is therefore acceptable because the blurring resulting from such techniques will not adversely affect the user experience. One or more of the thresholds at 74, 76, and 78 may be determined through user testing. Performing 74, 76, and 78 in a different order or eliminating one or more of the operations may also be possible.

[0043] In the described embodiment, if 74, 76, and 78 are all determined to be positive, the system proceeds to 82, where frame size determiner 32 determines a selected frame size corresponding to the second virtual distance. Various techniques may be employed to make the determination at 82, such as a technique based on the second virtual distance, a technique based on matching an appropriate rendering size, a technique based on the final rendering resolution, a technique based on the number of pixels available for content display, etc. The determination at 82 includes a determination at 84, where lookup algorithm 36 (I) selects a selected distance from lookup table 34 based on the second virtual distance, and (II) determines a selected frame size, which is the frame size corresponding to the selected distance. The selected frame size then forms the basis for the disambiguation routine.

[0044] Block 94 indicates that the buffer size remains unchanged from the buffer size referenced in block 56 .

[0045] At 98, application 20 executes a disambiguation routine that maintains image clarity based on the frame size allocated to the image when displayed to the user through stereoscopic viewer 18. Note that the disambiguation routine is responsive to detecting movement of the image to the second virtual distance at 64. At 100, application compositor 30 receives the selected frame size from frame size determiner 32. In this example, frame size determiner 32 has been provided with a selected frame size of 700.

[0046] At 102, texture size calculator 26 calculates the coordinates of the image based on the frame size. The vertical and horizontal coordinates may be adjusted by the same amount to maintain a fixed aspect ratio of the image.

[0047] At 104, the application compositor 30 draws the new image at the new image size into the application buffer 28. The size of the application buffer 28 remains unchanged, as indicated by block 94. In the given example where the frame size shrinks from 1,120 to 1,000, only 1,000 pixels are drawn into the application buffer 28, and a proportional adjustment is made to the vertical axis of the image. The updated image therefore has fewer pixels than if the entire buffer were used. By drawing fewer pixels into the application buffer 28, fewer processing cycles are required and less power is consumed than if the image were to fill the entire buffer.

[0048] At 105, updated image information such as pixel count, size, and / or resolution is sent to content associated with the application. Some examples of content associated with an application are application previews (e.g., a screenshot of the application that may appear upon hovering over an application icon), dialog boxes (e.g., a box stating that application content cannot be accessed or requesting a username and / or password), control boxes, information boxes, other popups, and / or any other widgets or views, or secondary windows that may be based, at least in part, on the resolution and / or size and / or pixel count of the application or application window. Content associated with an application may be, for example, a dialog box that allows a user to navigate files. Content associated with an application may not be content within the application window, but may be a secondary window, such as a parent-child application window, that may be triggered and controlled by the application window. Content associated with an application window may be a preview of the application window, such as a browser application that displays a preview of the browser window. The preview may be generated as a screenshot of the application window for use by the Application History feature and / or that may be displayed when the cursor hovers over the application icon. At 106, the application compositor 30 provides the overall buffer with the updated image (and corresponding texture / UV coordinates) to the display engine 16 for display to the user.

[0049] In some implementations, it may be useful to skip the routine at 105, which may result in a preview displaying the entire buffer (e.g., a screenshot image), including portions of the buffer in which no image had been drawn. Skipping routine 105 may therefore cause the system to not apply UV coordinates to content associated with the application window, and therefore cause the preview to duplicate the entire buffer frame (e.g., 1,000 x 1,000 pixels) and then draw the entire buffer frame at a smaller size (e.g., 50 x 50 pixels). The inclusion of routine 105 may allow the preview to duplicate only portions of the buffer frame that contain the updated image, thus preventing the preview from including portions of the buffer in which no image had been drawn.

[0050] In an application in which the content associated with the application is a dialog box and the application is a browser application that displays one or more browser windows, skipping routine 105 may result in the dialog box moving relative to the browser window each time the size and / or resolution of the browser window is updated. For example, the dialog box may start in the upper right corner of the application window and may appear in the center of the browser window if the distance between the user and the browser window increases. As an example, the browser window may have a first size of 1,000 x 1,000 pixels, and the dialog box may start at 200, 200 pixels. If a 1,000 x 1,000 pixel browser window is resized to 500 x 500 by all or part of the routines described in Figures 2A-2C, the dialog box would remain at 200, 200 if routine 105 were not executed. With inclusion of routine 105 in the flow, the dialog box would move to 100, 100 and maintain its relative location relative to the browser window.

[0051] At 88, display engine 16 performs a decompression operation. A decompression operation may be performed when an image is made larger or when an image is made smaller. A decompression operation may be performed when display engine 16 decompresses an updated image to fit the final display size as viewed by the user by utilizing texture coordinates and extracting the portion of the application buffer that the updated image fills. If an asynchronous frame update is performed, the process of performing the disambiguation routine at 98 occurs during the first frame update of the display engine, and the process of performing the decompression operation at 88 occurs during a different frame update of the display engine. In this case of asynchronous frame updates, the user may notice visual anomalies, such as jitter fluctuations in the image. To avoid fluctuations, the process of performing the decompression operation at 88 and the process of performing the disambiguation routine at 98 may occur within a single frame update of the display engine, i.e., a synchronous frame update. Synchronous frame updates, compared to asynchronous frame updates, provide a smoother visual transition when the displayed image updates from a first image at a first resolution to a second image at a second resolution.

[0052] After the display engine decompresses the updated image at 90, the updated image may be displayed to the user.

[0053] At 92, a new virtual distance or change in virtual distance is detected and the process loops back to step 74.

[0054] FIG. 3A illustrates the image before being resized. In FIG. 3B, the image has been made smaller. The image remains sharp due to the distorting routine 98 of FIG. 2C executed by application 20 of FIG. 1. FIG. 3C illustrates that the image becomes blurred if the distorting routine at 98 is not employed and instead a mipmapping routine is employed as described above. As noted above, the image may also be allowed to become blurred if a determination is made at 78 of FIG. 2B that the second virtual distance exceeds a maximum threshold distance.

[0055] Figures 4A-4F illustrate certain display aspects referenced above in more detail. In Figure 4A, as an example, a buffer has a frame size of 1,000 x 800 and has a texture that fills the entire space (1,1). In Figure 4B, the space in which the image will be displayed in the final rendered scene (to the user) has been reduced to a frame size of 900 x 750. In Figure 4C, the image fills the pixel space. In some embodiments, display engine 16 of Figure 1 may be required to apply a filter so that a larger texture (1,000 x 800) is displayed within a smaller browser window (900 x 750). Thus, there is a reduction in frame size. If the content is subsequently made smaller, the system will repeat Figures 4B and 4C, which is all that is required for a system running on a standalone computer display, without the disambiguation routine at 98 executed by application 20.

[0056] In FIG. 4D, the frame size (i.e., the number of pixels allocated to display the image to the user) is further reduced (700×600). As shown in FIG. 4E, the buffer size remains unchanged (1,000×800). However, the texture size, in some embodiments, is adjusted (800×700) according to lookup table 34. As shown in FIG. 4F, display engine 16 of FIG. 1 may apply filtering so that a larger texture (800×700) is displayed within a smaller browser window (700×600). A filter is still used in FIG. 4F, but the difference between the image size in the buffer and the final rendered size of the image is close enough that there is no reduction in image quality that a user may notice. In situations where a full buffer image (1,000×800) needs to be displayed at a relatively small final rendered size (e.g., 700×600), the filter may reduce image quality to a level that a user may notice. In some embodiments, Figures 4D-4F may be repeated, and the frame size may be updated as new distances between the user and the displayed image are measured. In some embodiments, the system may wait to adjust the image size until the image has not moved for a threshold time period, so that Figures 4D-4F need only occur once for a given movement of content relative to the user.

[0057] 5 is a high-level flowchart of the rescaling routine described above. At 200, application content is displayed in an application window (or prism) in the final rendering scene. At 202, the application window / prism is resized to the user. At 204, display engine 16 notifies application 20 of the distance between the user and the application window. At 206, rescaling of the content based on the distance between the user and the application window is performed. At 208, display engine 16 redisplays the application content.

[0058] Figure 6 illustrates a vision system 10A according to another embodiment of the present invention. The vision system 10A of Figure 6 is similar in many respects to the vision system 10 of Figure 1, with like reference numbers indicating like or similar components.

[0059] As discussed with reference to FIG. 1, display engine 16 provides the user with the distance to frame size determiner 32 of application 20. In FIG. 6, display engine 16A does not provide the user with the distance to frame size determiner 32A of application 20A. Instead, display engine 16A provides frame size determiner 32A with the screen size. The screen size may be, for example, the size corresponding to buffer texture 24A.

[0060] Frame size determiner 32A does not have a lookup table or lookup algorithm to determine texture size based on user distance. Instead, texture size calculator 26A uses the screen size received from display engine 16A to determine the desired texture size.

[0061] The application engine 14A receives texture sizes from the frame size determiner 32A. The application compositor 30A writes the image sizes into the application buffer 28A. The display engine 16A reads the images from the application buffer 28A and displays them to the user in the buffer texture 24A.

[0062] Figures 7A and 7B illustrate the functionality of the vision system 10A of Figure 6. Operations performed in Figures 7A and 7B are similar to those performed in Figures 2A, 2B, and 2C, and like reference numerals indicate like or similar operations. The algorithm represented by Figures 7A and 7B omits certain operations included in the algorithms of Figures 2A, 2B, and 2C, particularly the operations at blocks 62-78, block 84, and blocks 90 and 92.

[0063] The vision system 10 of FIG. 1 has the advantage that texture size can be adjusted to accommodate various viewing distances. However, viewing distance may not always be detected accurately or with a high level of accuracy. The vision system 10A of FIG. 6 has the advantage that the screen size is an exact size, resulting in a more accurately calculated final texture size displayed to the user. The screen size may still be adjusted based on detection of the user's movement. However, the screen size remains stable when the user is not moving, with less "drift" or inaccuracy associated with movement-based detection systems. The user therefore perceives less variability in the screen size displayed to the user. While the image can be adjusted from a first size to a second size using either the vision system 10 of FIG. 1 or the vision system 10A of FIG. 6, less jitter in the image size occurs using the vision system 10A of FIG. 6.

[0064] 8 shows a diagrammatic representation of a machine, in the exemplary form of a computer system 900, upon which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed according to some embodiments. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. Moreover, while only a single machine is illustrated, the term "machine" should also be taken to include any collection of machines that individually or together execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein.

[0065] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus 908.

[0066] The computer system 900 may further include a disk drive unit 916 and a network interface device 920 .

[0067] The disk drive unit 916 includes a machine-readable medium 922 on which one or more sets of instructions 924 (e.g., software) that embody any one or more of the methodologies or functions described herein are stored. The software may also reside, completely or at least partially, within the main memory 904 and / or within the processor 902 during its execution by the computer system 900; the main memory 904 and the processor 902 may also constitute machine-readable media.

[0068] The software may also be transmitted or received via the network interface device 920 over the network 928 .

[0069] The computer system 900 includes a laser driver chip 950, which is used to drive the projector and generate the laser light. The laser driver chip 950 includes its own data store 960 and its own processor 962.

[0070] In some embodiments, the computer system 900 and / or the vision system 10 may be all or part of a mixed reality system such as that described in U.S. Patent Application No. 14 / 331,218 (incorporated herein by reference).

[0071] While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are illustrative only and do not limit the invention, and that the invention is not limited to the specific construction and arrangement shown and described, as modifications may occur to those skilled in the art.

Claims

[Claim 1] The invention described in this specification.

Images