Perspective correction vector graphics with gaze point rendering

Abstract

The present disclosure relates to perspective correction vector graphics with foveated rendering. Various implementations disclosed herein include devices, systems, and methods that implement a rendering process that performs vector graphics rendering based on information received from a source application. Various implementations disclosed herein include devices, systems, and methods that implement foveated rendering using content received from a source by selectively rendering the content for only some regions based on gaze.

Description

Technical Field

[0001] This disclosure generally relates to electronic devices for rendering vector graphics, and more particularly to systems, methods, and apparatus for providing foveated rendering for perspective-corrected vector graphics. Background Technology

[0002] Existing rendering techniques may utilize undesirable amounts of electronic device resources (e.g., CPU and / or GPU computation, time, power, etc.). Existing rendering techniques may fail to accurately or efficiently depict complex vector graphics (e.g., text). Furthermore, existing rendering techniques may fail to provide perspective-corrected graphics for content imported from existing applications and / or different environments or otherwise provided. For example, when imported into and therefore viewed within a 3D rendering environment, 2D content may not display with accurate perspective. Summary of the Invention

[0003] The various specific embodiments disclosed herein include devices, systems, and methods for performing a rendering process that uses content received from a source (e.g., a mobile device application) to perform foveated text rendering. For example, content displayed by an application on a mobile device screen may be displayed at a location within a 3D environment using a flat virtual surface. Instead of receiving a bitmap from a source (e.g., a mobile device application), the rendering process renders the content using vector drawing commands that define the content from the source. Furthermore, such vector drawing commands can be used to provide foveated rendering of the content from the source. In some specific embodiments, the source is an application that provides content using API calls to a rendering engine's API program. The rendering process can be configured to render such content from a source without requiring, for example, changing the source (e.g., a mobile device application) by alternatively modifying the rendering engine's API program. The rendering process can provide foveated rendering by drawing areas based on the viewing direction (e.g., drawing only some areas, drawing some areas at higher resolution / higher frame rates, etc.). In some implementations, the rendering process provides improved quality in the focal area, compatibility with applications running on existing electronic devices such as smartphones, tablets, or watches, awareness of perspective-corrected vector graphics and animations, and / or reduced computational and memory requirements for providing perspective-corrected rendering.

[0004] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: obtaining vector graphics drawing commands corresponding to content generated by a source for display in a first display environment, wherein the source provides first drawing commands to a drawing engine configured to display the content in the first display environment by generating raster graphics using the first drawing commands, wherein these vector graphics drawing commands are obtained from the drawing engine based on the first drawing commands. In some embodiments, the gaze direction of a user of an electronic device having a second display environment different from the first display environment is identified. Then, a first zone and a second zone of the display space corresponding to the display of the electronic device are identified. In some embodiments, a rendering mode for rendering content in the first and second zones is determined based on the gaze direction, and the content is rendered on the display based on the vector graphics drawing commands and the rendering mode for rendering content in the first and second zones.

[0005] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: obtaining vector graphics drawing commands corresponding to content generated from a source for display in a first display environment. In some specific embodiments, the user's gaze direction of the electronic device is identified, and a first zone and a second zone of the display space corresponding to the display of the electronic device are identified. A rendering mode for rendering content in the first and second zones is then determined based on the gaze direction, wherein content is rendered only in the first zone based on the first zone having a first rendering mode and the second zone having a second rendering mode. The content is then rendered on the display based on the vector graphics drawing commands and the rendering mode for rendering content in the first and second zones. Attached Figure Description

[0006] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0007] Figure 1 The illustration shows stereoscopic images of the left and right eyes provided by an electronic device according to some specific implementations when providing an extended reality (XR) environment.

[0008] Figures 2A to 2F The illustration shows an XR environment displayed using foveated rendering, based on some specific implementations.

[0009] Figures 3 to 4 This shows a layer view of virtual content displayed using foveated rendering, according to some specific implementations.

[0010] Figure 5 This is a flowchart illustrating exemplary rendering methods that perform vector graphics rendering in different ways using content received from a source application, depending on some specific implementation.

[0011] Figure 6 This is a flowchart illustrating an exemplary method, according to some specific implementation, of using content received from a source application to perform foveated rendering of content selectively drawn for only some regions based on foveation.

[0012] Figure 7 Exemplary electronic devices according to some specific implementations are shown.

[0013] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0014] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all the specific details set forth herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0015] Figure 1 This illustrates left and right eye stereoscopic images that provide 3D virtual content when viewed by a user of an electronic device. (Example) Figure 1 As shown, the left-eye image 110L and the right-eye image 110R each include multiple areas with different display qualities. For example, the left-eye image 110L and the right-eye image 110R may each include multiple areas with different resolutions. The left-eye image 110L and the right-eye image 110R can be rendered using a foveation point when the user's gaze is known. The first area or focal area 112L, 112R includes the focal area where the user's gaze falls or is fixed, rendered with a first quality (e.g., high resolution or detail). The second area 114L, 114R is near the first area 112 and is rendered with a second quality (e.g., medium resolution) lower than the first quality. The third areas 116L, 116R are the remaining portions of the rendered images in the left-eye display 110L and the right-eye display 110R, respectively, and are rendered with a third quality (e.g., lower resolution) lower than the second quality. For example, the third areas 116L, 116R are background areas. The second zones 114L and 114R can provide a smooth transition between the focusing zones 112L and 112R and the third zones 116L and 116R. In some specific implementations, the left-eye image 110L and the right-eye image 110R have more than three zones.

[0016] In some implementations, the size of the focus area is based on the user's gaze on the electronic device. In some implementations, the size of the focus area is based on the radians / angles covered by the user's gaze. For example, the radians covered by the gaze could be 20 degrees, or radians / angles between 5 and 20 degrees. Therefore, the focus area can be based on the radians covered by the gaze and the viewing distance to the rendered content. In some implementations, the size of the focus area is based on the amount of movement of the user's gaze (e.g., increased movement increases the size). In some implementations, the focus area includes any number of pixels (e.g., 800×800) rendered at high resolution by the display device. In some implementations, multiple areas with different rendering qualities are implemented using variable rasterization rate (VRR). For example, a VRR map of the rendered image can be used to track or update the VRR.

[0017] In some implementations, the focus areas 112L and 112R are larger than the focus region because the user's gaze involves a constant, very small movement. Making the focus area 112 larger than the focus region reduces the amount of time required to redraw the focus area 112 for multiple frames of the display device.

[0018] In some implementations, multiple zones with different corresponding rendering qualities use arbitrary values, resolutions, or scales. Furthermore, the rendering qualities of these multiple zones can be dynamically changed. For example, zones 112L and 112R, zones 114L and 114R, and zones 116L and 116R can be rendered at 10x, 4x, and 1x quality, respectively. In some implementations, zones 114L and 114R and zones 116L and 116R are rendered as images or textures.

[0019] Some specific implementations disclosed herein provide foveated rendering of content from a source (e.g., a user interface provided by a mobile device application) within another rendering environment (e.g., within a view of an XR environment). For example, content to be displayed by an application on a mobile device screen can be displayed on a flat virtual surface at a location within a 3D environment. Figures 2A to 2B This provides an example of a user interface from a source (e.g., an application on a mobile device) being rendered in another environment. Figures 2C to 2F The depiction shown is of content from a source displayed in another environment and may include perspective-corrected foveation rendering of the content.

[0020] Figure 2A The application's user interface is shown. For example... Figure 2A As shown, the application's user interface 210 includes various content. For example, the content on the user interface 210 may include a title section 222, text 224, images 226, and icons or links 228 to other content or applications.

[0021] Figure 2B An electronic device 250 is shown within a physical environment 205. In this example, the physical environment 205 is a room or office within a house. The electronic device 250 is configured to display views of the XR environment (e.g., view 215), which include images of the physical environment 205 (or otherwise generated based on images and / or other sensor data from that physical environment). Figure 2B As shown, user interface 210 is displayed in view 215 of the XR environment. In some specific implementations, electronic device 250 is any suitable device used to generate and display the XR environment, such as a smartphone, tablet computer, or wearable device.

[0022] In some implementations, virtual content (e.g., user interface 210) is displayed as a view relative to the physical environment 205, and is therefore displayed based on the defined and potentially fixed position and orientation of the virtual display surface relative to the physical environment. Thus, the relative position of the electronic device 250 relative to the display surface may depend on, and therefore change based on, the position of the electronic device 250 within the physical environment 205. In one position, the view of the electronic device 250 may include close-ups and front views of the virtual display surface, while in another position, the view of the electronic device may depict a view of the virtual display surface further away from oblique perspective. Therefore, the rendering of virtual content on the virtual display surface (e.g., the manner in which foveated rendering is performed) may depend on the position of the virtual display surface relative to the electronic device 250. The position of the electronic device 250 may be determined using various techniques, such as computer vision-based localization, simultaneous localization and mapping (SLAM), visual inertial measurement (VIO), etc., and may be based on sensor data, such as image data, depth data, motion data, audio data, etc.

[0023] Figure 2C The image illustrates gaze-point rendering of the user interface 210 in a 3D environment. This rendering can occur, for example, when a virtual display surface is positioned relative to the electronic device 250 in a relatively close-up and frontal position. Figure 2C As shown, the focus area 212 is rendered at a first arbitrary foveated scale (e.g., high resolution), the second area 214 surrounding the focus area 212 is rendered at a second arbitrary foveated scale (e.g., medium resolution), and the background area 216 encompasses the entire user interface 210 and is rendered at a third arbitrary foveated scale (e.g., low resolution). For example, the background area 216 may be a 2D image representing the entire user interface 210. Alternatively, the background area 216 may not be rendered at all.

[0024] As the user's gaze moves around the user interface 210, portions of the user interface 210 are re-rendered proportionally. Therefore, as the focus area 212 and the second area 214 move around the user interface 210 following the user's gaze, the portions of the user interface 210 that newly intersect with the moving focus area 212 and the moving second area 214 are re-rendered proportionally. Similarly, portions of the user interface 210 that no longer intersect with the moving focus area 212 and the moving second area 214 are re-rendered proportionally with the background area 216.

[0025] In some implementations, vector graphics are rendered with corrected perspective in focus area 212 because the intersection of focus area 212 with the XR environment can be determined in the 3D coordinates of the XR environment. Therefore, even at extreme angles between the user interface 210 and the user's gaze at the electronic device 250, the vector graphics in focus area 212 are correctly sampled at the appropriate pixel rate in each dimension. In other words, the user's gaze is projected onto the 3D XR environment to determine the intersection with the user interface 210 (e.g., focus area 212), which is used to dynamically determine any scale that provides a sufficiently high rendering resolution for the perspective-corrected vector graphics in the 3D XR environment (e.g., in focus area 112). In some implementations, 3D and other special visual effects may be applied to the foveated rendering of the vector graphics in user interface 210.

[0026] like Figure 2D As shown, the foveated rendering user interface 210 rotates about the x and y axes. This rendering can occur, for example, when a virtual display surface is positioned in a relatively close-up and tilted orientation relative to the electronic device 250. Figure 2E In this context, the user interface 210 is rotated horizontally by 80 degrees. This rendering can occur, for example, when the virtual display surface is positioned in a relatively close-up and tilted orientation relative to the electronic device 250. Figure 2E As shown, the focus area 212 (e.g., the gaze or the arc covered by the gaze) intersects more than half of the entire width of the user interface 210. Figure 2F In the image, the user interface 210 is shown in close-up and rotated 85 degrees relative to the electronic device 250 about a vertical axis. (Example) Figure 2F As shown, the focus area 212 (e.g., the gaze or the arc covered by the gaze) intersects more than half of the vertical (e.g., from top to bottom) of the user interface 210.

[0027] Figure 3 The diagram illustrates the orientation and front view of an exemplary vector graphic displayed using foveated rendering in the application's user interface. (Example:) Figure 3As shown, the vector graphics are content 310 (e.g., a PDF of text) that covers the user interface and includes a first focal area 312 and a background area 316. Therefore, two rendering layers are used for the gaze rendering of the content 310. Similarly, the viewer's gaze direction for the content 310, and therefore the focal area (FA), is smaller than the focal area 312. In some specific implementations, the focal area 312 may be a small portion of the content 310. For example, when reading, most people read a few words at a time in a single line of text, and therefore, the focal area 312 may include four or five words on that single line of text or include a line above and / or below that single line.

[0028] In some embodiments, content 310 is divided into multiple groups of pixels. In some embodiments, content 310 is divided into multiple pixels called tiles, each of which has the same shape. In some embodiments, content 310 is divided into multiple tiles with the same shape and size.

[0029] Figure 4 A layer view of multiple instances of an exemplary vector graphic displayed using foveated rendering is shown, showing the transition of these instances over time. Figure 4 As shown, content 310 (e.g., a PDF of text) is divided into multiple tiles (e.g., 42 tiles) drawn with dashed lines, each tile comprising multiple pixels for rendering content 310. Figure 4 As shown, the focus area 312 includes 9 tiles in the tile, and the background area 316 includes 33 tiles in the tile.

[0030] Figure 4 The diagram illustrates that as the user's gaze moves to the right while reading text in the PDF forming content 310, the focus area 312, including the FA, will move to follow the user's gaze. In one specific implementation, just as the focus area or focus area 312 crosses into tiles 20, 27, or 34 (or just before), those tiles are added to the focus area 312 (e.g., re-rendered at the scale of focus area 312) and tiles 17, 24, and 31 are discarded from the focus area 312 (e.g., re-rendered at the scale of background area 316). Figure 4 In the context of the focus area 312, the movement direction is from left to right; however, the movement direction can be from right to left, from top to bottom, or jumping from one part of the content to another.

[0031] Therefore, in some specific implementations, text or other vector graphics can be divided into layers (e.g., in...). Figures 3 to 4 The text or other vector graphics are rendered using perspective-corrected vector graphics only within the focus area 312 (which has two layers). Areas outside the focus area 312 (e.g., vector graphics) are irrelevant or rendered using techniques to reduce the amount of computation or memory used for rendering.

[0032] The various specific embodiments disclosed herein include devices, systems, and methods for performing a rendering process that uses content received from a source (e.g., a mobile device application) to perform foveated text rendering. For example, content displayed by an application on a mobile device screen may be displayed at a location within a 3D environment using a flat virtual surface. Instead of receiving a bitmap from a source (e.g., a mobile device application), the rendering process renders the content using vector drawing commands that define the content from the source. Furthermore, such vector drawing commands can be used to provide foveated rendering of the content from the source. In some specific embodiments, the source is an application that provides content using API calls to a rendering engine's API program. The rendering process can be configured to render such content from a source without requiring, for example, changing the source (e.g., a mobile device application) by alternatively modifying the rendering engine's API program.

[0033] When bitmaps are received from these sources (e.g., mobile device applications), the rendering process cannot render perspective-corrected content because drawing commands are not available. In this example, the bitmap received from the source is at a fixed resolution, and when the fixed-resolution bitmap is zoomed in or out (e.g., enlarged), the rendering process (e.g., using virtual surfaces in a 3D environment) cannot provide perspective content.

[0034] Therefore, the rendering process requires corresponding vector graphics drawing commands to render perspective-corrected complex vector graphics, such as text received from a source (e.g., a mobile device application), in a 3D environment. In some implementations, the rendering process receives vector graphics drawing commands from the source (e.g., a mobile device application) and renders all vector graphics themselves using virtual surfaces in the 3D environment (e.g., the source is not rendered). In some implementations, the perspective-corrected vector graphics rendering process modifies the API program called by the source to receive vector graphics drawing commands from the source. Therefore, since the same API is called (e.g., a call to a library), there is no need to change the source (e.g., the mobile device application). Therefore, in some implementations, the rendering process changes what the source sends via a vector graphics rendering request (via an API call) without changing the source itself. Thus, the rendering process now renders all vector graphics to render what is received from the source and can dynamically change the quality / resolution of the vector graphics. Additionally, the rendering process can scale the vector graphics for foveated rendering at appropriate times. In some implementations, the rendering process itself can use a rendering engine, such as subdivision techniques (e.g., see...). Figure 4 CPU-based rendering, raster graphics, etc.

[0035] In some specific implementations, the vector graphics rendering techniques described herein provide foveated rendering with perspective-corrected vector graphics rendering only within the focal region. Perspective-corrected vector graphics rendering is provided using dynamically changing orientation, size, and / or rendering quality within the focal region. Therefore, perspective-corrected vector graphics rendering is provided only when needed (e.g., Figure 4 The focal area (9 tiles) is included. Furthermore, the perspective-corrected vector graphics rendering process receives all vector graphics drawing commands for the XR environment. Therefore, the focal area can be provided at any scale because all vector graphics rendering is performed by the rendering process (e.g., a compositing server). In addition, the perspective-corrected vector graphics rendering process is compatible with applications running on existing electronic devices such as smartphones, tablets, watches, or desktops. In some specific implementations, the vector graphics rendering technique described herein uses a layered hierarchical structure (e.g., multiple regions including the focal area) to provide foveated rendering with perspective-corrected vector graphics rendering, which reduces the computational and memory requirements for rendering.

[0036] Vector graphics include paths defined by start and end points, as well as other points, curves, and angles along the way. In some implementations, vector graphics are defined based on vector graphics drawing commands that define one or more paths (e.g., lines, curves, or shapes specified by mathematical formulas) and / or specify the visual properties of the vector graphics material (e.g., color, texture, etc.). Vector graphics can be used to define the appearance of a wide variety of things, including but not limited to text strings, PDF files, fonts, 2D graphics, virtual objects, emojis, etc.

[0037] In some implementations, the 2D canvas is part of a 2D plane, and this canvas includes all the drawn content of the vector graphics. In some implementations, the 2D canvas can be distorted in 3D (e.g., by reflection, distortion, or mirroring) to provide a 3D effect for the displayed vector graphics in a 3D environment.

[0038] In one implementation, subdivision techniques (e.g., preprocessing) are used for foveated rendering of perspective-corrected vector graphics. Subdivision techniques can be used to reduce the number of computations required to render the graphics and / or reduce the number of curves used to represent the 2D vector graphics. Subdivision techniques enable the rendering of vector graphics in a 3D environment in real time (e.g., per frame). In some implementations, subdivision techniques (e.g., preprocessing) are performed in a first processor (e.g., CPU) of one or more electronic devices, and the 2D canvas is rendered by a second processor (e.g., GPU). In some implementations, subdivision techniques divide the canvas into multiple pixel regions (e.g., see [link to relevant documentation]). Figure 4In some implementations, subdivision techniques divide the canvas into multiple uniformly shaped pixel tiles. In one example, all tiles are the same size. Therefore, each tile is a part of the canvas.

[0039] Subdivision techniques (e.g., preprocessing) are used to determine a list of relevant drawing commands (e.g., drawing commands for paths that contribute only to each tile, and then drawing commands for only the portions of the paths that traverse the tiles). In some implementations, subdivision techniques transfer a data structure, which is a series of drawing commands, to a processor (e.g., a GPU) to render a canvas in a 3D environment.

[0040] In some implementations, the GPU shader renders vector graphics by processing the transmitted data structures. In some implementations, the GPU shader renders the pixels that form the vector graphics by determining (i) which tile contains pixels, (ii) which drawing commands (e.g., paths) are associated with that tile, and then determining the coverage (e.g., percentage of pixels (with specific color or material properties)), color, and composition (e.g., blending visual effects of partially covered pixels or rendered pixels). The GPU shader then repeats this process for the remaining pixels that form the vector graphics.

[0041] In some implementations, a canvas (e.g., vector graphics) is rendered for each frame displayed in the 3D environment. However, subdivision techniques are only performed when the content of the canvas changes. In other words, for each unchanged canvas, subdivision is processed only once. For example, for a fixed PDF vector graphic, subdivision is performed only once for the same document, but the PDF vector graphic can be re-rendered in every frame of the 3D environment. In some implementations, only relevant pixels in the canvas are rendered with each frame. For example, only pixels in the canvas that have changed are updated. In some implementations, changed pixels in the canvas are identified by comparing the canvas of the next frame with the canvas of the current frame.

[0042] Figure 5This is a flowchart illustrating an exemplary method for performing vector graphics rendering using content received from a source (e.g., an application). In some implementations, the 3D environment provides a view including foveated vector graphics and other content (e.g., an XR environment or a physical environment). For example, the source may be an application that includes API calls to a rendering engine (e.g., a rendering library). In some implementations, the display of content from the source can be changed by modifying the rendering library API program, eliminating the need to modify the source application itself. For example, instead of receiving a bitmap for text vector graphics (e.g., "hello world") from the source, the rendering process uses vector graphics drawing commands from the source to render foveated vector graphics capable of providing text vector graphics. The foveated rendering process can render perspective-corrected vector graphics in areas based on foveated selection (e.g., in focus areas at higher resolutions / higher frame rates, etc.). In some implementations, method 500 is performed by a device (e.g., Figure 7 Method 500 is executed by an electronic device 700. Method 500 can be executed using an electronic device, or by multiple devices communicating with each other. In some embodiments, method 500 is executed by a processing logic unit (including hardware, firmware, software, or a combination thereof). In some embodiments, method 500 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory). In some embodiments, method 500 is executed by an electronic device having a processor.

[0043] At box 510, method 500 obtains a vector graphics drawing command corresponding to content generated by a source for display in a first display environment, wherein the source provides the first drawing command to a drawing engine configured to display the content in the first display environment by generating raster graphics (e.g., bitmaps) using the first drawing command, wherein the vector graphics drawing command is obtained from the drawing engine based on the first drawing command. For example, the source is an application running on a smartphone that generates a view containing text for display in a first display environment, which is a display on the smartphone. The source / application provides the first drawing command to the drawing engine (e.g., via an API call), which is configured to display the content in the first display environment by generating raster graphics (e.g., bitmaps) of the text using the first drawing command. However, a newer drawing engine may alternatively obtain vector graphics drawing commands for text based on the first drawing command. For example, an API call to a newer drawing engine (e.g., an API program, a compositing server) does not change the source / application making the API call, but may be configured to use vector graphics drawing commands to render the text as vector graphics instead of raster graphics. For example, an updated drawing engine can obtain a list or container of drawing commands from the source based on the content (e.g., text) to be displayed in the second display environment.

[0044] At box 520, method 500 identifies the gaze direction of a user of an electronic device having a second display environment different from the first display environment. In some implementations, active or passive eye tracking (e.g., illumination) of the user of the electronic device is used to determine the gaze direction. The gaze direction may be compared with content generated by the source / application.

[0045] At box 530, method 500 identifies a first area and a second area of ​​the display space corresponding to the display of the electronic device. In some embodiments, the first and second areas are based on the user's gaze direction of the electronic device. For example, the first area may intersect with the gaze direction, and the second area may not intersect with the gaze direction. In some embodiments, the first area is a focus area, and the second area is a background area. In some embodiments, the size of the first area is based on the angle of the gaze point and the distance to the content. In one example, the display of the electronic device is divided into multiple tiles, and the first area is a first subset of the tiles based on the gaze direction, and the second area is the remaining tiles.

[0046] At box 540, method 500 determines the rendering mode for rendering content in the first and second zones based on the gaze direction. In one implementation, the rendering mode for the first zone is a perspective-corrected vector graphics rendering mode, and the rendering mode for the second zone is a 2D image or texture. Alternatively, the rendering mode for the first zone is 8x visual quality, and the rendering mode for the second zone is 1x visual quality. In one example, the content in the first zone is rendered at the display frame rate. In one example, the content in the second zone is rendered once or as it changes. For example, a tablet or HMD can compare a view of the application's current content with the first and second zones.

[0047] At box 550, method 500 renders content on the display based on vector graphics drawing commands and a rendering mode for rendering content in the first and second zones. In some embodiments, boxes 510 through 550 are repeated. In some embodiments, the techniques disclosed herein can be implemented on smartphones, tablets, or wearable devices such as head-mounted displays (HMDs) with optically transparent or opaque displays.

[0048] Figure 6This is a flowchart illustrating an exemplary method of foveated rendering using content received from a source, selectively rendering content for only some areas based on the gaze direction. In some implementations, content is not drawn outside the user's focus area on the electronic device. In some implementations, perspective-corrected vector graphics are redrawn in the focus area at the display device's frame rate. In some implementations, content is redrawn only in the focus area. In one example, the source is an application running on the electronic device. In some implementations, method 600 is performed by the device (e.g., Figure 7 Method 600 is executed by an electronic device 700. Method 600 can be executed using an electronic device, or by multiple devices communicating with each other. In some embodiments, method 600 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 600 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory). In some embodiments, method 600 is executed by an electronic device having a processor.

[0049] At box 610, method 600 obtains a vector graphics drawing command corresponding to content generated by the source for display in a first display environment. In some implementations, the source is a client / mobile device application. For example, the source could be an application running on a source electronic device (e.g., a smartphone or tablet) that generates a view including text and / or other virtual content (e.g., a user interface) for viewing.

[0050] At box 620, method 600 identifies the user's gaze direction on the electronic device. In some implementations, passive or active eye-tracking functionality on the electronic device is used to determine the user's gaze direction. For example, the electronic device may be an HMD that displays a view of the user interface of the mobile device application based on information generated by the current application providing displayable content for the mobile device application.

[0051] At box 630, method 600 identifies a first area and a second area of ​​a display space corresponding to the display of an electronic device. In some embodiments, the first area is the focal area of ​​the display, and the second area is a background area. In some embodiments, the first area is a first subset of a plurality of areas (e.g., pixel groups), and the second area is a second distinct subset of a plurality of areas outside the first subset of pixels (e.g., adjacent to or surrounding the first pixel subset). For example, pixel groups may be of the same shape and size (each group of pixels is a rectangular patch in the display space).

[0052] At box 640, method 600 determines a rendering mode (e.g., render or not render) for rendering content in a first region and a second region based on the gaze direction, wherein content is rendered only in the first region based on the first region having a first rendering mode and the second region having a second rendering mode. In some specific implementations, content is not rendered in the second region using the second rendering mode.

[0053] At box 650, method 600 renders content on the display based on vector graphics drawing commands and rendering modes for rendering content in a first area and a second area. In some embodiments, the first rendering mode renders only the content within the focus area of ​​the first area. In some embodiments, the first rendering mode renders content in the first area at a first resolution, and the second rendering mode renders content in the second area at a second, reduced resolution. In some embodiments, rendering in the first area is performed at a first frame rate, and rendering in the second area is performed at a second frame rate less than the first frame rate. For example, the content in the second area is rendered once via a second rendering process or rendered on change.

[0054] In some implementations, boxes 610 through 650 are repeated. In some implementations, the techniques disclosed herein can be implemented on smartphones, tablets, or wearable devices such as HMDs with optically transparent or opaque displays.

[0055] A physical environment refers to the physical world that people can interact with and / or sense without the aid of electronic systems. A physical environment can include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

[0056] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0057] In some implementations, the electronic device presenting the XR environment can be a single device that is handheld (e.g., mobile phone, tablet, laptop, etc.) or wearable (e.g., watch, head-mounted display (HMD), etc.). In some implementations, the functionality of the electronic device is achieved via two or more communication (e.g., wired or wireless) devices (e.g., additionally including optional base stations). Other examples include laptops, desktop computers, servers, or other such devices that include additional capabilities in terms of power, CPU capacity, GPU capacity, storage capacity, memory capacity, etc.

[0058] Figure 7This is a block diagram of an exemplary device 700. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, electronic device 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C, or similar interfaces), one or more programming (e.g., I / O) interfaces 710, one or more displays 712, one or more internal or external sensor systems 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0059] In some embodiments, the one or more communication buses 704 include circuitry for communication between interconnecting system components and control system components. In some embodiments, the one or more I / O devices and sensors 706 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time-of-flight, etc.) or similar devices.

[0060] In some embodiments, one or more displays 712 are configured to present content to a user. In some embodiments, one or more displays 712 correspond to display types such as holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), or the like. In some embodiments, one or more displays 712 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, electronic device 700 may include a single display. As another example, electronic device 700 may include displays for each of the user's eyes.

[0061] In some embodiments, one or more internally or externally oriented sensor systems 714 include an image capture device or array that captures image data or an audio capture device or array (e.g., a microphone) that captures audio data. The one or more image sensor systems 714 may include one or more RGB cameras (e.g., those with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, etc. In various embodiments, the one or more image sensor systems 714 also include an illumination source that emits light, such as a flash. In some embodiments, the one or more image sensor systems 714 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0062] Memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 720 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 720 optionally includes one or more storage devices remotely located to one or more processing units 702. Memory 720 includes a non-transitory computer-readable storage medium.

[0063] In some embodiments, memory 720 or a non-transitory computer-readable storage medium of memory 720 stores an optional operating system 730 and one or more instruction sets 740. Operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 740 includes executable software defined by binary information stored in charge form. In some embodiments, instruction set 740 is software executable by one or more processing units 702 to implement one or more of the techniques described herein.

[0064] In some specific implementations, instruction set 740 includes vector graphics generator 742, which can be executed by processing unit 702 to perform foveated perspective-corrected vector graphics rendering (e.g., focus area) according to one or more of the techniques disclosed herein.

[0065] Although instruction set 740 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside in a separate computing device. Figure 7This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, the actual number of instruction sets and the division of specific functions, and how features are allocated therein, will vary depending on the specific implementation, and in some implementations, depend in part on the specific combination of hardware, software, or firmware chosen for that particular implementation.

[0066] It should be understood that the specific embodiments described above are cited by way of example, and this disclosure is not limited to what has been specifically shown and described above. Rather, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to those skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

[0067] Those skilled in the art will recognize that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein. Furthermore, other effective aspects and / or variations do not include all the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings illustrate only some exemplary embodiments of this disclosure and should not be considered limiting.

[0068] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in certain circumstances, and the claimed combination may involve sub-combinations or variations thereof.

[0069] Similarly, although operations are shown in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the division of various system components in the above embodiments should not be construed as requiring such division in all embodiments, and it should be understood that the program components and the system may generally be integrated together in a single software product or packaged into multiple software products.

[0070] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes shown in the accompanying drawings do not necessarily require a specific order or sequence to achieve the desired result. In some specific embodiments, multitasking and parallel processing may be advantageous.

[0071] The embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or otherwise, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media can also be one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices), or included in one or more separate physical components or media.

[0072] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, systems-on-a-chip, or multiple or combinations thereof. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0073] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein can be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0074] Specific implementations of the methods disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Certain boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0075] The use of “applies to” or “configured to” in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Similarly, the use of “based on” implies openness and inclusivity, as processes, steps, calculations, or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included in this document are for illustrative purposes only and are not intended to be restrictive.

[0076] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term “comprising” as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0078] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A rendering method comprising: at a processor of an electronic device: obtaining vector graphics rendering commands corresponding to content generated by a source for display in a first display environment in which a content item is positioned in a two-dimensional (2D) region corresponding to a display surface, wherein the source provides first rendering commands to a rendering engine on the electronic device, the rendering engine configured to generate information to display the content in the first display environment by generating raster graphics using the first rendering commands, wherein the vector graphics rendering commands are obtained from the rendering engine based on the first rendering commands; identifying a gaze direction of a user of the electronic device having a second display environment in which the content item is positioned in a three-dimensional (3D) coordinate system and a view of the content item is provided based on a user-determined viewpoint within the 3D coordinate system, wherein the second display environment is different from the first display environment; identifying a first zone and a second zone of a display space corresponding to a display of the electronic device; determining a rendering mode for rendering the content in the first zone and the second zone based on the gaze direction; and rendering the content on the display based on the vector graphics rendering commands and the rendering mode for rendering the content in the first zone and the second zone.

2. The rendering method of claim 1, wherein the content is XR content including text.

3. The rendering method of claim 1, wherein the electronic device is a first electronic device, the source is an application executing on the first electronic device, and the first display environment is a display at the first electronic device.

4. The rendering method of claim 1, wherein the source provides the raster graphics to display the content in the first display environment through an API call to the rendering engine, and wherein the raster graphics are bitmaps.

5. The rendering method of claim 1, wherein the vector graphics rendering commands are generated from a list of rendering commands obtained by the rendering engine in the second display environment through an API call by the source.

6. The rendering method of claim 5, wherein the electronic device is a first electronic device, and the source is an application executing on a second electronic device different from the first electronic device.

7. The rendering method of claim 1, wherein the first zone is a focus zone based on the gaze direction zone, and the second zone is a background zone.

8. The rendering method of claim 1, wherein a first rendering mode in the first zone is a gaze point perspective correction rendering mode based on the vector graphics rendering commands.

9. The rendering method of claim 1, wherein a first rendering mode in the first zone is a first variable resolution rendering mode, and a second rendering mode in the second zone is a second fixed resolution rendering mode, wherein the second fixed resolution is always less than the first variable resolution. ​ 10. The rendering method of claim 1, wherein rendering in the first zone is performed at a frame rate of the display, and rendering in the second zone is performed at a second frame rate that is less than the frame rate of the display or on a change.

11. A rendering system comprising: a memory; and one or more processors coupled to the memory at an electronic device, wherein the memory comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: obtaining vector graphics rendering commands corresponding to content generated by a source for display in a first display environment in which a content item is positioned in a two-dimensional (2D) region corresponding to a display surface, wherein the source provides first rendering commands to a rendering engine on the electronic device, the rendering engine configured to generate information to display the content in the first display environment by generating raster graphics using the first rendering commands, wherein the vector graphics rendering commands are obtained from the rendering engine based on the first rendering commands; identifying a gaze direction of a user of the electronic device having a second display environment in which the content item is positioned in a three-dimensional (3D) coordinate system and a view of the content item is provided based on a user-determined viewpoint within the 3D coordinate system, wherein the second display environment is different from the first display environment; identifying a first zone and a second zone of a display space corresponding to a display of the electronic device; determining a rendering mode for rendering the content in the first zone and the second zone based on the gaze direction; and rendering the content on the display based on the vector graphics rendering commands and the rendering mode for rendering the content in the first zone and the second zone.

12. The rendering system of claim 11, wherein the content is XR content comprising text.

13. The rendering system of claim 11, wherein the electronic device is a first electronic device, the source is an application program executing on the first electronic device, and the first display environment is a display at the first electronic device.

14. The rendering system of claim 11, wherein the source provides the raster graphics to display the content in the first display environment through an API call to the rendering engine, and wherein the raster graphics are bitmaps.

15. The rendering system of claim 11, wherein the vector graphics rendering commands are generated from a list of rendering commands obtained by the rendering engine in the second display environment through an API call by the source.

16. The rendering system of claim 15, wherein the electronic device is a first electronic device, and the source is an application program executing on a second electronic device different from the first electronic device.

17. The rendering system of claim 11, wherein the first zone is a focus zone based on a gaze direction zone, and the second zone is a background zone. ​ 18. The rendering system of claim 11, wherein the first rendering mode in the first zone is a foveated perspective correction rendering mode based on a gaze point of the vector graphics rendering commands.

19. The rendering system of claim 11, wherein the first rendering mode in the first zone is a first variable resolution rendering mode and the second rendering mode in the second zone is a second fixed resolution rendering mode, wherein the second fixed resolution is always less than the first variable resolution.

20. A non-transitory computer-readable storage medium storing program instructions executable by one or more processors of an electronic device to perform operations comprising: obtaining vector graphics rendering commands corresponding to content generated by a source for display in a first display environment in which a content item is positioned in a two-dimensional (2D) region corresponding to a display surface, wherein the source provides first rendering commands to a rendering engine on the electronic device, the rendering engine configured to generate information to display the content in the first display environment by generating raster graphics using the first rendering commands, wherein the vector graphics rendering commands are obtained from the rendering engine based on the first rendering commands; identifying a gaze direction of a user of the electronic device having a second display environment in which the content item is positioned in a three-dimensional (3D) coordinate system and a view of the content item is provided based on a user-determined viewpoint within the 3D coordinate system, wherein the second display environment is different than the first display environment; identifying a first zone and a second zone of a display space corresponding to a display of the electronic device; determining rendering modes for rendering the content in the first zone and the second zone based on the gaze direction; and rendering the content on the display based on the vector graphics rendering commands and the rendering modes for rendering the content in the first zone and the second zone.

21. A computer program product comprising a computer program including program instructions executable by one or more processors to perform the method of any of claims 1-10.

Images