Adjusting video rendering rate and stereoscopic image processing of virtual reality content

Abstract

The present disclosure relates to adjusting video rendering rate and stereoscopic image processing for virtual reality content. An example technique can include performing video rendering at a first video rendering rate by updating an entire image on a screen of a computing device at a first update rate, via a virtual reality application provided on the computing device; determining that a performance of the video rendering is less than a threshold value; based on the determining, performing video rendering at a second video rendering rate by updating a first portion of the image at the first update rate and by updating a second portion of the image at a second update rate that is less than the first update rate. Another example technique can include shifting one or both of a left eye image and a right eye image during a blink period to reduce a disparity between a left view object and a right view object.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201680053664.1, filed on December 20, 2016.

[0003] Cross-reference to related applications

[0004] This application is a continuation-to-priority application to US Application No. 14 / 978,320, filed on December 22, 2015, the entire contents of which are incorporated herein by reference. Technical Field

[0005] This specification relates to image processing and rendering of virtual reality content, and more specifically, to rendering performance and processing based on stereoscopic images, and techniques for adjusting the video rendering rate of virtual reality content. Background Technology

[0006] Performing video rendering consumes significant computing resources. In some cases, multiple applications running on a computing device may share these resources, which can degrade the performance of one or more applications.

[0007] Furthermore, in the real world, there is rarely a conflict between accommodation demand and convergence demand. However, for VR (virtual reality) or stereoscopic images displayed on a screen, a difference or conflict sometimes exists between accommodation and convergence demands. For displays, accommodation demands are generally fixed because the user's eyes are focused on the display (e.g., the distance from the eyes to the screen is fixed). However, in some cases, the parallax (or distance or separation) between the left and right views of a stereoscopic image creates variable convergence demands, and in some cases, these convergence demands can differ from accommodation demands. This conflict between accommodation and convergence demands can cause eye strain for the user. Summary of the Invention

[0008] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate based on updating an entire image on a screen of the computing device at an update rate via a virtual reality application provided on a computing device; determining that the performance of the video rendering is less than a threshold; and, based on the determination, performing video rendering at a second video rendering rate by updating only a portion of the image at the update rate.

[0009] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate based on updating an entire image on a screen of the computing device at a first update rate via a virtual reality application provided on a computing device; determining that the performance of the video rendering is less than a threshold; and, based on the determination, performing video rendering at a second video rendering rate by updating a first portion of the image at the first update rate and by updating a second portion of the image at a second update rate less than the first update rate.

[0010] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate by a virtual reality application provided on a computing device; determining that the performance of the video rendering is less than a threshold; and, based on the determination, performing video rendering at a second video rendering rate by updating a first portion of the image at a first resolution and by updating a second portion of the image at a second resolution different from the first resolution.

[0011] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: receiving an encoded video signal; performing video rendering at a first video rendering rate to display a first set of display frames on a screen of the computing device based on the encoded video signal through a virtual reality application provided on a computing device; measuring the performance of the video rendering to display the first set of display frames; determining a second video rendering rate based on the measured performance of the video rendering, based on at least adjusting a portion of the screen to be used to display a second set of display frames; and performing video rendering at the second video rendering rate based on the encoded video signal to display the second set of display frames on the adjusted portion of the screen.

[0012] According to an exemplary embodiment, an apparatus may include at least one processor and at least one memory including computer instructions, which, when executed by the at least one processor, cause the apparatus to: receive an encoded video signal; perform video rendering at a first video rendering rate based on the encoded video signal to display a first set of display frames on a screen of the computing device via a virtual reality application provided on a computing device; measure the performance of the video rendering to display the first set of display frames; determine a second video rendering rate based on the measured performance of the video rendering, based on at least adjusting a portion of the screen to be used to display a second set of display frames; and perform video rendering at the second video rendering rate based on the encoded video signal to display the second set of display frames on the adjusted portion of the screen.

[0013] According to an exemplary embodiment, a method may include: receiving an encoded video signal; a virtual reality application provided on a computing device performing video rendering based on the encoded video signal to display a first set of display frames on a screen of the computing device; detecting the start of a blinking period of a user of the computing device; and during at least a portion of the blinking period, allocating computing resources from the virtual reality application to one or more non-virtual reality applications running on the computing device.

[0014] According to an exemplary embodiment, an apparatus may include at least one processor and at least one memory including computer instructions, which, when executed by the at least one processor, cause the apparatus to: receive an encoded video signal; perform video rendering based on the encoded video signal using a virtual reality application provided on a computing device to display a first set of display frames on a screen of the computing device; detect the start of a blinking period of a user of the computing device; and during at least a portion of the blinking period, allocate computing resources from the virtual reality application to one or more non-virtual reality applications running on the computing device.

[0015] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate based on updating the entire image on a screen of the computing device at an update rate via a virtual reality application provided on a computing device; detecting motion or movement of the screen; and performing video rendering at a second video rendering rate based on the detection by updating only a portion of the image at the update rate.

[0016] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate based on updating an entire image on a screen of the computing device at a first update rate through a virtual reality application provided on a computing device; detecting motion or movement of the screen; and performing video rendering at a second video rendering rate based on the detection by updating a first portion of the image at the first update rate and updating a second portion of the image at a second update rate different from the first update rate.

[0017] According to an exemplary embodiment, a computer-implemented method is provided for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: performing video rendering at a first video rendering rate by a virtual reality application provided on a computing device; detecting motion or movement of a screen; and, based on the detection, performing video rendering at a second video rendering rate by updating a first portion of the image at a first resolution and by updating a second portion of the image at a second resolution different from the first resolution.

[0018] According to another exemplary embodiment, a method may include: receiving an encoded video signal; performing video rendering based on the encoded video signal using a virtual reality application provided on a computing device to display a first set of display frames of virtual reality content on a screen of the computing device; detecting motion or movement of the computing device; and adjusting the frame rate of one or more display frames used to display the virtual reality content on the screen for at least a period of time in response to detecting the motion or movement.

[0019] According to another exemplary embodiment, an apparatus may include at least one processor and at least one memory including computer instructions, which, when executed by the at least one processor, cause the apparatus to: receive an encoded video signal; perform video rendering based on the encoded video signal to display a first set of display frames of virtual reality content on a screen of the computing device via a virtual reality application provided on the computing device; detect motion or movement of the computing device; and, in response to detecting the motion or movement, adjust the frame rate of one or more display frames for displaying the virtual reality content on the screen for at least a period of time.

[0020] According to another exemplary embodiment, a method includes: displaying a stereoscopic image including a right-eye image and a left-eye image on a screen of a computing device, the left-eye image and the right-eye image respectively illustrating one or more objects; determining viewing objects viewed by a user of the computing device, the viewing objects including a left-viewing object as part of the left-eye image and a right-viewing object as part of the right-eye image; measuring the parallax between the left-viewing object and the right-viewing object; detecting the start of a blinking period of the user of the computing device; and during the blinking period, shifting one or both of the left-eye image and the right-eye image to reduce the parallax between the left-viewing object and the right-viewing object.

[0021] According to another exemplary embodiment, an apparatus may include at least one processor and at least one memory including computer instructions, which, when executed by the at least one processor, cause the apparatus to: display a stereoscopic image including a right-eye image and a left-eye image on a screen of a computing device, the left-eye image and the right-eye image respectively illustrating one or more objects; determine viewing objects viewed by a user of the computing device, the viewing objects including a left-viewing object as part of the left-eye image and a right-viewing object as part of the right-eye image; measure the parallax between the left-viewing object and the right-viewing object; detect the start of a blinking period of the user of the computing device; and during the blinking period, shift one or both of the left-eye image and the right-eye image to reduce the parallax between the left-viewing object and the right-viewing object.

[0022] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the specification, drawings, and claims. Attached Figure Description

[0023] Figure 1 This is a block diagram illustrating a system according to an exemplary embodiment.

[0024] Figure 2 This is a diagram of a display screen 130, a portion of which is used to display frames, according to an exemplary embodiment.

[0025] Figure 3 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment.

[0026] Figure 4 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment.

[0027] Figure 5 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment.

[0028] Figure 6 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment.

[0029] Figure 7 This is a block diagram of a computing device according to an exemplary embodiment.

[0030] Figure 8 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment.

[0031] Figure 9 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment.

[0032] Figure 10 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment.

[0033] Figure 11 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment.

[0034] Figure 12 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment.

[0035] Figure 13 This is a diagram illustrating an exemplary conflict between adaptive and convergent requirements according to an exemplary implementation.

[0036] Figure 14 This is a diagram illustrating an example of reducing or eliminating the conflict between the adaptive and convergent needs of a viewed object according to an exemplary implementation.

[0037] Figure 15 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment.

[0038] Figure 16 Examples of general-purpose computer devices and general-purpose mobile computer devices that can be used with the techniques described herein are shown. Detailed Implementation

[0039] Figure 1 This is a block diagram illustrating a system 100 according to an exemplary embodiment. Reference Figure 1 Audio / video source 106 can generate and output audio and video signals that can be distributed or transmitted to one or more computing devices via network 104. In an exemplary embodiment, the audio / video output by audio / video signal source 106 can be provided as part of virtual reality (VR) content streamed or distributed to one or more computing devices. According to the illustrative exemplary embodiment, virtual reality (VR) (also referred to as immersive multimedia or computer-simulated life) can, at least in some cases, replicate or simulate the physical existence of an environment or space in the real world or an imagined world or environment to varying degrees. Network 104 can be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network. For example, computing device 105 can receive audio / video signals, which, in the illustrative exemplary embodiment, can be provided as part of VR content.

[0040] In various exemplary embodiments, computing device 105 may include, for example, VR glasses or VR goggles 110, which can provide a virtual reality (VR) experience to a user, for example, by allowing the user to view virtual reality content on a display screen (e.g., displaying images rendered from VR content) and listen to the sound of the virtual reality content from speakers. In an illustrative exemplary embodiment, VR goggles 110 may display stereoscopic images, including a left-eye image that can typically be viewed by the user's left eye and a right-eye image that can typically be viewed by the user's right eye. Computing device 105 may also include, for example, mobile device 108, laptop, notebook computer, PC, computer, portable or handheld computer or computing device, or any other computing device. Mobile device 108 may include, for example, a cellular phone, smartphone, PDA (personal digital assistant), or other mobile computing device. In an exemplary embodiment, computing device 105 may be a mobile device (e.g., a smartphone) that can be configured to provide or output VR content to a user while running one or more non-VR applications in parallel.

[0041] The computing device 105 may include a processor for executing or running instructions or software, memory for storing instructions and data, a display screen 130 (e.g., which may be a touch-sensitive screen or a touch screen) for displaying or outputting display frames or images or other information, speakers and microphones, and / or other input / output devices, such as those for displaying or outputting frames, images, or other information. The processor / CPU and memory of the computing device 105 may be... Figure 1 The computing resource 132 is shown as computing resource 132. Computing resource 132 may include CPU or processor resources, such as CPU cycles for executing instructions or processing data, and / or computer memory for storing instructions or data, as well as other computing resources.

[0042] refer to Figure 1 The computing device 105 may include a VR application 120, which can receive signals from an audio / video source 106 and can present or provide VR content to a user via one or more output devices of the computing device 105, such as a display screen 130, a speaker (not shown), or other output devices. For example, the display screen 130 may include an LCD (liquid crystal display) screen, a touch screen, or any other screen or display that displays images or information to a user. The computing device 105 may also include one or more sensors 128 for detecting motion or movement of the computing device 105, such as an accelerometer or any other sensor that can detect motion or movement or acceleration of the computing device 105 and / or the display screen 130.

[0043] VR application 124 may include, for example, a VR rendering module 122 for rendering audio / video content onto computing device 105, and a control module 124 for controlling the operation of VR rendering module 122 and / or VR application 120. VR rendering module 122 may receive audio / video signals from audio / video source 106 and may perform audio rendering and / or video rendering of virtual reality content. For example, VR rendering module 122 may receive audio signals from VR content and may output audio signals to a speaker (not shown) provided for a portion of computing device 105.

[0044] VR rendering module 122 can also receive encoded video signals from audio / video source 106 and can perform video rendering based on the encoded video signals to display a set (or more) of display frames on display screen 130 of computing device 105. In an illustrative exemplary embodiment, video rendering may include a process in which a computer or computing device processes information from an encoded data source and uses that information to generate and display images for display on a screen. For example, video rendering may include decoding the received encoded video signal, generating one or more display frames, and outputting each display frame to a frame buffer for output or display on display screen 130. Video rendering may include additional functions or tasks, depending on the type of encoded signal received. In some cases, video rendering may be computationally intensive or resource-intensive and may require significant computing resources 132 within computing device 105.

[0045] According to exemplary embodiments, video rendering may be performed by a processor or CPU that executes instructions or software to implement various tasks or functions associated with video rendering, such as video decoding, and / or by dedicated hardware such as a graphics processing unit (GPU). For example, if a GPU is present, some video rendering tasks can be offloaded from the main processor / CPU to the GPU. Computing resources 132 may include CPU resources, such as CPU execution time / cycles, memory, or any other resources within the computing device 105 that can be used to process data. In an exemplary embodiment, the computing device 105 may include a GPU.

[0046] The computing device 105 may also include a performance measurement module 126 for measuring the performance of the VR application 120, such as measuring rendering performance or determining video rendering rate, or other performance measurements of the VR rendering module 122 and / or the VR application. For example, rendering performance can be measured as the amount of data rendered per unit of time; for instance, rendering rate can be measured as pixels rendered per second (e.g., pixel rendering rate), bits rendered per second (e.g., bit rendering rate), frame rate, or frame rendering rate (e.g., frames per second) or other rendering performance measurements.

[0047] In an exemplary embodiment, control module 124 can compare the video rendering performance of VR application 120 with a threshold, and then adjust (e.g., increase or decrease) the video rendering rate of VR application 120 or VR rendering module 122 based on the comparison. For example, if the video rendering rate of VR application 120 is less than the threshold, control module 124 can determine that the video rendering rate can be updated to a lower level than the previous video rendering rate, for example, because VR application 120 does not meet or satisfy the threshold video rendering rate performance. In an exemplary embodiment, for example, if the measured video rendering rate is greater than the threshold rendering rate, control module 124 can determine that the video rendering rate can be updated or adjusted to a higher level than the previous video rendering rate. In other words, according to an exemplary embodiment, if the measured video rendering rate performance exceeds the threshold video rendering rate, the video rendering rate of VR application 120 is increased. As described in more detail below, VR application 120 can use a variety of different techniques to adjust or update the video rendering rate, such as: adjusting a portion of display screen 130 that can be used to display one or more images of VR content, and / or adjusting the update rate (on all or part of the display screen), and / or adjusting the resolution of displaying one or more images or display frames (all or part of the display screen). For example, different portions of an image can be updated at different update rates (e.g., the center portion versus the periphery), such as allowing the first (e.g., the center) portion of the image to be updated at a rate greater than the second (e.g., the periphery) portion of the image. Similarly, the first (e.g., the center) portion of the image can be updated at a first resolution (e.g., high resolution), while the second (e.g., the periphery) portion of the image can be updated at a second resolution lower than the first resolution. Other techniques may also be used. In this way, one or more exemplary techniques can be used to adjust the video rendering rate. According to an exemplary implementation, the refresh rate may be the rate at which the display screen is refreshed, and (at least in some cases) this refresh rate may be fixed based on the screen. The update rate may be the rate at which new image data is written to the display screen. The frame rate may be the rate at which frames are updated, which is generally the same as the refresh rate. Furthermore, in one exemplary implementation, the update rate and the frame rate may be the same rate.

[0048] As described above, computing device 105 may include one or more computing resources 132, such as CPU or processor resources, CPU cycles for executing instructions or processing data, and / or computer memory for storing instructions or data. According to an exemplary embodiment, allocating more available computing resources 132 to VR application 120 can allow VR application 120 to perform video rendering at a higher video rendering rate. On the other hand, if insufficient computing resources 132 are allocated to VR application 120, the performance of VR application 120 (e.g., including the video rendering rate implemented by VR application 120) may be prohibited or limited based on such limited computing resources 132. As a result, without sufficient computing resources 132 allocated to VR application 120, the video rendering performance of VR application 120 may not achieve, at least in some cases, the target or threshold video rendering rate.

[0049] In addition to VR application 120, computing device 105 may also include various non-VR applications 134. In an illustrative exemplary embodiment, non-VR application 134 may include any application that does not involve rendering or other processing signals to present VR content to a user of computing device 105. For example, non-VR application 134 may include, for example, email programs that send and receive emails, social media applications, weather applications that receive weather information, text or messaging applications that send or receive messages from other users, music players or music applications that play songs or music, web browsers that download web pages, or other applications that may run on or be available on computing device 105. In some cases, computing resources 132 may be allocated or shared among multiple applications running in parallel on computing device 105; for example, computing resources may be shared by, for example, VR application 120 and one or more non-VR applications 134.

[0050] Figure 2 This is a diagram of a display screen 130, a portion of which is used to display frames or images, according to an exemplary embodiment. (See reference...) Figure 2The full display boundary 210 of screen 130 can define the outer boundary of the entire area / part of screen 130 used for displaying images. The partial display boundary 212 of screen 130 identifies the boundary of the display area of ​​screen 130 that is smaller than the full display boundary 210. Therefore, the full display boundary 210 can provide the outer boundary including all pixels (picture elements) of the display screen 130, while the partial display boundary 212 can define the outer boundary for a subset (or less than all) of the pixels of the display screen 130. For example, a central portion 224 of the image can be provided within the partial display boundary 212, while a peripheral portion 222 of the image can be provided between the full display boundary 210 and the partial display boundary 212. Thus, for example, the entire area / part of screen 130 (within the full display boundary 210) can include the central portion 224 (within the partial display boundary 212) plus the peripheral portion 222 located between boundaries 210 and 212.

[0051] According to an exemplary embodiment, the control module 124 of the VR application 120 can measure the performance of video rendering performed by the VR rendering module 122. For example, the control module 124 can measure video rendering performance, such as by measuring the video rendering rate implemented by the VR rendering module 122 to display a set of frames. The control module 124 can also, for example, compare the video rendering performance (e.g., a measured first video rendering rate) with a threshold. For example, the control module 124 can determine a second or adjusted video rendering rate to be used to display a set of display frames as part of VR content based on the comparison of the measured first video rendering rate with a threshold. For example, the second or adjusted video rendering rate can be determined based on, for example, adjusting or changing a portion of the display screen 130 of the display frames (or selecting the adjusted portion). Although only one threshold is described in this illustrative example, multiple thresholds can be used to compare video rendering rates and then determine the selected or adjusted portion of the display screen used to display one or more display frames, for example, adjusting the video rendering rate or selecting a newer or second video rendering rate based on the comparison.

[0052] For example, according to an exemplary embodiment, if the video rendering performance (or the measured video rendering rate) is less than a threshold, the control module 124 may determine that the adjusted or second video rendering rate is less than the initial or first rendering rate. For example, a lower (or reduced) video rendering rate can be achieved by adjusting a portion of the display screen 130 used to display one or more display frames, or by selecting the adjusted portion. Reference Figure 2 According to an exemplary implementation, the full display boundary 210 ( Figure 2The display screen 130 within the VR application 120 can initially be used to display images of VR content at a first video rendering rate. In this illustrative example, in response to determining that the measured first video rendering rate is less than a threshold, the control module 124 can select an adjustment portion (or adjust that portion) of the screen 130, including a portion smaller than the entire display screen 130 (such as the central portion 224 within the partial display boundary 212), to display one or more display frames according to a second video rendering rate less than the first video rendering rate. In this way, for example, by at least reducing a portion of the screen 130 used to display frames (e.g., from a portion of the screen 130 within the full-screen display boundary 210 to a smaller portion of the screen 130 within the partial display boundary 212), this results in a reduction in the amount of data (or pixels per frame) to be rendered, for example, allowing the target frame rate to be achieved by the video rendering module 122, thereby reducing the video rendering rate of the VR application 120. In some exemplary embodiments, other steps may also be taken to reduce the video rendering rate (e.g., by reducing the frame rate and / or reducing the image or display frame resolution). This, for example, leads to a reduced video rendering rate, which may be useful or advantageous, for instance, if VR application 120 or VR rendering module 122 cannot achieve a threshold video rendering rate. As mentioned above, if one or more non-VR applications 134 consume or use a considerable amount of computing resources 132, it can result in poor performance of VR application 120 or VR rendering module 122. In such cases, for example, one solution could be to reduce the video rendering rate by, for instance, reducing the number of pixels rendered per display frame, thereby reducing the amount of data rendered per display frame, which, for example, allows the target frame rate to be achieved. This can be achieved, for example, by selecting an adjusted portion of screen 132 used to display one or more display frames (as an example, this could correspond to the central portion 224 within the partial display boundary 212).

[0053] For example, if the video rendering module 122 cannot achieve the target or threshold frame rate (frame rendering rate in frames per second) due to insufficient resources, the video rendering load (pixels per frame) of each (or one or more) display frames can be reduced by adjusting or selecting an adjustment portion of the display 130 (e.g., selecting a smaller portion of the display to reduce the number of pixels per display frame for display), which can reduce the overall video rendering rate.

[0054] According to another exemplary embodiment, for example, if the video rendering performance (or the measured video rendering rate) is greater than a threshold, the control module 124 can determine that the adjusted or second video rendering rate is greater than the initial or first rendering rate. In an exemplary embodiment, for example, while maintaining the same or similar frame rate, a larger (or increased) video rendering rate can be achieved by adjusting a portion of the display screen 130 used to display one or more display frames, or by selecting the adjusted portion. Reference Figure 2 According to an exemplary embodiment, a portion of the display boundary 212 ( Figure 2 The entire portion of the display screen 130 within the screen can initially be used to display images of VR content at a first video rendering rate. In this illustrative example, in response to determining that the measured first video rendering rate is greater than a threshold, the control module 124 can select an adjusted portion (or adjust, e.g., enlarge) of the screen 130, which may include an area or portion or more pixels larger than the portion within the partial display boundary 212, corresponding to the entire portion of the screen within the full display boundary 210, to display one or more display frames according to a second video rendering rate greater than the first video rendering rate. Thus, for example, the portion of the screen 130 used to display images can be enlarged by adding the peripheral portion 222 (provided between boundaries 212 and 210) to the central portion 224, so that the entire or entire portion of the screen within the full display boundary 210 can now be used to display images. Therefore, in this example, a higher or increased rendering rate can be achieved by increasing the number or amount of pixels rendered per display frame, for example, by increasing the portion of screen 130 used to display the frames, such as according to an exemplary embodiment (e.g., increasing a portion of the screen or the number of pixels due to the addition of the peripheral portion 222 between the partial display boundary 212 and the full display boundary 210), by increasing the selected portion of screen 130 from the central portion 224 of screen 130 within the partial display boundary 212 to the entire portion within the full display boundary 210. Other steps can also be taken to increase the video rendering rate, such as by increasing the frame rate used to display one or more display frames, and / or increasing the resolution of the image or display frame displayed on screen 130.

[0055] Additionally, an increased video rendering rate can typically be performed when sufficient resources (e.g., computing resources, memory resources, etc.) are available to handle such an increase in video rendering rate. According to another exemplary implementation, in response to determining that a first video rendering rate is less than a threshold, computing resources 132 can be allocated from one or more non-VR applications 134 to VR application 120. For example, this increases or improves the performance or video rendering rate achieved by VR application 120 or VR rendering module 122. Similarly, if the video rendering rate or video rendering performance of VR application 120 is greater than a threshold, in some cases, computing resources 132 can be allocated from VR application 120 to one or more non-VR applications 134, for example, at least for a period of time during which non-VR applications 134 are allowed to process some data before these resources are reallocated back to VR application 120.

[0056] In some cases, if video rendering performance is insufficient or fails to reach a minimum (or threshold) video rendering rate, users watching or experiencing VR content may experience significant latency or lag. For example, if VR application 120 or VR rendering module 122 performs poorly (e.g., fails to meet performance thresholds), users may experience significant lag or delay in the display of VR content changes or updates in response to user movement or actions. In some cases, significant latency in VR applications (e.g., in rendering VR content) can cause user nausea or aversion. Therefore, it is desirable to reduce the latency experienced by users where possible. In some cases, experiencing VR latency may be at least in part due to (e.g.) poor (or inappropriate) performance of VR application 120 or VR rendering module 122, caused by insufficient computing resources 132 available or allocated to VR application 120, for example, based on the use or demand of computing resources 132 by non-VR application 134. Therefore, according to exemplary embodiments, when the amount of data to be rendered is reduced, such as by selecting a smaller, adjusted portion of the display 130 (e.g., the central portion 224 within the partial display boundary 212) that may be smaller than the entire display (e.g., smaller than the area within the full display boundary 210) to display one or more images, the latency or lag of VR content perceived by the user can be reduced. In this way, improved rendering performance (e.g., lower latency in VR content output) can be achieved by reducing the video rendering load / burden per display frame, for example by adjusting or selecting an adjusted portion of the display 130 (such as the central portion 224 within the partial display boundary 212) that is smaller (fewer pixels) than the area within the full display boundary 210. Furthermore, at least in some cases, reduced latency in VR content output and / or improved performance of VR application 120 / VR rendering module 122 can be achieved by allocating computing resources 132 from one or more non-VR applications 134 to VR application 120 for at least a period of time.

[0057] Figure 3 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment. Operation 310 may include receiving an encoded video signal. Operation 320 may include performing video rendering at a first video rendering rate based on the encoded video signal by a virtual reality (VR) application provided on the computing device, to display a first set of display frames on the screen of the computing device. Operation 330 may include measuring the performance of the video rendering to display the first set of display frames. Operation 340 may include determining a second video rendering rate based on the measured performance of the video rendering, adjusting at least a portion of the screen to be used to display a second set of display frames. Furthermore, operation 350 may include performing video rendering at the second video rendering rate based on the encoded video signal to display the second set of display frames on the adjusted portion of the screen.

[0058] according to Figure 3 An exemplary implementation of the method shown may include: comparing the performance of video rendering with a first threshold; and if the performance of video rendering is less than the first threshold, determining that a second video rendering rate is less than a first rendering rate, wherein adjusting the screen portion includes reducing the portion of the screen that will be used to display the second set of display frames.

[0059] according to Figure 3 An exemplary implementation of the method shown further includes: comparing the performance of video rendering with a second threshold; and if the performance of video rendering is greater than the second threshold, determining that the second video rendering rate is greater than the first rendering rate, wherein adjusting the screen portion includes increasing the screen portion that will be used to display the second set of display frames.

[0060] according to Figure 3 An exemplary implementation of the method shown may include measuring the amount of time or processor cycles required to display the amount of data or multiple display frames.

[0061] according to Figure 3 An exemplary implementation of the method shown may include determining a second video rendering rate based on the measured video rendering performance, based on at least adjusting the screen portion to be used to display the second set of display frames, and further including determining a second video rendering rate based on the measured video rendering performance, based on at least adjusting the number of pixels in each display frame to be used to display the second set of display frames.

[0062] according to Figure 3 An exemplary implementation of the method shown further includes adjusting the frame rate for displaying at least a second set of display frames on the screen, in order to determine the second video rendering rate.

[0063] according to Figure 3 An exemplary implementation of the method shown may further include adjusting the display frame or image resolution used to display at least a second set of display frames on the screen.

[0064] according to Figure 3 An exemplary implementation of the method shown may further include: comparing the performance of video rendering with a first threshold; determining that the performance of video rendering is less than the first threshold; and allocating computing resources of the computing device from one or more non-virtual reality applications running on the computing device to virtual reality applications.

[0065] according to Figure 3An exemplary implementation of the method shown may further include: estimating the blink duration of a user on the computing device; and allocating computing resources from a virtual reality application to one or more non-virtual reality applications running on the computing device, at least during a portion of the blink duration. Figure 3 An exemplary implementation of the method shown may include estimation that includes at least one of the following: at least predicting the start of a blinking period of a user of the computing device; and detecting a blink or the start of a blinking period of a user of the computing device.

[0066] according to Figure 3 An exemplary implementation of the method shown may further include: detecting motion or movement of the VR display device; and, at least for a period of time after detection, adjusting (e.g., increasing or decreasing) the field of view used to display one or more display frames on the screen.

[0067] according to Figure 3 An exemplary implementation of the method shown may further include: detecting motion or movement of the computing device; and adjusting (e.g., increasing or decreasing) the frame rate used to display one or more display frames on the screen for at least a period of time after detection.

[0068] According to another exemplary embodiment, the apparatus may include at least one processor and at least one memory including computer instructions that, when executed by the at least one processor, cause the apparatus to: receive an encoded video signal; perform video rendering at a first video rendering rate based on the encoded video signal by a virtual reality application provided on a computing device to display a first set of display frames on a screen of the computing device; measure the performance of the video rendering to display the first set of display frames; determine a second video rendering rate based on the measured performance of the video rendering, based on at least adjusting a portion of the screen to be used to display a second set of display frames; and perform video rendering at the second video rendering rate based on the encoded video signal to display the second set of display frames on the adjusted portion of the screen.

[0069] According to exemplary embodiments, refresh rate (or frame rate) can refer to the rate at which screen 130 refreshes screen data. There is also update rate, which is the rate at which an image (or a new image) is updated / displayed on screen 130. Typically, refresh rate can be the same as image update rate. However, according to various exemplary embodiments, there may be situations where the update rate is adjusted (e.g., reduced) to decrease the video rendering rate (and rendering load) on computing device 105. In this case, the image update rate is less than the refresh rate of screen 130.

[0070] According to exemplary implementations, there are several exemplary scenarios where the video rendering rate / load can be reduced. For example, if the video rendering performance is below a threshold, one or more techniques can be used to adjust (e.g., increase or decrease) the video rendering rate / load, such as, in at least some cases, freeing up resources for other tasks, or selecting a video rendering rate more suitable for currently available resources for video rendering. Furthermore, in the event of motion detected on screen 130, one or more techniques can be used to adjust (e.g., increase or decrease) the video rendering rate.

[0071] Several other exemplary techniques for reducing video rendering speed will be briefly described. First, such as Figure 2 As shown, the display screen 130 can display an image including a central portion 224 and a peripheral portion 222. For example, the central portion 224 may be provided at least in or near the center of the image and may be provided within the partial display boundary 212. The central portion 224 of the image is generally more likely to fall around the user's fovea. The peripheral portion 222 of the image is less likely to fall near the user's fovea. The fovea is a small depression in the retina of the eye with the highest visual acuity. The center of visual field is concentrated in this area, where, in particular, the retinal cone is concentrated. Therefore, according to various exemplary embodiments, techniques can be used to reduce the video rendering rate, which can take advantage of the lower visual acuity outside the fovea, for example, for the peripheral portion 222 of the image.

[0072] According to a first exemplary embodiment, video rendering rate / load can be reduced by updating only a portion of the image. For example, to reduce video rendering rate, only the central portion 224 of the image is updated, while (e.g., at least for a period of time) the peripheral portion 222 is not updated. Therefore, since fewer pixels need to be updated, video rendering load / rate may be reduced.

[0073] According to a second exemplary embodiment, the video rendering rate can be reduced by updating a first portion of an image at a first update rate and updating a second portion of the image at a second update rate less than the first update rate. For example, the central portion 224 of the image (e.g., at least a portion located near the user's fovea) can be updated at the first update rate (e.g., in at least some examples, it can be the same as the refresh rate), while the peripheral portion 222 of the image is updated at a second update rate less than the first update rate. Thus, in this example, the peripheral portion 222 of the image can be updated at a lower rate (or not at all for a period of time) compared to the central portion 224 of the image, which can reduce the overall video rendering rate / load. In this example, the reduced update rate used for the peripheral portion 222 is not noticeable to the user because, in at least some cases, the peripheral portion 222 (or at least a portion thereof) can be further away from the user's fovea, for example, assuming the user is viewing or looking at a point in the central portion 224.

[0074] According to a third exemplary embodiment, the video rendering rate can be reduced by using different resolutions for different parts of an image. For example, a reduced video rendering rate can be achieved by updating a first part of the image (e.g., the central part 224) at a first resolution and updating a second part of the image (e.g., the peripheral part 222) at a second resolution lower than the first resolution. These various techniques, as well as other techniques described herein, can also be combined in various combinations. Thus, using a lower resolution to update a part of the image used for video (e.g., the peripheral part 222) will reduce the video rendering rate.

[0075] Figure 4 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment. Operation 410 includes performing video rendering at a first video rendering rate by means of a virtual reality application provided on the computing device, based on updating the entire image on the screen of the computing device at an update rate. Operation 420 includes determining that the performance of the video rendering is less than a threshold. Furthermore, operation 430 includes, based on this determination, performing video rendering at a second video rendering rate by updating only a portion of the image at an update rate.

[0076] according to Figure 4 An exemplary implementation of the method, performing video rendering at a second video rendering rate, may include: based on the determination, performing video rendering at a second video rendering rate by updating only the central portion of the image at an update rate, without updating the peripheral portion of the image.

[0077] Figure 5This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment. Operation 510 includes performing video rendering at a first video rendering rate by a virtual reality application provided on the computing device, based on updating the entire image on the screen of the computing device at a first update rate. Operation 520 includes determining that the performance of the video rendering is less than a threshold. Operation 530 includes performing video rendering at a second video rendering rate based on this determination, by updating a first portion of the image at the first update rate and by updating a second portion of the image at a second update rate less than the first update rate.

[0078] according to Figure 5 An exemplary implementation of the method, performing video rendering at a second video rendering rate, may include: based on the determination, performing video rendering at a second video rendering rate by updating the central portion of the image at a first update rate and by updating the peripheral portion of the image at a second update rate less than the first update rate.

[0079] according to Figure 5 An exemplary implementation of the method, performing video rendering at a second video rendering rate, may include: based on the determination, performing video rendering at a second video rendering rate, where the second update rate is less than the first update rate, by updating the central portion of an image at a first update rate and a first image resolution, and updating the peripheral portion of an image at a second update rate and a second image resolution less than the first image resolution.

[0080] Figure 6 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment. Operation 610 includes performing video rendering at a first video rendering rate by a virtual reality application provided on the computing device. Operation 620 includes determining that the performance of the video rendering is less than a threshold. Operation 630 includes: based on the determination, performing video rendering at a second video rendering rate by updating a first portion of an image at a first resolution and by updating a second portion of the image at a second resolution less than the first resolution.

[0081] according to Figure 6 In an exemplary implementation of the method, the first part may include the central portion of the image, while the second part may include the peripheral portion of the image.

[0082] Figure 7 This is a block diagram of a computing device 105 according to an exemplary embodiment. According to the exemplary embodiment, computing device 105 ( Figure 17) may include a VR application 120 and one or more non-VR applications 134 that can share computing resources 132. In some cases, due to the demanding nature of video rendering and / or other tasks associated with the operation of VR application 120, in exemplary embodiments, it is desirable to allocate all or at least most of the computing resources 132 to VR application 120 when VR application 120 is running and / or performing video. However, by allocating all or most of the computing resources 132 to VR application 120, the operation of one or more non-VR applications 134 may be completely or at least partially prohibited during the period when VR application 120 is running, for example, because during that period, sufficient computing resources 132 may not be available for such non-VR applications 134.

[0083] Furthermore, according to an exemplary implementation, at least some computing resources 132 can be allocated from VR application 120 to one or more non-VR applications 134 to allow non-VR applications 134 to perform application tasks / process data. However, as mentioned above, allocating resources from VR application 120 to non-VR applications 134 reduces the performance of VR application 120, including, for example, reducing the video rendering performance of VR rendering module 122, which unfortunately introduces or causes significant latency or degrades the user's VR experience.

[0084] Therefore, according to an exemplary embodiment, for example, computing device 105 can detect the start of a blinking period during which the user blinks or closes their eyes. According to an exemplary embodiment, during the blinking period, resources can be allocated from VR application 120 to one or more non-VR applications 134 to allow these non-VR applications 134 to operate or process data during at least a portion of the user's blinking period. According to an exemplary embodiment, although allocating computing resources 132 from VR application 120 to non-VR applications 134 during the blinking period reduces the performance of VR application 120 and / or increases the latency of VR rendering of VR content, the user of computing device 105 typically cannot perceive this increased latency (or reduced rendering performance) because the user closes their eyes during the blinking period. Therefore, for example, the blinking period can be used to allocate resources to non-VR application tasks and / or allow non-VR applications 134 to operate or process data.

[0085] For example, the resource allocation module 716 may allocate computing resources 132 from VR application 120 to one or more non-VR applications 134 during at least a portion of the user's blink time. Similarly, during at least a portion of the user's blink time, VR application 120 and / or VR rendering module 122 may reduce the video rendering rate of VR content, and / or may even stop or pause video rendering. For example, this may allow computing resources 132 to be allocated to non-VR applications 134 during blink time and / or may allow non-VR applications 134 to process data.

[0086] like Figure 7 As shown, computing device 105 may include one or more cameras 718 or other sensors that can be used to capture or receive images, pictures, videos, or other signals from (or in association with) a user's eyes, for example, to determine or detect the start of a blinking period (e.g., when the user temporarily closes their eyes). According to an exemplary embodiment, the image or picture of the user's eyes captured or received by camera 718 or other sensors may be forwarded to blink estimation module 710. Blink estimation module 710 may include image processing logic or software capable of detecting or estimating when the user blinks (e.g., temporarily closes their eyes). Blink estimation module 710 may include blink detection module 714 for detecting, for example, changes in the color of the user's eyes based on images or pictures received from camera 718 or sensors, using image processing logic or software, to detect the start of a blinking period (or the user's eye closing / closing).

[0087] The blink estimation module 710 may also include a blink prediction module 712 for predicting when the user will blink and / or predicting or estimating the duration of the user's blinks. For example, based on images or videos of the user's eyes received from camera 718 or other sensors, the blink prediction module 712 may determine various statistics related to the user's blinking (or temporary closing / closing of his or her eyes), such as the user's average blink rate (e.g., blinks per minute, such as 23 blinks per minute), average blink duration (e.g., the duration or period of eye closure during a blink, such as 200 ms), average time between blinks (e.g., 7 seconds), and / or other statistics. According to an exemplary embodiment, the blink prediction module 712 may predict or estimate when the user might blink next, or, for example, determine the most likely next point in time when the user will blink based on various blink statistics. For example, in this illustrative example, the probability that the user will blink generally increases as the time interval or period between blinks increases and approaches (and / or even exceeds) the user's average time between blinks.

[0088] Therefore, according to an exemplary embodiment, the blink detection module 714 can detect the start of a blinking period, for example, by detecting eye closing / closing. The blink prediction module 712 can predict the length of the blinking period. In response to a notification or indication of the start of a blinking period, the resource allocation module 716 can, for example, allocate at least some of the computing resources 132 from the VR application 120 to one or more non-VR applications 134 during at least a portion of the blinking period. Additionally or alternatively, the VR application 120 and / or the VR rendering module 122 can reduce the rate of video rendering during at least a portion of the blinking period, or even pause or stop video rendering, and during at least a portion of the blinking period, one or more non-VR applications 134 can run or operate and process data.

[0089] According to the illustrative example, VR application 120 and VR rendering module 122 may operate and perform video rendering based on all or a significant portion of computing resources 132 during an initial period, while one or more non-VR applications 134 do not run and / or receive significant computing resources 132 during this initial period. For example, during this initial period, the operation of email applications and / or SMS applications (or other non-VR applications) may be suspended (temporarily stopped) or slowed down, for example, to allow more computing resources 132 to be allocated to VR application 120. This could increase the video rendering performance of VR application 120 during this initial period, thereby reducing the latency of VR application 120 perceived by the user during this initial period. Thus, according to the illustrative example, during this initial period, based on the fact that no (or fewer) computing resources are allocated to email applications and / or SMS applications, for example, the email application does not send or receive emails, and the SMS application does not send or receive received text messages (or sends / receives at least fewer emails and fewer text messages). Therefore, for example, during this initial period, the operation of one or more non-VR applications (such as email, messaging / SMS applications, etc.) can be suspended (e.g., temporarily stopped / paused) or reduced, for example, to allow, for example, more resources to be allocated for video rendering and / or allocated to VR application 120 and / or VR processing (such as video rendering).

[0090] When the blink detection module 714 detects the start of a blinking period of the user of the computing device 105, for example, during at least a portion of the blinking period, the resource allocation module 716 may allocate at least some computing resources 132 from the VR application 120 to one or more non-VR applications, such as email and SMS applications, and web browsers. For example, a user may blink (e.g., temporarily close / close his / her eyes) for a period of approximately 50ms to 400ms, for example, an average of approximately 200-300ms. These figures are merely illustrative examples, and the blinking duration can be different time periods. Therefore, during a 300ms blinking period (as an illustrative example), one or more non-VR applications may resume processing; for example, an email application may send and / or receive multiple emails, and an SMS / messaging application may send and / or receive multiple texts or messages that have been pending processing. Additionally, during at least a portion of the blinking period, the VR application 120 may reduce the video rendering rate, or may even pause or temporarily stop (or suspend) the operation or VR rendering (e.g., during the blinking period, allocate resources previously used for VR processing / video rendering to non-VR applications). According to an exemplary implementation, any increased latency or degraded performance (or even no performance) of the VR application 120 during the blink period may be undetectable / imperceptible or unseen by the user because the user's eyes are closed during the blink period. At or near the end of the blink period, the resource allocation module 716 may, for example, reallocate some computing resources 132 from the email and SMS applications (non-VR application 134) back to the VR application 120, for example, to allow the VR application 120 / VR rendering module 122 to increase its video rendering rate and / or continue video rendering at an acceptable rate, as the user's eyes are now open and any latency or degraded performance of the VR application 120 may be perceptible to the user.

[0091] Figure 8 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment. Operation 810 may include receiving an encoded video signal. Operation 820 may include a virtual reality application provided on the computing device performing video rendering based on the encoded video signal to display a first set of display frames on the screen of the computing device. Operation 830 may include detecting the start of a blinking period of a user of the computing device. Furthermore, operation 840 may include allocating computing resources from the virtual reality application to one or more non-virtual reality applications running on the computing device during at least a portion of the blinking period.

[0092] according to Figure 8An exemplary implementation of the method shown may include at least one of the following: detecting at least the start of a blinking period of a user of the computing device; and estimating the blinking period of a user of the computing device.

[0093] according to Figure 8 An exemplary implementation of the method shown may further include: during at least a portion of a blink period, a virtual reality application suspends video rendering; and during at least a portion of a blink period, one or more non-virtual reality applications running on a computing device perform one or more non-video rendering tasks.

[0094] according to Figure 8 An exemplary implementation of the method shown may further include: reducing the video rendering rate by a virtual reality application during at least a portion of the blink time; allocating at least some computing resources from the virtual reality application to one or more non-virtual reality applications running on a computing device during at least a portion of the blink time; and performing one or more non-video rendering tasks by one or more non-virtual reality applications running on the computing device during at least a portion of the blink time.

[0095] according to Figure 8 An exemplary implementation of the method shown may further include performing the following steps at or near the end of a blinking period: reallocating at least some computing resources from one or more non-virtual reality applications running on a computing device to a virtual reality application; and increasing the video rendering rate by the virtual reality application.

[0096] According to another exemplary embodiment, the device may include at least one processor and at least one memory including computer instructions that, when executed by the at least one processor, cause the device to: receive an encoded video signal; perform video rendering based on the encoded video signal by a virtual reality application provided on a computing device to display a first set of display frames on the screen of the computing device; detect the start of a blinking period for a user of the computing device; and during at least a portion of the blinking period, allocate computing resources from the virtual reality application to one or more non-virtual reality applications running on the computing device.

[0097] In an exemplary embodiment, sensor 128 ( Figure 1The device can detect motion, movement, or acceleration of the display screen 130 or the computing device 105. According to another exemplary embodiment, in response to detecting motion, movement, or acceleration of the display screen 130, the computing device 105 can adjust (e.g., increase or decrease) the field of view of one or more displayed images on the screen 130, and / or the computing device 105 can adjust (e.g., increase or decrease) the frame rate of the displayed images. The field of view (FOV) can include the extent of the observable world / environment seen at any given moment. For example, in a VR experience, the field of view can include the extent of the VR world / environment seen on the display at any given moment.

[0098] According to an exemplary embodiment, computing device 105 may include eye-tracking device 142 that tracks the eyes or gaze of a user of computing device 105 and determines which pixel or object the user is looking at (e.g., a set of associated pixels on screen 130).

[0099] When a user is moving, rotating, or changing their gaze direction, adjusting (e.g., increasing or decreasing) the field of view can, at least in some cases, reduce the amount of blur or distortion of the displayed image on screen 130. Furthermore, a high frame rate is unnecessary when the user is moving, because, at least in some cases, during this movement of display screen 130, the image displayed to the user on screen 130 will be blurred or distorted—the viewed or displayed frame / image. Therefore, for example, since many of these display frames displayed on screen 130 during the user / computing device / screen movement will be blurred or distorted anyway, it is not worthwhile for computing device 105 to expend significant computing resources 132 to perform video rendering at a high video rendering rate or a threshold video rendering rate. Therefore, during the period when the user on display screen 130 or the computing device is moving, the frame rate and / or video rendering rate can be adjusted (e.g., increased or decreased), and the field of view can be adjusted (e.g., increased or decreased).

[0100] Figure 9 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment. Operation 910 includes performing video rendering at a first video rendering rate by a virtual reality application provided on the computing device, based on updating the entire image on the screen of the computing device at an update rate. Operation 920 includes detecting motion or movement of the screen. Furthermore, operation 930 includes performing video rendering at a second video rendering rate based on this detection, by updating only a portion of the image at an update rate.

[0101] according to Figure 9 An exemplary implementation of the method, performing video rendering at a second video rendering rate, may include: based on the detection, performing video rendering at a second video rendering rate by updating only the central portion of the image at an update rate.

[0102] Figure 10 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment. Operation 1010 includes performing video rendering at a first video rendering rate by a virtual reality application provided on the computing device, based on updating the entire image on the screen of the computing device at a first update rate. Operation 1020 includes detecting motion or movement of the screen. Furthermore, operation 1030 includes performing video rendering at a second video rendering rate based on the detection, by updating a first portion of the image at the first update rate and by updating a second portion of the image at a second update rate different from (e.g., greater than or less than) the first update rate.

[0103] according to Figure 10 An exemplary implementation of the method, performing video rendering at a second video rendering rate may include: based on the determination, performing video rendering at a second video rendering rate by updating the central portion of the image at a first update rate and by updating the peripheral portion of the image at a second update rate that is different from (e.g., greater than or less than) the first update rate.

[0104] according to Figure 10 An exemplary implementation of the method, performing video rendering at a second video rendering rate, may include: based on the detection, performing video rendering at a second video rendering rate, where the second update rate is less than the first update rate, by updating the central portion of the image at a first update rate and a first image resolution, and by updating the peripheral portion of the image at a second update rate and a second image resolution less than the first image resolution.

[0105] Figure 11 This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment. Operation 1110 includes performing video rendering by a virtual reality application provided on the computing device at a first video rendering rate. Operation 1120 includes detecting motion or movement of the screen. Operation 1130 includes performing video rendering at a second video rendering rate based on the detection, by updating a first portion of an image at a first resolution and by updating a second portion of the image at a second resolution different from (e.g., greater than or less than) the first resolution.

[0106] according to Figure 11 In an exemplary implementation of the method, the first part may include the central portion of the image, while the second part may include the peripheral portion of the image.

[0107] Figure 12This is a flowchart illustrating the operation of a computing device according to another exemplary embodiment. Operation 1210 may include receiving an encoded video signal. Operation 1220 may include performing video rendering based on the encoded video signal by a virtual reality application provided on the computing device to display a first set of display frames of virtual reality content on the screen of the computing device. Operation 1230 may include detecting motion or movement of the screen. Operation 1240 may include adjusting (e.g., increasing or decreasing) the frame rate of one or more display frames used to display the virtual reality content on the screen for at least a period of time in response to detecting motion or movement.

[0108] according to Figure 12 In an exemplary implementation of the method shown, operation 1240 may include reducing the frame rate of one or more display frames used to display virtual reality content on the screen for at least a period of time in response to the detection of motion or movement.

[0109] according to Figure 12 In an exemplary implementation of the method shown, operation 1240 may include: in response to detecting motion or movement, increasing the frame rate of one or more display frames used to display virtual reality content on the screen for at least a period of time.

[0110] according to Figure 12 An exemplary implementation of the method shown may further include: detecting that the motion or movement of the computing device has stopped; and in response to detecting that the motion or movement of the display screen of the computing device has stopped, increasing the frame rate of one or more display frames for displaying virtual reality content on the screen.

[0111] according to Figure 12 An exemplary implementation of the method shown may further include: adjusting (e.g., increasing or decreasing) the field of view of one or more display frames used to display virtual reality content on the screen for at least a period of time after detecting motion or movement of the display screen of the computing device.

[0112] according to Figure 12 An exemplary implementation of the method shown may further include: detecting that the movement or motion of the display screen of the computing device has stopped; and adjusting (e.g., decreasing or increasing) the field of view of one or more display frames used to display virtual reality content on the screen.

[0113] According to an exemplary embodiment, an apparatus may include at least one processor and at least one memory including computer instructions that, when executed by the at least one processor, cause the apparatus to: receive an encoded video signal; perform video rendering based on the encoded video signal by a virtual reality application provided on a computing device to display a first set of display frames of virtual reality content on a screen of the computing device; detect motion or movement of the screen; and, in response to detecting motion or movement, adjust the frame rate of one or more display frames used to display virtual reality content on the screen for at least a period of time.

[0114] As described above, in the illustrative exemplary embodiment, VR goggles 110 ( Figure 1 The VR headset 110 can display stereoscopic images, such as a left-eye image that can be viewed by a user through a left aperture or left eye opening in the VR headset 110, and a right-eye image that can be viewed by a user through a right aperture or right eye opening in the VR headset 110. According to an exemplary embodiment, the left-eye and right-eye images may include one or more objects displayed on the screen 130. These objects may be, for example, a single pixel, or a group of associated pixels displayed on the screen 130. Each object may be any object, such as a person, animal, object, or other object. The viewed image is displayed based on both the right-eye and left-eye images. Various depth cues can be used to convey depth information to the user, such as occlusion (one object blocking or occluding another object), size, viewing angle, etc.

[0115] In an exemplary embodiment, in the illustrative stereoscopic image, depth information of one or more objects may also be provided or transmitted to the user via the parallax (or distance / spacing) between the left-view object (a portion of the left-eye image) and the right-view object (a portion of the right-eye image).

[0116] Convergence refers to the angle formed when a person's (user's) eyes aim at an object being observed / watched. Adaptability refers to focusing the user's eyes on the object being observed / watched. The need for adaptability is inversely proportional to the distance to the object.

[0117] In the real world, there is almost no conflict between adaptive and convergent requirements. However, for VR (virtual reality) or stereoscopic images displayed on a screen (such as screen 130), there can sometimes be a difference or conflict between adaptive and convergent requirements. For a display screen, the adaptive requirement is fixed because the eyes focus on the screen (e.g., the distance from the eyes to the screen is fixed). (The display in a VR HMD system is a virtual display formed by lenses. In this case, the adaptive requirement is the distance from the virtual image formed by the lenses). However, in some cases, the parallax (or distance) between the left and right view images creates a variable convergent requirement, and in some cases, this convergent requirement can differ from the adaptive requirement. This conflict between adaptive and convergent requirements can cause users to feel tense and uncomfortable or experience eye fatigue.

[0118] Figure 13 This diagram illustrates an exemplary conflict between adaptive and convergent requirements according to an exemplary implementation. A stereoscopic (or stereoscopic vision) image 1312 can be displayed on screen 130. Image 1312 may include multiple objects, each of which is part of image 1312 and may include one pixel or a set of related pixels that can display a person, animal, thing, or any other object. In the illustrative exemplary stereoscopic image 1312 displayed on screen 130, objects may include, for example, a dog object 1318, an umbrella object 1316, and a dinosaur object 1314. These are merely some exemplary objects. Figure 13 The image shows the user's eyes, including the left eye 1310L and the right eye 1310R. In this illustrative example, aiming and converging indicate that the user is looking at the dinosaur object 1314.

[0119] To present the stereoscopic image 1312, a left-eye image is displayed to the user's left eye 1310L, and a right-eye image is displayed to the user's right eye 1310R. The left-eye image includes a left dog object 1318L, a left umbrella object (not shown), and a left dinosaur object 1314L. The right-eye image includes a right dog object 1318R, a right umbrella object (not shown), and a right dinosaur object 1314R.

[0120] according to Figure 13 The illustrative example shown includes the left umbrella object and the right umbrella object. Figure 13 (Not shown in the image) are superimposed at the same location, meaning there is no parallax between the left and right umbrella objects. Since there is no parallax between the left and right umbrella objects 1316, umbrella 1316 is located at a distance / depth of screen 130. The user will typically focus on screen 130 (adjust), causing the objects in the image to be in focus. Therefore, this means that in... Figure 13In this example, the convergence requirement and the adaptability requirement of umbrella object 1316 are the same, just like in the real world. Therefore, when a user views umbrella object 1316, because the convergence requirement and the adaptability requirement are the same (as in this example, ...), Figure 13 As shown, because the umbrella object 1316 has no parallax, the user focuses and converges at the depth of the screen 130 to view the umbrella object 1316, so there is no conflict or tension.

[0121] according to Figure 13 In the exemplary embodiment shown, the dog object 1318 exhibits cross-parallax. Due to cross-parallax, this means that the dog object 1318R for the right eye is displayed on the left, while the dog object 1318L for the left eye is located on the right. For such cross-parallax (in this example, the dog object 1318), the object will appear in front of or closer to the viewer than the screen 130 (at a smaller depth than the screen 130).

[0122] In addition, according to Figure 13 In the exemplary embodiment shown, the dinosaur object 1314 exhibits a non-crossing parallax 1320 (the distance between the left and right dinosaur objects 1314L and 1314R). Because the right-eye dinosaur object 1314R is located on the right side, while the left-eye dinosaur object 1314L is located on the left side, this parallax 1320 is non-crossing. This means that the dinosaur object 1314 will appear to the user as if behind the screen 130 (away from the screen 130 or at a greater depth than the screen 130). Therefore, for example, when a user views the dinosaur object 1314 or the dog object 1318, there will be a conflict or tension between adaptive and convergent needs. This is because, for example, a user viewing the dinosaur object 1314 will typically focus on the screen 130, causing the dinosaur object 1314 to be focused, but the convergent needs differ due to the parallax 1320.

[0123] exist Figure 13 In the example shown, eye-tracking device 142 ( Figure 1This can be used within the display device 105 to determine whether a user is viewing the dinosaur object 1314. As described above, when a user views the dinosaur object 1314, a conflict or tension arises between adaptive and convergent needs due to the focus on the screen 130 and the parallax 1320 of the dinosaur object 1314 (or the distance between the left and right dinosaur objects 1314L, 1314R). This conflict or tension is very uncomfortable for the user, and it is desirable to reduce this conflict, especially if the conflict can be reduced while maintaining the relative depth of the viewed object (e.g., the dinosaur object 1314) compared to other objects (e.g., the umbrella object 1316, the dog object 1318) in the image 1312. Regarding relative depth, this means that the dog object 1318 appears to the user as the closest object (having the smallest depth among these objects), followed by the umbrella object 1316, and then the dinosaur object 1314 as the farthest object from the user (having the largest depth among these objects).

[0124] According to an illustrative exemplary embodiment, eye-tracking device 142 ( Figure 1 It can determine the user's gaze angle (the angle or direction from which the user is watching / viewing). Furthermore, for example, VR application 120 can project or cast light into a 3D scene at that user's gaze angle to determine objects intersecting with the light projection, thus identifying the object the user is viewing.

[0125] Therefore, according to exemplary embodiments, techniques for reducing the conflict / tension between the adaptive and convergent requirements for viewing objects are described. According to exemplary embodiments, for example, conflict reduction can be performed while maintaining the relative depth of the viewed object relative to other objects in the image, and while maintaining the relative depth between other objects.

[0126] According to an exemplary implementation, for example, the parallax 1320 between a left-view object (e.g., left dinosaur object 1314L) and a right-view object (e.g., right dinosaur object 1314R) can be measured, calculated, or determined by distance (inches) or pixels. The blink detection module 714 can detect the start of a user's blinking period. The computing device 105 can then shift one or both of the right-eye and left-eye images by parallax (or parallax distance) during the blinking period to reduce the parallax between the left-view object (e.g., 1314L) and the right-view object (e.g., 1314R). For example, only the left-eye image can be shifted, or only the right-eye image can be shifted, or both the left-eye and right-eye images can be shifted to reduce conflict / tension. In some cases, one or both of the left-view and right-view objects can be shifted toward each other, and / or can be shifted to eliminate parallax (when the right-view and left-view objects are superimposed in the same position). Once the left and / or right eye images have been shifted by parallax (parallax amount), tension / conflict between the left and right view images (e.g., 1314L, 1314R) will be reduced (and should be eliminated if the shift is complete). Simultaneously, the parallax of other objects (e.g., the umbrella object and the dog object) will change accordingly, ensuring that their relative display depth is maintained relative to the dinosaur object.

[0127] According to an exemplary embodiment, during the blinking period, the shifting of the left and right eye images can be advantageously performed such that, for example, the user will not detect or will hardly detect the change. Furthermore, in the exemplary embodiment, the entire left-eye image and the entire right-eye image (e.g., including all objects thereon) can be shifted to reduce the parallax 1320 of the viewed object, and also to maintain the relative depth of objects in image 1312. Therefore, both images (left-eye and / or right-eye images) should be shifted overall to maintain the relative depth of other objects relative to the viewed object, although the absolute depth of each object will generally change with this shift. This means that all left and right objects on image 1312 (parts of the left and right eye images, respectively) will be shifted by the same amount as the left and right dinosaur objects 1314L, 1314R will be shifted respectively. Therefore, for example, the left and / or right umbrella objects 1316L, 1316R will also be shifted (by distance parallax 1320 measured relative to dinosaur object 1314) and now have parallax between them. Similarly, the left and / or right dog objects 1318L, 1318R will be shifted by the same distance as the parallax 1320 (measured relative to the dinosaur object 1314), and can have the maximum parallax between the left and right dog objects 1318L, 1318R. This is perceptually acceptable because the short-term goal is to reduce the conflict between the adaptive and convergent needs of viewing the dinosaur object 1314 only while the user is viewing the dinosaur object 1314.

[0128] Figure 14 This is a diagram illustrating an example of reducing or eliminating the conflict between adaptive and convergent needs for the viewed object, according to an exemplary embodiment. Figure 14 In the example shown, for instance during the blinking period, the left-eye image is shifted to the right and / or the right-eye image is shifted to the left, such that (at least in some cases) the parallax 1320 between the left and right dinosaur objects 1314L and 1314R is eliminated. Figure 13 Therefore, after the shift, as Figure 14 As shown, the left and right dinosaur objects 1314L and 1314R are stacked in the same position, eliminating parallax 1320 and providing the appearance of the dinosaurs at a depth of 130 on the screen. Thus, as... Figure 14 As shown, after shifting the left-eye and / or right-eye images based on disparity 1320, the convergence and adaptation requirements of the dinosaur object 1314 are the same (there is no conflict / tension between these requirements). Furthermore, in this illustrative example, there is now a cross-disparity between the left and right umbrella objects 1316L, 1316R (e.g., now making umbrella object 1316 appear closer than screen 130), and the cross-disparity between the left and right dog objects 1318L, 1318R is even greater than that between the left and right dog objects 1318L, 1318R. Figure 13 The parallax shown by these objects in the image. As mentioned above, increasing the parallax of one or more unviewed objects is acceptable, and in fact, desirable, because the goal or objective of this illustrative example is to reduce, and in some cases eliminate, the parallax of the viewed dinosaur object 1314, while maintaining the relative depth between the various objects in the image.

[0129] In an exemplary embodiment, the VR application may include a 3D game engine. The VR application may send a request to the 3D game engine via an application programming interface (API) to reduce / eliminate parallax, and may include a parallax amount 1320. In an exemplary embodiment, the 3D game engine may know or be able to determine the parallax, and the VR application may send the user's gaze direction to the 3D game engine. The 3D game engine can then use the gaze direction to identify the object being viewed and its parallax. The 3D game engine can then determine how much to adjust / shift the left-eye and right-eye images to reduce or eliminate the parallax of the viewed object. The VR application and / or the 3D game engine can then reduce (or even eliminate) the parallax by shifting one or both of the left-eye and right-eye images.

[0130] Furthermore, according to the exemplary implementation, the size of the displayed object is not changed based on the shift of the left and / or right images. Therefore, the perceived absolute distance of the viewed object is not affected by changes in absolute parallax, and the fixed relative depth (relative to other objects in the image) also supports stable perception of the display space.

[0131] As described above, for example during blink periods, the left-eye and / or right-eye images can be shifted to reduce or even eliminate parallax so that the user will not detect the shift. According to another exemplary implementation, the left-eye and / or right-eye images can be shifted by a portion of the parallax 1320 in each of multiple frames to allow the shift or image adjustment to be performed progressively across multiple frames when the user's eyes are open (during non-blink periods), so that the shift or image adjustment will be less noticeable to the user (during non-blink periods). For example, if the parallax is 1 inch, shifting the right-eye image 0.01 inches to the left in each of 100 frames to provide a 1-inch shift would cause the left and right dinosaur objects to be superimposed in the same (or approximately the same) position to reduce or even potentially eliminate parallax 1320. Similarly, for each of the 100 frames, the left-eye and right-eye images can be shifted 0.005 inches toward each other, until the left and right dinosaur objects 1314L and 1314R are close together, or until the parallax is eliminated or at least reduced by 720.

[0132] Furthermore, according to another exemplary embodiment, during non-blinking periods, the left-eye and / or right-eye images are shifted by a portion (e.g., 30%) of the parallax 1320 (e.g., to begin reducing parallax), or when or after the eye-tracking device 142 determines that the user is viewing a new object or that the object has moved, the parallax is reduced over multiple frames. Then, after the start of a blinking period is detected, the shifting of the left-eye and right-eye images can be completed to further reduce parallax 720 under the coverage of the blink.

[0133] According to an exemplary implementation, a VR application or computing device can partially reduce the parallax between the left-eye and right-eye objects before the blink period by shifting one or both of the left-eye and right-eye images at a first (slow) shift rate. The start of the blink period is then detected. During the blink period, the VR application or computing device can further reduce the parallax between the left-eye and right-eye objects by continuing to shift one or both of the left-eye and right-eye images at a second (faster) shift rate, where the second shift rate is faster than the first shift rate. Therefore, the VR application can slowly shift the left-eye and / or right-eye images at a first shift rate before the blink period, making this image shift less perceptible to the user, and then complete the image shift by shifting at a second (faster) shift rate during the blink period. Thus, a faster shift rate can be used during the blink period because the user is not viewing an image / screen during this period. For example, before the blink period, the image can be shifted at a first shift rate of 0.005 inches per frame (e.g., performing a portion of the shift), and then during the blink period, it can be shifted from a second shift rate of 0.05 inches per frame (e.g., completing the shift during the blink period). These are just some example figures for illustration, and other shift rates can be used.

[0134] The process can be repeated each time the user views a new object, or when the object moves, or when the parallax of the viewed object changes, or periodically, for example, every 0.5 seconds. For example, the video can display multiple objects that move over time. For example, periodically, such as every 0.1 seconds, 0.2 seconds, 0.5 seconds, 0.7 seconds, or other periods, the eye-tracking device 142 can identify the object the user is viewing (which may be a new object or the same object, wherein the parallax may change relative to a previous parallax measurement). The computing device 105 can measure the parallax 1320 for such viewed object. The blink detection module 715 can detect the start of a blink period. The computing device 105 can shift one or both of the left and right eye images during the blink period to reduce and / or eliminate the parallax of the viewed object.

[0135] According to an exemplary embodiment, after the start of the blinking period is detected, the computing system can shift only the image of the user's non-dominant eye. The user's monocular dominance can be established by the eye-tracking device 142. For example, the eye-tracking device 142 can record the user's eye movements (both left and right eyes) during a calibration phase of rapid lateral and depth shifting of the target / object. In an illustrative exemplary embodiment, the eye-tracking device 142 can use saccades to determine which eye moves first to the new target location and will be the dominant eye. Therefore, if image shifting is performed during a non-blinking period (when the user's eyes are open), it is advantageous to shift only the image of the non-dominant eye because, for example, the image shift of the non-dominant eye is less noticeable when the dominant eye is fixed on a static image while only the image of the non-dominant eye is shifted. Therefore, for example, during a non-blinking period, parallax can be reduced / eliminated by performing full or partial image shifting by shifting only the image of the non-dominant eye and / or by shifting the image of the non-dominant eye by an amount greater than the offset of the image of the dominant eye.

[0136] Figure 15 This is a flowchart illustrating the operation of a computing device according to an exemplary embodiment. Operation 1510 includes displaying a stereoscopic image, including a right-eye image and a left-eye image, on the screen of the computing device, each illustrating one or more objects. Operation 1520 includes determining a viewing object being viewed by a user of the computing device, the viewing object including a left-view object as part of the left-eye image and a right-view object as part of the right-eye image. Operation 1530 includes calculating or measuring the parallax between the left-view object and the right-view object. Operation 1540 includes detecting the start of a blinking period for the user of the computing device. Furthermore, operation 1550 includes shifting one or both of the left-eye and right-eye images during the blinking period to reduce the parallax between the left-view object and the right-view object.

[0137] according to Figure 15 An exemplary implementation of the method may include shifting one or both of the left-eye and right-eye images during a blink period to eliminate parallax between the left-eye and right-eye objects.

[0138] according to Figure 15 An exemplary implementation of the method may include: before the blink period, beginning to shift one or both of the left-eye image and the right-eye image to reduce the parallax between the left-view object and the right-view object; and continuing to shift one or both of the left-eye image and the right-eye image during the blink period to further reduce the parallax between the left-view object and the right-view object.

[0139] according to Figure 15An exemplary implementation of the method may further include: determining which of the user's eyes is the non-dominant eye, and during non-blinking periods, shifting only the eye image of the user's non-dominant eye to reduce the parallax between left-viewed and right-viewed objects.

[0140] according to Figure 15 An exemplary implementation of the method includes a viewing object comprising a first viewing object, a stereoscopic image comprising a first stereoscopic image, and a blinking period comprising a first blinking period. The method further comprises: displaying a second stereoscopic image comprising a second right-eye image and a second left-eye image on a screen of a computing device; determining a second viewing object viewed by a user of the computing device, the second viewing object comprising a second left-eye object as part of a second left-eye image and a second right-eye object as part of a second right-eye image; measuring the parallax between the second left-eye object and the second right-eye object; detecting the start of the second blinking period of the user of the computing device; and during the second blinking period, shifting one or both of the second left-eye image and the second right-eye image to reduce the parallax between the second left-eye object and the second right-eye object.

[0141] Figure 16 Examples of general-purpose computer devices 1600 and general-purpose mobile computer devices 1650 that can be used with the techniques described herein are shown. Computing device 1600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the embodiments described and / or claimed herein.

[0142] Computing device 1600 includes a processor 1602, a memory 1604, a storage device 1606, a high-speed interface 1608 connected to the memory 1604 and a high-speed expansion port 1610, and a low-speed interface 1612 connected to a low-speed bus 1614 and the storage device 1606. Each of components 1602, 1604, 1606, 1608, 1610, and 1612 is interconnected using various buses and can be mounted on a common motherboard or otherwise mounted as appropriate. Processor 1602 is capable of processing instructions that execute within computing device 500, including instructions stored on memory 1604 or storage device 1606 to display graphical information of a GUI on an external input / output device—such as a display 1616 coupled to the high-speed interface 1608. In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and multiple types of memories as appropriate. Multiple computing devices 1600 can also be connected, each of which provides some of the necessary operations (e.g., as a server cluster, a group of blade servers, or a multiprocessor system).

[0143] Memory 1604 stores information within computing device 1600. In one embodiment, memory 1604 is one or more volatile memory cells. In another embodiment, memory 1604 is one or more non-volatile memory cells. Memory 1604 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0144] Storage device 1606 provides mass storage for computing device 1600. In one embodiment, storage device 1606 may be or include: computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices; flash memory or other similar solid-state storage devices; or an array of devices including devices in a storage area network or other configurations. A computer program product can be tangibly implemented as an information carrier. A computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. An information carrier is a computer or machine-readable medium, such as memory 1604, storage device 1606, or memory on processor 1602.

[0145] High-speed controller 1608 manages bandwidth-intensive operations of computing device 1600, while low-speed controller 1612 manages lower bandwidth-intensive operations. This functional allocation is merely exemplary. In one embodiment, high-speed controller 1608 is coupled to memory 1604, display 1616 (e.g., via a graphics processor or accelerator), and high-speed expansion port 1610, which can accept various expansion cards (not shown). In this embodiment, low-speed controller 1612 is coupled to storage device 1606 and low-speed expansion port 1614. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices—such as keyboards, pointing devices, scanners, or, for example, networked devices such as switches or routers via network adapters.

[0146] As shown in the figure, computing device 1600 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 1620, or multiple times in such a server group. It can also be implemented as part of a rack server system 1624. Furthermore, it can be implemented in a personal computer such as a laptop computer 1622. Alternatively, components from computing device 1600 can be combined with other components (not shown) in mobile devices such as device 1650. Each of such devices can contain one or more of computing devices 1600, 1650, and the entire system can consist of multiple computing devices 1600, 1650 communicating with each other.

[0147] Computing device 1650 includes a processor 1652, memory 1664, input / output devices such as a display 1654, a communication interface 1666, and a transceiver 1668, along with other components. Device 1650 may also be provided with storage devices such as microdrives or other devices to provide additional storage. Each of components 1650, 1652, 1664, 1654, 1666, and 1668 is interconnected using various buses, and some components may be mounted on a common motherboard or otherwise mounted as appropriate.

[0148] Processor 1652 is capable of executing instructions within computing device 1650, including instructions stored in memory 1664. The processor can be implemented as a chipset comprising individual and multiple analog and digital processor chips. For example, the processor can provide coordination of other components of device 1650, such as user interfaces, applications running by device 1650, and control of wireless communications performed by device 1650.

[0149] Processor 1652 can communicate with the user via control interface 1658 and display interface 1656 coupled to display 1654. For example, display 1654 may be a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology. Display interface 1656 may include suitable circuitry for driving display 1654 to present graphics and other information to the user. Control interface 1658 can receive commands from the user and translate them for submission to processor 1652. Additionally, external interface 1662 may be provided to communicate with processor 1652 to enable near-field communication between device 1650 and other devices. For example, external interface 1662 may provide wired communication in some embodiments or wireless communication in others, and multiple interfaces may be used.

[0150] Memory 1664 stores information within computing device 1650. Memory 1664 can be implemented as one or more computer-readable media, one or more volatile memory cells, or one or more non-volatile memory cells. Extended memory 1674 can also be provided and connected to device 1650 via extended interface 1672, which may include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 1674 can provide additional storage space for device 1650, or it can store applications or other information of device 1650. Specifically, extended memory 1674 may include instructions for performing or supplementing the above processes, and may also include security information. Therefore, for example, extended memory 1674 can be provided as a security module of device 1650 and can be programmed with instructions that allow secure use of device 1650. Additionally, security applications along with additional information, such as placing identification information on the SIMM card in a non-intrusive manner, can be provided via a SIMM card.

[0151] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, the computer program product is tangibly implemented in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 1664, extended memory 1674, or memory on processor 1652, which may be received, for example, via transceiver 1668 or external interface 1662.

[0152] Device 1650 can communicate wirelessly via communication interface 1666, which may include digital signal processing circuitry if necessary. Communication interface 1666 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS message sending and receiving, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. Such communication can occur, for example, via radio frequency transceiver 1668. Additionally, short-range communication may occur, such as using Bluetooth, WiFi, or other transceivers (not shown). Furthermore, GPS (Global Positioning System) receiver module 1670 can provide device 1650 with additional navigation-related and location-related wireless data, which can be used by applications running on device 1650 as appropriate.

[0153] Device 1650 can also communicate audibly using audio codec 1660, which can receive spoken information from a user and convert it into usable digital information. Audio codec 1660 can also generate audible sounds for the user, such as through a speaker in, for example, the handset of device 1650. Such sounds can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on device 1650.

[0154] As shown in the figure, the computing device 1650 can be implemented in a variety of different forms. For example, it can be implemented as a cellular phone 1680. It can also be implemented as a smartphone 1682, a personal digital assistant, or part of another similar mobile device.

[0155] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to a storage system, at least one input device, and at least one output device to receive data and instructions from and transmit data and instructions to the device.

[0156] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level programming and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0157] To provide interaction with the user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0158] The systems and technologies described herein can be implemented as: a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through communication networks. The relationship between clients and servers arises from computer programs running on their respective computers that have a client-server relationship with each other.

[0160] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the invention.

[0161] Example 1. A computer-implemented method for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising:

[0162] By using a virtual reality application provided on a computing device, video rendering is performed at a first video rendering rate based on an image on the screen of the computing device that is updated at a first update rate;

[0163] It is determined that the performance of the video rendering is less than a threshold;

[0164] Based on the determination, video rendering is performed at a second video rendering rate by updating a first portion of the image at the first update rate and updating a second portion of the image at a second update rate less than the first update rate.

[0165] Example 2: The method as described in Example 1, wherein performing video rendering at a second video rendering rate includes:

[0166] Based on the determination, video rendering is performed at a second video rendering rate by updating the central portion of the image at the first update rate and updating the peripheral portion of the image at a second update rate less than the first update rate.

[0167] Example 3: The method as described in Example 2, wherein performing video rendering includes:

[0168] Based on the determination, video rendering is performed at a second video rendering rate by updating the central portion of the image at the first update rate and the first image resolution, and updating the peripheral portion of the image at the second update rate and a second image resolution less than the first image resolution, wherein the second update rate is less than the first update rate.

[0169] Example 4: The method as described in Examples 1, 2 or 3, wherein performing video rendering at a second video rendering rate further includes: adjusting the number of pixels in the screen that will be used to display one or more display frames.

[0170] Example 5: The method as described in Examples 1, 2 or 3, wherein performing video rendering at a second video rendering rate further includes: adjusting the display frame or image resolution for displaying one or more display frames on the screen.

[0171] Example 6: The method described in one of Examples 1 to 5 further includes:

[0172] The computing resources of the computing device are allocated from one or more non-virtual reality applications running on the computing device to the virtual reality application.

[0173] Example 7: The method described in any one of Examples 1 through 6 further includes:

[0174] Detect the start of the blinking period of the user of the computing device;

[0175] In response to the detection, during at least a portion of the blinking period, computing resources are allocated from the virtual reality application to one or more non-virtual reality applications running on the computing device.

[0176] Example 8: The method described in any one of Examples 1 through 7 further includes:

[0177] Detecting movement of the screen; and

[0178] For at least a period of time after the detection, the field of view for displaying one or more display frames on the screen is adjusted.

[0179] Example 9. A computer-implemented method comprising:

[0180] Receive encoded video signals;

[0181] Video rendering is performed based on the encoded video signal using a virtual reality application provided on the computing device to display a first set of display frames on the screen of the computing device;

[0182] Detect the start of the blinking period of the user of the computing device;

[0183] During at least a portion of the blinking period, computing resources are allocated from the virtual reality application to one or more non-virtual reality applications running on the computing device.

[0184] Example 10: The method described in Example 9 further includes:

[0185] During at least a portion of the blinking period, the virtual reality application stops video rendering; and

[0186] During at least a portion of the blinking period, one or more non-virtual reality applications running on the computing device perform one or more non-video rendering tasks.

[0187] Example 11: The method described in Example 9 further includes:

[0188] The virtual reality application reduces the video rendering rate during at least a portion of the blink period;

[0189] During at least a portion of the blinking period, at least some computing resources are allocated from the virtual reality application to one or more non-virtual reality applications running on the computing device; and

[0190] During at least a portion of the blinking period, one or more non-virtual reality applications running on the computing device perform one or more non-video rendering tasks.

[0191] Example 12: The method of Example 11 further includes performing one or more of the following at the end or near the end of the blinking period:

[0192] Reassigning at least some computing resources from the one or more non-virtual reality applications running on the computing device to the virtual reality application; and

[0193] The virtual reality application increases the video rendering rate.

[0194] Example 13: A computer-implemented method for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising:

[0195] By using a virtual reality application provided on a computing device, video rendering is performed at a first video rendering rate based on an image on the screen of the computing device that is updated at a first update rate;

[0196] Detect the movement of the screen;

[0197] Based on the detection, video rendering is performed at a second video rendering rate by updating a first portion of the image at the first update rate and updating a second portion of the image at a second update rate different from the first update rate.

[0198] Example 14: The method as described in Example 13, wherein performing video rendering at a second video rendering rate includes:

[0199] Based on the determination, video rendering is performed at a second video rendering rate by updating the central portion of the image at the first update rate and updating the peripheral portion of the image at a second update rate less than the first update rate.

[0200] Example 15: The method as described in Example 13, wherein performing video rendering at the second video rendering rate includes:

[0201] Based on the detection, video rendering is performed at a second video rendering rate by updating the central portion of the image at the first update rate and the first image resolution, and by updating the peripheral portion of the image at a second update rate and a second image resolution less than the first image resolution, wherein the second update rate is less than the first update rate.

[0202] Example 16: The method as described in Example 13, wherein performing video rendering at the second video rendering rate includes:

[0203] Based on the detection, video rendering is performed at a second video rendering rate by updating a first portion of the image at a first resolution and updating a second portion of the image at a second resolution different from the first resolution.

[0204] Example 17: A computer-implemented method for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising:

[0205] A stereoscopic image including a right-eye image and a left-eye image is displayed on the screen of a computing device, wherein the left-eye image and the right-eye image respectively illustrate one or more objects;

[0206] Determine the viewing object viewed by the user of the computing device, the viewing object including a left-viewing object as part of the left-eye image and a right-viewing object as part of the right-eye image;

[0207] Measure the parallax between the left-viewed object and the right-viewed object;

[0208] Detecting the start of a user's blinking episode on the computing device; and

[0209] During the blinking period, one or both of the left-eye image and the right-eye image are shifted to reduce the parallax between the left-viewed object and the right-viewed object.

[0210] Example 18: The method as described in Example 17, wherein the shift includes:

[0211] During the blinking period, one or both of the left-eye image and the right-eye image are shifted to eliminate the parallax between the left-viewed object and the right-viewed object.

[0212] Example 19: The method as described in Example 17, wherein the shift includes:

[0213] Before the blinking period, shifting one or both of the left-eye and right-eye images begins to reduce the parallax between the left-viewed and right-viewed objects; and

[0214] During the blinking period, one or both of the left-eye and right-eye images are continuously shifted to further reduce the parallax between the left-viewed and right-viewed objects.

[0215] Example 20: The method as described in any one of Examples 17 to 19, wherein the shift comprises:

[0216] Before the blinking period, one or both of the left-eye and right-eye images are shifted at a first shift rate to reduce the parallax between the left-viewed object and the right-viewed object; and

[0217] During the blinking period, one or both of the left-eye image and the right-eye image are shifted at a second shift rate to further reduce the parallax between the left-view object and the right-view object, wherein the second shift rate is faster than the first shift rate.

[0218] Example 21: The method as described in any one of Examples 17 to 20 further includes:

[0219] Determine which eye of the user is the non-dominant eye; and

[0220] During non-blinking periods, only the image of the user's non-dominant eye is shifted to reduce the parallax between the left-viewed and right-viewed objects.

[0221] Furthermore, the logical flow shown in the figure does not require the specific order or sequence shown to achieve the desired result. Additionally, other steps may be provided, or steps may be removed from the flow, and other components may be added to or removed from the system or method. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A computer-implemented method, the method comprising: performing video rendering of a portion of a virtual world based on an initial field of view; triggering display of the rendered video on a screen; in response to detecting that the screen is physically moving, changing the performance of the video rendering for at least a time period; allocating a portion of at least a computing resource from performance of the video rendering of the portion of the virtual world to performance of a non-virtual world processing task for at least a portion of the time period; and in response to detecting an end of the time period, reallocating the portion of the allocated computing resource from the non-virtual world processing task to the video rendering of the portion of the virtual world.

2. The computer-implemented method of claim 1, wherein: changing the performance of the video rendering comprises reducing the portion of the virtual world based on an updated field of view, the updated field of view being a smaller field of view that is smaller than the initial field of view; and the allocation of the portion of the computing resource is performed such that the video rendering of the portion of the virtual world is paused or temporarily stopped. the smaller field of view reduces a range of the virtual world being rendered. performing video rendering of the portion of the virtual world based on an initial field of view comprises performing video rendering at a first video rendering rate, and changing the performance of the video rendering comprises performing video rendering at a second video rendering rate.

3. The computer-implemented method of claim 2, wherein, performing video rendering at a second video rendering rate comprises updating only a center portion of the screen at the second video rendering rate.

4. The computer-implemented method of claim 1, wherein, performing video rendering at a second video rendering rate comprises:

5. The computer-implemented method of claim 4, wherein, updating a center portion of the screen at a first update rate; and 6. The computer-implemented method of claim 4, wherein, updating a peripheral portion of the screen at a second update rate that is less than the first update rate. changing the performance of the video rendering comprises adjusting a resolution of the video rendering. changing the performance of the video rendering comprises:

7. The computer-implemented method of claim 1, wherein, updating a first portion of the screen at a first resolution; and 8. The computer-implemented method of claim 1, wherein, updating a second portion of the screen at a second resolution that is different than the first resolution. detecting that the screen is physically moving comprises detecting by a sensor that detects motion, movement, or acceleration of the screen. the changing in response to detecting that the screen is physically moving further comprises adjusting a frame rate associated with rendering video for display on a display screen.

9. The computer-implemented method of claim 1, wherein, the allocation enables one or more non-virtual reality applications to process data for a portion of the time period.

10. The computer-implemented method of claim 1, wherein, 12. A computing device, comprising:

11. The computer-implemented method of claim 1, wherein, a screen; a sensor, the sensor comprising an accelerometer to detect motion or movement of the screen; at least one processor; and at least one memory comprising computer instructions that, when executed by the at least one processor, cause the computing device to: perform, by a virtual reality application provided on the computing device, video rendering to display a first set of display frames of virtual reality content on a screen of the computing device; in response to detecting, by the sensor, motion or movement of the screen, adjust a frame rate of a second set of display frames of the virtual reality content for display on the screen for at least a time period; ​ ​ ​ allocating a portion of computing resources from execution of the video rendering of the virtual reality content to performance of a non-virtual reality application task for at least a portion of the time period; and re-allocating the portion of allocated computing resources from the non-virtual reality application task to the video rendering of the virtual reality content in response to detecting an end of the time period.

13. The computing device of claim 12, wherein, The instructions that cause the computing device to adjust the frame rate for displaying the virtual reality content on the screen include instructions that cause the computing device to reduce a size of a portion of a virtual world for video rendering the second set of display frames of the virtual reality content, and wherein the allocation of the portion of computing resources is performed such that the video rendering of the virtual reality content is paused or temporarily stopped.

14. The computing device of claim 13, wherein, The reduced size of the portion of the virtual world corresponds to a reduced extent of a field of view of the virtual world.

15. The computing device of claim 12, wherein, The instructions that cause the computing device to adjust the frame rate for displaying the virtual reality content on the screen include instructions that cause the computing device to increase the frame rate for displaying the virtual reality content on the screen.

16. A computer-implemented method for executing instructions stored on a non-transitory computer-readable storage medium, the method comprising: executing, by a virtual reality application provided on a computing device, video rendering at a first video rendering rate based on updating an entire image on a screen of the computing device at a first update rate; in response to detecting a physical movement of the screen, executing video rendering at a second video rendering rate by updating a first portion of the image at the first update rate and by updating a second portion of the image at a second update rate different from the first update rate for at least a time period; allocating a portion of computing resources from execution of the video rendering to performance of a non-virtual reality application task for at least a portion of the time period; and re-allocating the portion of allocated computing resources from the non-virtual reality application task to the video rendering in response to detecting an end of the time period.

17. The method of claim 16, wherein, executing video rendering at a second video rendering rate includes: based on the detection, executing video rendering at a second video rendering rate by updating a central portion of the image at the first update rate and by updating a peripheral portion of the image at the second update rate, the second update rate being less than the first update rate.

18. The method of claim 16, wherein, executing video rendering at a second video rendering rate includes: based on the detection, executing video rendering at a second video rendering rate by updating a central portion of the image at the first update rate and first image resolution and by updating a peripheral portion of the image at the second update rate and a second image resolution less than the first image resolution, the second update rate being less than the first update rate.

19. The method of claim 16, wherein, executing video rendering at a second video rendering rate includes: based on the detection, executing video rendering at a second video rendering rate by updating a central portion of the image at the first update rate and first image resolution and by updating a peripheral portion of the image at the second update rate and a second image resolution less than the first image resolution, the second update rate being less than the first update rate. Based on the detecting, performing video rendering at a second video rendering rate by updating a first portion of the image at a first resolution and by updating a second portion of the image at a second resolution different from the first resolution.

20. The method of claim 16, wherein: performing video rendering at the first video rendering rate includes performing video rendering of a portion of a virtual world based on an initial field of view; performing video rendering at the second video rendering rate includes performing video rendering of a reduced portion of the virtual world based on a reduced field of view; and the second video rendering rate is greater than the first video rendering rate.

Images