Information processing device, information processing method, and program

Abstract

This improves the clarity of images when screen captures are taken from videos. [Solution] The information processing device includes setting means to set different rendering resolutions for areas corresponding to the user's line of sight on the screen and other areas; rendering means to render an image to be displayed on the screen according to the set rendering resolution; and state determination means to determine whether capture of the image displayed on the screen is in progress. If the setting means determines that capture is in progress, it stops setting the rendering resolution for areas corresponding to the user's line of sight and other areas, and sets the rendering resolution for a specific area of ​​the screen to a predetermined resolution.

Description

[Technical Field]

[0001] This invention relates to information processing technology for capturing images. [Background technology]

[0002] Head-mounted displays (HMDs) are used as devices for experiencing virtual reality content. Hereafter, virtual reality will be referred to as "VR" and head-mounted displays as "HMDs." HMDs can provide users with an experience that makes them feel as if they are moving around in a virtual space by displaying images generated according to the user's position and posture.

[0003] Generally, images displayed on an HMD are generated by rendering computer graphics (CG) as seen from the viewpoint of a virtual camera. To reduce the processing load of rendering, a process called foveated rendering is performed. Hereafter, foveated rendering will be referred to as "FR". FR is a rendering technique that takes into account the visual resolution characteristics of the human eye, rendering the central area of ​​the field of view with high resolution and the peripheral areas with low resolution. This reduces the processing load during rendering so that the user using the HMD does not notice the decrease in image quality. There are two types of FR: fixed foveated rendering and eye-tracked foveated rendering. Hereafter, fixed foveated rendering will be referred to as "FFR", and eye-tracked foveated rendering will be referred to as "ETFR". FFR is an FR that fixes the gaze position to the center of the screen. ETFR is an FR that tracks the user's moving gaze position and renders the area of ​​the moving gaze position with high resolution and the rest with low resolution. ETFR can reduce the processing load more effectively compared to FFR.

[0004] In addition, a screen capture function is provided to record the image that the user is viewing through the MHD as a still image or a video file. This function is mainly used for the purpose of the user checking the image himself / herself later or sharing the image with a third party.

[0005] On the other hand, Patent Document 1 discloses a technique for performing high-quality image output when performing screen capture by performing off-screen rendering separately from real-time rendering.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] When screen capture is performed while rendering processing by ETFR is being performed, if the screen capture is performed, the saved image will be an image with only a partial area in the image having high resolution and the rest having low resolution. In particular, in the case of a video, if the line-of-sight position of the user wearing the HMD is moving, the high-resolution image area on the captured video will also appear to move accordingly. In particular, when this captured video is viewed by a third party, the high-resolution image area on the screen moves irregularly, and at the same time, the low-resolution blurred area also moves irregularly on the screen, resulting in a very difficult-to-watch video. Note that screen capture by off-screen rendering as described in Patent Document 1 is not a real-time process, so it is not suitable for cases where it is necessary to render several images per second like a video. <00??097> Therefore, an object of the present invention is to make it possible to improve the uncomfortableness of viewing an image when performing screen capture of a video.

Means for Solving the Problems

[0009] The information processing apparatus of the present invention comprises setting means for setting the rendering resolution to differ between an area corresponding to the user's line of sight on the screen and other areas; rendering means for rendering an image to be displayed on the screen according to the rendering resolution set by the setting means; and state determination means for determining whether capture of the image displayed on the screen is in progress. The setting means is characterized in that, when the state determination means determines that capture is in progress, it stops setting the rendering resolution for the area corresponding to the user's line of sight and other areas, and sets the rendering resolution for a specific area of ​​the screen to a predetermined resolution. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the difficulty of viewing images when screen capturing videos. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram of the HMD system as an image display system. [Figure 2] This is an internal diagram of the HMD. [Figure 3] This is a diagram showing the system configuration of an information processing device. [Figure 4] This figure shows the functional configuration of the information processing device according to the first embodiment. [Figure 5] This is a flowchart of the information processing according to the first embodiment. [Figure 6] This figure shows an example of a resolution map. [Figure 7] This figure shows the functional configuration of the information processing device according to the second embodiment. [Figure 8] This is a flowchart of the information processing according to the second embodiment. [Figure 9] This figure shows the functional configuration of the information processing device according to the third embodiment. [Figure 10]This is a flowchart of the information processing according to the third embodiment. [Figure 11] This figure shows an example of how information indicating that a screen capture is in progress is displayed. [Figure 12] This figure shows the functional configuration of the information processing device according to the fourth embodiment. [Figure 13] This is a flowchart of the information processing according to the fourth embodiment. [Figure 14] This figure shows the functional configuration of the information processing device according to the fifth embodiment. [Figure 15] This is a flowchart of the information processing according to the fifth embodiment. [Figure 16] This is a flowchart of the state transition process. [Figure 17] This figure shows an example of a display image with confirmation information superimposed. [Figure 18] This figure shows the functional configuration of the information processing device according to the sixth embodiment. [Figure 19] This is a flowchart of the information processing according to the sixth embodiment. [Figure 20] This figure shows an example of a display image where the instruction UI and confirmation information are superimposed. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications and various conditions (usage conditions, usage environment, etc.) of the device to which the present invention is applied. Furthermore, some of the embodiments and modified examples described later may be combined as appropriate. In each of the following embodiments, redundant explanations of the same hardware configuration, functional configuration, and processing steps will be omitted.

[0013] <First Embodiment> Figure 1 is a diagram showing a schematic configuration example of the image display system according to this embodiment. In this embodiment, an example in which a user experiences VR (virtual reality) content using an HMD (head-mounted display device) 101 will be described. As shown in Figure 1, the image display system includes an HMD 101 and an information processing device 102. The HMD 101 and the information processing device 102 are connected via a predetermined communication path, enabling them to mutually send and receive various data such as image data and various control signals. In the example shown in Figure 1, the HMD 101 and the information processing device 102 are connected via a cable conforming to standards such as HDMI (High-Definition Multimedia Interface®) or USB (Universal Serial Bus). Of course, the type of communication path connecting the HMD 101 and the information processing device 102 is not particularly limited. As a specific example, a communication path between the HMD 101 and the information processing device 102 may be established by wireless communication such as Bluetooth®. Furthermore, the system configuration shown in Figure 1 is merely an example, and the configuration of the image display system according to this embodiment is not limited to the example in Figure 1. For example, an input device such as a controller or keyboard (not shown) for receiving user input may be connected to the information processing device 102 via a predetermined communication path.

[0014] Figure 2 shows an example of the internal configuration of the HMD101. As shown in Figure 2, the HMD101 is equipped with multiple imaging devices 201, distance sensors 202, displays 203L and 203R, eyepieces 204L and 204R, etc. The multiple imaging devices 201 are imaging devices for imaging the real space, and RGB cameras and the like are used. The distance sensors 202 are sensors for acquiring depth information indicating the distance to objects located in the external environment, and LiDAR (Light Detection And Ranging) or imaging devices are used. The displays 203L and 203R are configurations for displaying images, and display panels such as liquid crystal panels and organic EL panels are used. Display 203L is a left-eye display corresponding to the user's left eye when wearing the HMD101, and display 203R is a right-eye display corresponding to the user's right eye. Eyepiece 204L is a left-eye eyepiece corresponding to the user's left eye when wearing the HMD101, and eyepiece 204R is a right-eye eyepiece corresponding to the user's right eye. Although not shown in the diagram, the HMD101 also includes an IMU (Inertial Measurement Unit) such as a gyro sensor and accelerometer, as well as an imaging device, to enable its own position tracking.

[0015] The information processing device 102 generates a left-eye display image to be displayed on the left-eye display 203L of the HMD 101, and generates a right-eye display image to be displayed on the right-eye display 203R, and transmits them to the HMD 101. The HMD 101 is worn on the head of a user (not shown), and displays the left-eye display image on the left-eye display 203L and the right-eye display image on the right-eye display 203R. As a result, the user can observe the magnified virtual image of the left-eye display image displayed on the display 203L via the eyepiece 204L, and observe the magnified virtual image of the right-eye display image displayed on the display 203R via the eyepiece 204R.

[0016] Furthermore, the information processing device 102 may apply a parallax between the left-eye display image and the right-eye display image, corresponding to the distance between the left and right eyes of the user wearing the HMD 101 (for example, the distance between the eyepieces 204 corresponding to each of the left and right eyes). By applying such control, it becomes possible to provide the user wearing the HMD 101 with an image perception that has a sense of depth. In this embodiment, the system configuration described focuses on a system in which the information processing device 102 is implemented as a device independent of the HMD 101. However, this does not necessarily limit the configuration of the image display system according to this embodiment. For example, the image display system according to this embodiment may be implemented by an integrated HMD system in which a configuration equivalent to the information processing device 102 is included inside the HMD 101.

[0017] Figure 3 shows an example of the hardware configuration of the image display system according to this embodiment. In Figure 3, we will focus on explaining the hardware configuration of the information processing device 102 in particular. The information processing device 102 includes a CPU (Central Processing Unit) 301, RAM (Random Access Memory) 302, and ROM (Read Only Memory) 303. The information processing device 102 also includes a storage device 304, a general-purpose interface 305, an image output interface 306, and a network interface 307, etc. These components of the information processing device 102 are connected to each other via the main bus 300, enabling them to send and receive information.

[0018] The CPU 301 is a processor that comprehensively controls each part of the information processing device 102. The RAM 302 functions as the main memory, work area, etc., of the CPU 301. The ROM 303 stores a group of control programs executed by the CPU 301. The storage device 304 stores the OS (Operating System), various application programs including the information processing program according to this embodiment executed by the CPU 301, data used for image processing, etc. Examples of storage devices 304 include HDD (Hard Disk Drive) and SSD (Solid State Drive). The information processing program according to this embodiment may also be stored in the ROM 303.

[0019] The general-purpose I / F305 is a serial bus interface compliant with standards such as USB and IEEE 1394 (Institute of Electrical and Electronics Engineers 1394). The general-purpose I / F305 is connected, for example, to the IMU and distance sensor provided in the HMD101. This allows the information processing device 102 to acquire posture information and depth images to target objects (images in which depth information corresponding to the measurement result of the distance to the target object is mapped to each pixel) from the HMD101. The general-purpose I / F305 is also used to acquire images corresponding to the imaging results from the imaging device 201 of the HMD101. The image output interface 306 is an interface such as HDMI or DisplayPort, and is used to transmit display images to the HMD101 for display on the HMD101's display 203. The network interface 307 is an interface for connecting the information processing device 102 to a predetermined network. The configuration of the network interface 307 may be appropriately changed depending on the type of network to be connected and the communication method applied.

[0020] Figure 4 is a diagram showing the functional configuration of the image display system according to this embodiment, with particular attention paid to the configuration of the information processing device 102. The information processing device 102 includes a state determination unit 401, a rendering setting unit 402, a rendering unit 403, a display unit 404, and an output unit 405. The state determination unit 401 determines whether the screen capture function is being executed.

[0021] The rendering settings unit 402 performs various settings when performing rendering processing. For example, the rendering settings unit 402 sets parameters used for rendering, such as the rendering resolution, field of view, and the position and orientation of the virtual camera corresponding to the position and orientation of the HMD. In addition, if FR (foveal rendering) processing is performed as the rendering process, the rendering settings unit 402 sets the information necessary for that FR processing. As mentioned above, there are FFR (fixed foveal rendering) and ETFR (foveal rendering with eye tracking), and the rendering settings unit 402 can set which of FFR and ETFR to use. Furthermore, the rendering settings unit 402 sets the resolution map to be used for FR processing as map area information indicating the rendering resolution, as shown in Figure 6, which will be described later.

[0022] The rendering unit 403 performs rendering processing based on the rendering settings set by the rendering setting unit 402 and generates a rendered image. In this embodiment, the rendering unit 403 generates rendered images corresponding to the displays 203L and 203R of the HMD 101. The display unit 404 transmits the rendered image to the HMD 101 via the image output I / F 306, causing it to be displayed on the HMD 101's displays 203L and 203R. The output unit 405 records the rendered image as a file in the storage device 304.

[0023] Figure 5 is a flowchart showing the flow of information processing performed by the information processing device 102 in the image display system according to this embodiment. The series of processes shown in the flowchart in Figure 5 are realized by the information processing device 102 loading the information processing program according to this embodiment, stored in the ROM 303 or storage device 304, into the RAM 302 and executing it by the CPU 301. In this way, the CPU 301 fulfills the role of each component shown in Figure 4. The processing in the flowchart in Figure 5 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, the CPU 301 can display the video on the HMD 101 for the user to watch, and can also save the screen-captured video as a file in the storage device 304. In the flowcharts thereafter, the symbol S represents a processing step (processing step).

[0024] First, in S501, the state determination unit 401 determines whether the information processing device 102 is in a state of executing the screen capture function for video or still images. If the state determination unit 401 determines that the screen capture function is in operation (hereinafter referred to as "capturing"), the processing of the information processing device 102 proceeds to S502. On the other hand, if the state determination unit 401 determines that the screen capture function is not in operation (not capturing), the processing of the information processing device 102 proceeds to S503.

[0025] If the process proceeds to S502, the rendering setting unit 402 sets the rendering mode to such that the rendering resolution for a specific area of ​​the HMD101's display screen is set to a predetermined resolution. In this embodiment, the rendering setting unit 402 fixes the specific area to the central area of ​​the screen and sets the rendering mode to such that the predetermined resolution for rendering in that central area is higher than that of other areas. More specifically, if capture is in progress, the rendering setting unit 402 sets the rendering mode (FR mode) to FFR (Fixed Foveal Rendering) mode. In FFR mode, as described above, the central area of ​​the screen is set to a high-resolution area. The rendering setting unit 402 then sets the resolution map necessary to perform rendering in such FFR mode.

[0026] Figure 6 shows an example of a resolution map required for FR. In the case of FFR, the rendering settings unit 402 sets a resolution map as shown in Figure 6(b). In Figure 6(b), the map region 605 indicates the area where rendering is performed at high resolution, and in the case of FFR, it is located in the center of the image. In the rendering unit 403, as shown in each sub-region 609 obtained by enlarging a part of the map region 605 and dividing it into 4x4 sections, rendering calculations are performed for all pixels, as indicated by the black circles in the figure.

[0027] In Figure 6(b), map region 606 represents an area where rendering is performed at a lower resolution than map region 605. For example, the resolution in map region 606 is halved in both the vertical and horizontal directions compared to map region 605. In this case, the rendering unit 403 performs one rendering calculation for every 4 pixels (2x2 pixels vertically and horizontally), as shown in each sub-region 610, which is created by enlarging a part of map region 606 and dividing it into 4x4 sections. In other words, the number of rendering calculations is reduced to 1 / 4 in this case. Note that the same result is stored in each of the 4 pixels (2x2 pixels vertically and horizontally).

[0028] In Figure 6(b), map region 607 represents an area where rendering is performed at an even lower resolution than map region 606. For example, the resolution in map region 607 is reduced to 1 / 4 in both the vertical and horizontal directions compared to map region 605. In this case, the rendering unit 403 performs one rendering calculation for every 16 pixels (4x4 pixels vertically and horizontally), as shown in each sub-region 611 obtained by enlarging a part of map region 607 and dividing it into 4x4 sections. In other words, the number of rendering calculations is reduced to 1 / 16 in this case. Note that the same result is stored in each of the 16 pixels (4x4 pixels vertically and horizontally).

[0029] When rendering is performed by the rendering unit 403 based on the resolution map shown in Figure 6(b), the processing load can be reduced compared to rendering the entire image at high resolution without performing FR. In addition, in the case of FFR, as shown in Figure 6(b), the map area 605 where high-resolution rendering is performed is always fixed to the central area regardless of the gaze position of the user using the HMD 101. Furthermore, in FFR, it is generally necessary to set a wider high-resolution area compared to ETFR. This is because if the area to which the gaze moves due to human eye movement is not rendered at high resolution, the user using the HMD will notice the decrease in resolution.

[0030] Thus, when capturing is in progress, the rendering unit 403 performs rendering processing in which a specific area to be rendered at high resolution is fixed to the central area of ​​the screen. In other words, when capturing is in progress, the information processing device 102 of this embodiment stops rendering processing that changes the position of the high-resolution area on the screen in accordance with the user's gaze tracking, such as ETFR, and switches to rendering processing using FFR.

[0031] On the other hand, if the process proceeds to S503, the rendering setting unit 402 sets the rendering mode to such that the rendering resolution for the area corresponding to the user's gaze differs from the rendering resolution for other areas. In this embodiment, the rendering setting unit 402 sets the rendering mode to ETFR (Eye-Tracking Foveal Rendering) as the rendering mode to such that the rendering resolution for the area corresponding to the user's gaze differs from that for other areas. That is, the rendering setting unit 402 sets the rendering mode to such that rendering is performed at a high resolution at the position corresponding to the user's eye-tracking and at a low resolution in other areas. The rendering setting unit 402 then sets the resolution map necessary for performing the rendering process in such an ETFR mode.

[0032] Figure 6(a) shows an example of a resolution map set by the rendering settings unit 402 in the case of ETFR. In Figure 6(a), map region 602 represents the area where rendering is performed at high resolution, map region 603 represents the area where rendering is performed at a lower resolution than map region 602, and map region 604 represents the area where rendering is performed at an even lower resolution than map region 603. The number of rendering calculations in each of these map regions 602, 603, and 604 varies depending on the resolution setting of each map region, which is the same as in the case of FFR explained in S502.

[0033] In ETFR, unlike FFR, the center of the high-resolution area is not the center of the image, but the user's gaze position 601 using the HMD. In the example in Figure 6(a), the resolution map is shown when the user's gaze position 601 is located in the upper right of the image. In ETFR, the rendering settings unit 402 tracks the user's gaze position in real time and sets a resolution map according to that gaze position, so in reality, a different resolution map is used for each frame. Furthermore, with ETFR, when the user's gaze moves due to eye movements, the resolution map can be changed according to that movement of the gaze, so the area to be rendered at a higher resolution can be further narrowed compared to FFR.

[0034] Note that while Figures 6(a) and 6(b) show examples of circular resolution maps with three different resolution levels, the resolution map settings are not limited to these examples. The rendering settings unit 402 can arbitrarily set the size, number, and shape of each map area. For example, Figure 6(c) shows an example of a resolution map (map area 608) that renders the entire image at high resolution. In other words, the rendering settings unit 402 can also set a resolution map like the one shown in Figure 6(c) as the resolution map for FFR as explained in S502. In this example, the processing is equivalent to not performing FR processing.

[0035] After the aforementioned S502 or S503, the information processing device 102 proceeds to S504. When the process proceeds to S504, the rendering unit 403 executes rendering processes with different resolutions for each map area, according to the rendering processing mode set in the rendering setting unit 402, and generates a rendered image for the corresponding frame. For performing rendering processes with different resolutions for each area, known techniques such as variable rate shading can be used.

[0036] Next, in S505, the display unit 404 transmits the rendered image generated by the rendering unit 403 to the HMD 101 via the image output I / F 306, and displays it on the display 203. Next, in S506, the state determination unit 401 determines again whether video or still image capture is in progress. If the state determination unit 401 determines that capture is in progress, the information processing device 102 proceeds to S507. On the other hand, if the state determination unit 401 determines that capture is not in progress, the information processing device 102 terminates the process shown in the flowchart of Figure 5.

[0037] Proceeding to S507, the output unit 405 records the rendered image as a file in the storage device 304. The output unit 405 may write the file synchronously with each frame, but if writing the file takes time, it may also sequentially write the rendered image asynchronously while buffering it in a buffer provided in the RAM 302. In this way, the output unit 405 can select an appropriate method according to the data size of the rendered image and the processing capacity of the information processing device.

[0038] As described above, in the information processing device 102 of the first embodiment, rendering load is reduced by using ETFR when screen capture is not performed, but when screen capture is performed, it is set to FFR. In other words, when screen capture is performed, the high-resolution area can always be fixed to the center of the image. This can improve the difficulty of viewing, especially when screen capturing videos. According to this embodiment, it is possible to generate a screen capture video that eliminates the position-independent, irregular movement of high-resolution areas and low-resolution blurred areas that occur in videos captured with ETFR.

[0039] <Second Embodiment> In the first embodiment, the process of setting to FFR when performing screen capture was described, but as mentioned above, FFR generally requires setting a wider high-resolution area compared to ETFR. For this reason, the rendering processing load tends to be higher with FFR than with ETFR, and if the CG to be rendered is complex and computationally intensive, the frame rate will drop significantly. Therefore, in the second embodiment, in addition to changing the setting to FFR during screen capture, a process to optimize the rendering processing load will be described by further narrowing the rendering field of view to counteract the increase in processing load. Note that the configuration of the image display system, the configuration of the HMD101, and the hardware configuration in the second embodiment are the same as in the examples in Figures 1 to 3 described above, so their illustrations and descriptions will be omitted.

[0040] Figure 7 shows the functional configuration of the image display system according to the second embodiment, focusing particularly on the configuration of the information processing device 102. The information processing device 102 according to the second embodiment has a load acquisition unit 701 in addition to the configuration described in the first embodiment described above. The other configurations, excluding the load acquisition unit 701, are the same as those in Figure 4, so their explanation is omitted.

[0041] If the state determination unit 401 determines that capture is in progress, the load acquisition unit 701 acquires the increase in rendering processing load when changing the FR setting from ETFR to FFR. The increase in processing load may be acquired by reading a value that has been calculated and stored in advance, or it may be calculated and acquired in real time.

[0042] Figure 8 is a flowchart showing the information processing flow, particularly that performed by the information processing device 102, in the image display system according to the second embodiment. The series of processes shown in the flowchart of Figure 8 are realized when the information processing program according to the second embodiment, stored in the ROM 303 or storage device 304, is loaded into the RAM 302 and executed by the CPU 301. In this way, the CPU 301 fulfills the role of each component shown in Figure 7. The processing shown in the flowchart of Figure 8 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, images can be displayed as a video on the HMD 101 for the user to view, and the screen-captured video can be saved as a file in the storage device 304. In the flowchart of Figure 8, the same processing steps as in the flowchart of Figure 5 described above are denoted by the same reference numerals as in Figure 5, and their explanations are omitted.

[0043] In the flowchart shown in Figure 8, at S501, the state determination unit 401 determines that capture is in progress, and then at S502, the rendering setting unit 402 sets it to FFR, after which the information processing device 102 proceeds to S801. Proceeding to S801, the load acquisition unit 701 calculates and acquires the increase in rendering processing load when the FR setting is changed from ETFR to FFR. For example, the number of rendering calculations in the case of ETFR is N ETFR Let N be the number of rendering calculations in the case of FFR. FFR Accordingly, the load acquisition unit 701 calculates the ratio of these ratios as the increase in processing load R using the following equation (1).

[0044] R=N FFR / N ETER Formula (1)

[0045] Note that the calculation of the increase in processing load R does not necessarily need to be performed every frame. If the processing load does not change between frames, the increase in processing load R calculated in the previous frame may be stored and used. Alternatively, the increase in processing load may be calculated and stored in advance from, for example, the design data of the resolution map, before rendering, and then retrieved. After S801, the processing of the information processing device 102 proceeds to S802.

[0046] In step S802, the rendering setting unit 402 sets the rendering field of view so as to offset the increase in rendering processing load calculated in step S801. For example, if the default rendering horizontal field of view is θ and the vertical field of view is φ, the rendering setting unit 402 calculates the horizontal field of view θ' and the vertical field of view φ' of the rendering field of view using the following equations (2) and (3).

[0047] θ' = θ / √(R) Equation (2) φ' = φ / √(R) Equation (3)

[0048] Here, we have explained an example in which the horizontal and vertical fields of view are changed by the same proportion to maintain the aspect ratio, but this is not the only way to set the fields of view, and the proportion of change between the horizontal and vertical fields of view can be set arbitrarily. Similar to the calculation of the increase in processing load R, if the increase in processing load R does not change between frames, the field of view calculated in the previous frame may be stored and used. After S802, the processing of the information processing device 102 proceeds to S504 and beyond as described above.

[0049] On the other hand, if it is determined in S501 that capture is not in progress, and further, after the rendering settings unit 402 performs settings such as ETFR in S503, the processing of the information processing device 102 proceeds to S803. When the process proceeds to S803, the rendering setting unit 402 sets the rendering field of view to the default horizontal field of view θ and vertical field of view φ. The default field of view is assumed to be the field of view matched to the HMD 101. After S803, the processing of the information processing device 102 proceeds to S504 and beyond as described above.

[0050] Furthermore, if a rendering field of view is set in S802 to offset the increase in rendering processing load, in S505 the display unit 404 displays the rendering image generated by the rendering unit 403 on the display 203 according to that rendering field of view.

[0051] In the second embodiment of the information processing device 102, similar to the first embodiment, rendering is performed using ETFR under normal circumstances to reduce the rendering load, and only when screen capture is performed is it set to FFR to keep the high-resolution area always centered on the image. Furthermore, in the second embodiment of the information processing device 102, the rendering load is reduced by narrowing the rendering field of view to offset the increase in processing load when FFR is set during capture. In other words, according to this embodiment, it is possible to improve the difficulty in viewing images caused by the irregular, position-independent movement of high-resolution and low-resolution blurred areas, especially when screen capture is performed on video, while also optimizing the rendering load.

[0052] <Third Embodiment> In the second embodiment, an example was described in which the rendering processing load is optimized by changing the FFR setting when performing a screen capture, and further narrowing the rendering field of view to offset the increased processing load. However, if the rendering field of view is narrowed significantly, there is a concern that it may give the user using the HMD101 a visually and psychologically cramped feeling. Therefore, in the third embodiment, a process is described to mitigate the visual and psychological impact on the user by narrowing the rendering field of view and displaying an indication that screen capture is being performed in the area where the rendered image is no longer displayed. Note that the configuration of the image display system, the HMD101, and the hardware configuration in the third embodiment are the same as those in the examples in Figures 1 to 3 described above, so their illustrations and descriptions are omitted.

[0053] Figure 9 shows the functional configuration of the image display system according to the third embodiment, focusing particularly on the configuration of the information processing device 102. The information processing device 102 according to the third embodiment has a state presentation unit 901 in addition to the configuration described in the second embodiment described above. The other configurations, excluding the state presentation unit 901, are the same as those in Figure 7, so their explanation is omitted.

[0054] If the status determination unit 401 determines that capturing is in progress, the status indication unit 901 generates a display image for display on the display unit 404 by superimposing information indicating that capturing is in progress onto the rendered image. The display unit 404 displays the display image with the information indicating that capturing is in progress superimposed on the HMD 101's display. As a result, the user can recognize that the rendering field of view has been narrowed because capturing is in progress, and is less likely to be visually or psychologically affected.

[0055] Figure 10 is a flowchart showing the information processing flow, particularly that performed by the information processing device 102, in the image display system according to the third embodiment. The series of processes shown in the flowchart in Figure 10 are realized when the information processing program according to the third embodiment, stored in the ROM 303 or storage device 304, is loaded into the RAM 302 and executed by the CPU 301. In this way, the CPU 301 fulfills the role of each component shown in Figure 9. The processing in the flowchart in Figure 10 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, images can be displayed as a video on the HMD 101 for the user to view, and the screen-captured video can be saved as a file in the storage device 304. In the flowchart shown in Figure 10, the same processing steps as in the flowcharts of Figures 5 and 8 described above are denoted by the same reference numerals as in Figures 5 and 8, and their explanations are omitted.

[0056] In the flowchart shown in Figure 10, after the rendering process by the rendering unit 403 in S504, the processing of the information processing device 102 proceeds to S506. In S506, the state determination unit 401 re-determines whether or not capture is in progress. If it is determined that capture is in progress, the information processing device 102 proceeds to S507. On the other hand, if the state determination unit 401 determines that capture is not in progress, the information processing device 102 proceeds to S1002.

[0057] At S507, the output unit 405 records the rendered image as a file in the storage device 304. As previously mentioned, the output unit 405 may write the file synchronously with each frame, or, if writing the file takes time, it may sequentially write the rendered image asynchronously while buffering it in a buffer provided in the RAM 302. After S507, the processing of the information processing device 102 proceeds to S1001.

[0058] When the process proceeds to S1001, the state presentation unit 901 superimposes information indicating that capture is in progress onto the rendered image to generate a display image for display on the display unit 404. Figure 11 shows an example of a display image generated by the status display unit 901. As shown in Figure 11, the display image consists of a rendering image display area 1101 and a rendering image non-display area 1102, which is created by narrowing the rendering field of view, similar to the example of the second embodiment described above. If the rendering image non-display area 1102 is displayed in black, for example, the user using the HMD 101 may experience a sense of visual pressure. Therefore, in this embodiment, the status display unit 901 generates a display image in which information 1103 indicating that capture is in progress is superimposed on the rendering image non-display area 1102. This reduces the sense of visual pressure on the user compared to simply displaying the rendering image non-display area 1102 in solid black. In Figure 11, an example of information 1103 indicating that capture is in progress is shown, which includes the word "REC" indicating that recording is in progress and corner marks indicating the four corners of the frame surrounding the rendering image display area 1101. Note that the display example of information indicating that capture is in progress is not limited to this example. Then, after S1001, the processing of the information processing device 102 proceeds to S1002.

[0059] In S1002, if the rendering field of view is changed in S802, for example, the display unit 404 displays the display image shown in Figure 11 on the display 203. After that, the information processing device 102 terminates the processing of the flowchart in Figure 10.

[0060] The information processing device 102 of the third embodiment can reduce the visual and psychological pressure caused by narrowing the rendering field of view by displaying information indicating that capture is in progress. In other words, according to this embodiment, it is possible to improve the difficulty of viewing images when screen capturing is performed on video, while optimizing the rendering processing load and reducing the visual and psychological pressure on the user.

[0061] <Fourth Embodiment> In the second embodiment described above, an example was explained in which the rendering field of view was narrowed in order to reduce the processing load. However, when a still image is captured instead of a video screen capture, only one frame is generated, so the temporary increase in processing load for rendering is small. In other words, when a still image is captured, it is not necessary to perform the process of narrowing the rendering field of view as described in the second embodiment. Therefore, in the fourth embodiment, we will describe an example in which the processing is switched depending on whether the screen capture mode is a still image capture mode or a video capture mode.

[0062] Figure 12 shows the functional configuration of the image display system according to the fourth embodiment, focusing particularly on the configuration of the information processing device 102. The information processing device 102 according to the fourth embodiment has a mode determination unit 1201 in addition to the configuration described in the second embodiment described above. The other configurations, excluding the mode determination unit 1201, are the same as those in Figure 7, so their explanation is omitted.

[0063] If the state determination unit 401 determines that a capture is in progress, the mode determination unit 1201 determines whether the mode of the currently running screen capture is a still image capture mode or a video capture mode. The mode determination unit 1201 then notifies the load acquisition unit 701 of the result of the capture mode determination.

[0064] Figure 13 is a flowchart showing the information processing flow, particularly that performed by the information processing device 102, in the image display system according to the fourth embodiment. The series of processes shown in the flowchart in Figure 14 are realized when the information processing program according to the fourth embodiment, stored in the ROM 303 or storage device 304, is loaded into the RAM 302 and executed by the CPU 301. As a result, the CPU 301 fulfills the role of each component shown in Figure 12. The processing in the flowchart in Figure 13 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, images can be displayed as a video on the HMD 101 for the user to view, and the screen-captured video can be saved as a file in the storage device 304. Similarly, when a still image is screen-captured, the screen-captured still image can be displayed on the HMD 101 for the user to view, and the screen-captured still image can be saved as a file in the storage device 304. In the flowchart shown in Figure 13, the same processing steps as in the flowchart in Figure 8 described above are denoted by the same reference numerals as in Figure 8, and their explanations are omitted.

[0065] In the flowchart shown in Figure 13, at S501, the state determination unit 401 determines that capture is in progress, and then at S502, the rendering setting unit 402 sets it to FFR, after which the information processing device 102 proceeds to S1301. In S1301, the mode determination unit 1201 determines whether the screen capture mode is video capture mode. If the mode determination unit 1201 determines that it is video capture mode, the information processing device 102 proceeds to S801. From S801 onward, the information processing device 102 performs a process to change the rendering field of view according to the increase in processing load, similar to what was described in the second embodiment.

[0066] On the other hand, if the mode determination unit 1201 determines in S1301 that it is not in video capture mode, that is, if it is determined to be in still image capture mode, the information processing device 102 proceeds to S803. In this case, if it is in still image capture mode, the rendering setting unit 402 sets the rendering field of view to the default field of view. In other words, if it is in still image capture mode, the rendering setting unit 402 does not change the rendering field of view in accordance with the increase in processing load. After S803, the information processing device 102 proceeds to S504 and beyond as described above.

[0067] In the fourth embodiment, the information processing device 102 performs a process to change the rendering field of view according to the increase in processing load, as described in the second embodiment, when in video capture mode, and sets it to the default rendering field of view when in still image capture mode. In other words, according to this embodiment, when in still image capture mode, the process to narrow the rendering field of view, as described in the second embodiment, is not performed, so it becomes possible to save images with a wider field of view to a file than in video capture mode. In addition, in the fourth embodiment, as described in the third embodiment above, it is also possible to apply a configuration that includes a process to superimpose information indicating that capture is in progress when using video capture mode.

[0068] <Fifth Embodiment> In the second embodiment described above, an example was explained in which, when the FFR settings are changed during screen capture, the rendering processing load is optimized by narrowing the rendering field of view to offset the increase in processing load. This means that the position and size of the high-resolution area in FR and the rendering field of view change before and during screen capture. However, if the user cannot know the high-resolution area or rendering field of view beforehand, the displayed image may appear unnatural to the user. Therefore, in the fifth embodiment, when a user gives a screen capture instruction, an example is described in which confirmation information is displayed before the start of the screen capture so that the user can confirm the high-resolution area and rendering angle during the screen capture.

[0069] Figure 14 is a diagram showing the functional configuration of the image display system according to the fifth embodiment, focusing particularly on the configuration of the information processing device 102. In addition to the configuration described in the second embodiment above, the information processing device 102 according to the fifth embodiment has a receiving unit 1401 and an information presentation unit 1402. The other configurations, excluding the receiving unit 1401 and the information presentation unit 1402, are the same as those in Figure 7, so their explanation is omitted.

[0070] The receiving unit 1401 receives instructions from the user to start and end screen captures, as well as instructions to end confirmation. The user instruction information received by the receiving unit 1401 is sent to the status determination unit 401. Therefore, in the fifth embodiment, the status determination unit 401 determines that a capture is in progress, for example, when it receives a screen capture start instruction from the receiving unit 1401. The information display unit 1402 superimposes information indicating the high-resolution area during screen capture onto the rendered image to generate a display image for display on the display unit 404.

[0071] Figure 15 is a flowchart showing the information processing flow, particularly that performed by the information processing device 102, in the image display system according to the fifth embodiment. The series of processes shown in the flowchart of Figure 15 are realized when the information processing program according to the fifth embodiment, stored in the ROM 303 or storage device 304, is loaded into the RAM 302 and executed by the CPU 301. In this way, the CPU 301 fulfills the role of each component shown in Figure 14. The processing in the flowchart of Figure 15 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, images can be displayed as a video on the HMD 101 for the user to view, and the screen-captured video can be saved as a file in the storage device 304. In the flowchart of Figure 15, the same processing steps as in the flowchart of Figure 8 described above are denoted by the same reference numerals as in Figure 8, and their explanations are omitted.

[0072] In the flowchart shown in Figure 15, the information processing device 102 first proceeds to S1501. In S1501, the state determination unit 401 transitions the state of the information processing device as necessary, based on the current state of the information processing device and instructions from the user. In this embodiment, there are three states for the information processing device: the state in which the user is viewing the rendered image, the state in which the user is confirming the image to be screen captured, and the state in the process of capturing. Hereinafter, the state in which the user is viewing the rendered image will be described as "viewing," and the state in which the user is confirming the image to be screen captured will be described as "confirming." Details of the state transition process in S1501 will be explained later with reference to Figure 16.

[0073] Next, in S1502, the state determination unit 401 determines whether the current state of the information processing device is viewing, checking, or capturing. If the state determination unit 401 determines that the current state of the information processing device is checking or capturing, the processing of the information processing device 102 proceeds to S502 or later. In the fifth embodiment, after S502, S801, S802, and S504, the processing of the information processing device 102 proceeds to S1503. On the other hand, if the state determination unit 401 determines that the current state of the information processing device is viewing, the processing of the information processing device 102 proceeds to S503. In the fifth embodiment, after S503, S803, and S504, the processing of the information processing device 102 proceeds to S505.

[0074] If the process proceeds to S1503, the state determination unit 401 determines whether the current state of the information processing device is being checked. If the state determination unit 401 determines that the current state of the information processing device is being checked, the information processing device 102 proceeds to S1504. On the other hand, if the state of the information processing device is determined not to be being checked, the information processing device 102 proceeds to S505.

[0075] When the process proceeds to S1504, the information presentation unit 1402 superimposes information indicating the high-resolution area during screen capture onto the rendered image to generate a display image for display on the display unit 404. After S1504, the information processing device 102 proceeds to S505.

[0076] Figure 17 shows an example of a display image generated by the information presentation unit 1402. As shown in Figure 17, the display image consists of a high-resolution area 1701, a low-resolution area 1702, and a rendering image non-display area 1703. Here, the high-resolution area 1701 is the image area rendered at high resolution, and the low-resolution area 1702 is the image area rendered at a lower resolution than high resolution. The rendering image non-display area 1703 is the same area as described in the third embodiment above.

[0077] Furthermore, in S1504, the information display unit 1402 superimposes information indicating the boundary between the high-resolution area 1701 and the low-resolution area 1702. This allows the user to easily recognize the high-resolution area. The superimposition of information indicating the boundary between the high-resolution area 1701 and the low-resolution area 1702 is realized, for example, by a rectangular dotted line frame 1704. Note that the method of displaying the boundary information is not limited to this example; for example, a solid line or a circular frame may be used, or a method may be used to highlight the high-resolution area 1701 by lowering the brightness of the low-resolution area 1702 compared to the brightness of the high-resolution area 1701. In addition, to clearly indicate that the current state of the information processing device is the confirmation screen of the high-resolution area during screen capture, information 1705 indicating that confirmation is in progress may be displayed at the same time. Figure 17 shows an example of information 1705 indicating that confirmation is in progress, such as the display of the word "Preview". After S1504, the processing of the information processing device 102 proceeds to S505.

[0078] In the fifth embodiment, after S505, the processing of the information processing device 102 proceeds to S1505. In S1505, the state determination unit 401 determines whether the current state of the information processing device is capturing data. If the state determination unit 401 determines that the data is capturing data, the information processing device 102 proceeds to S507. If the current state of the information processing device is determined not to be capturing data, or after the file is saved in S507, the information processing device 102 terminates the process shown in the flowchart of Figure 15.

[0079] Figure 16 is a flowchart of the state transition process performed by the state determination unit 401 in S1501. In S1601, the state determination unit 401 determines whether the current state of the information processing device is viewing, checking, or capturing. If the state determination unit 401 determines that the device is viewing, the information processing device 102 proceeds to S1602. If it determines that the device is checking, the information processing device 102 proceeds to S1604. If it determines that the device is capturing, the information processing device 102 proceeds to S1606.

[0080] If the process proceeds to S1602, the receiving unit 1401 determines whether a screen capture start command has been sent from the user. If the receiving unit 1401 determines that a screen capture start command has been sent, the information processing device 102 proceeds to S1603. On the other hand, if it determines that no screen capture start command has been sent, the information processing device 102 terminates the state transition process shown in Figure 16. When the process proceeds to S1603, the state determination unit 401 changes the state of the current information processing device to "checking". After S1603, the information processing device 102 terminates the state transition process shown in Figure 16.

[0081] If the process proceeds to S1604, the state determination unit 401 determines whether the receiving unit 1401 has received an instruction from the user to end the confirmation process. If the state determination unit 401 determines that the receiving unit 1401 has received an instruction to end the confirmation process, the information processing device 102 proceeds to S1605. On the other hand, if the state determination unit 401 determines that no instruction to end the confirmation process has been received, the information processing device 102 terminates the state transition process shown in Figure 16. When the process proceeds to S1605, the state determination unit 401 changes the current state of the information processing device while capturing data. After S1605, the information processing device 102 terminates the state transition process shown in Figure 16.

[0082] If the process proceeds to S1606, the state determination unit 401 determines whether the receiving unit 1401 has received a screen capture termination instruction from the user. If the state determination unit 401 determines that the receiving unit 1401 has received a screen capture termination instruction, the information processing device 102 proceeds to S1607. On the other hand, if it determines that no screen capture termination instruction has been received, the information processing device 102 terminates the state transition process shown in Figure 16. When the process proceeds to S1607, the state determination unit 401 changes the current state of the information processing device to "viewing". After S1607, the information processing device 102 terminates the state transition process shown in Figure 16.

[0083] According to the fifth embodiment, the user can confirm the rendering angle and high-resolution area at the time of screen capture execution before executing the screen capture. This allows the user to move the subject to be captured to a position where it can be saved in high resolution before instructing the start of the screen capture.

[0084] <Sixth Embodiment> In the fifth embodiment described above, an example was explained in which the user can check the rendering field of view and high-resolution area before executing the screen capture. If the user is not satisfied with the rendering field of view or the size of the high-resolution area that they have checked, it is conceivable that they may want to change these settings. Therefore, in this sixth embodiment, we will describe an example in which the user can change the rendering settings applied during screen capture while checking. In addition to the user changing the rendering settings, the sixth embodiment will also describe a display method that allows the user to check at once how the rendering field of view and high-resolution area change when the rendering settings are changed.

[0085] Figure 18 shows the functional configuration of the image display system according to the sixth embodiment, focusing particularly on the configuration of the information processing device 102. In addition to the configuration described in the fifth embodiment above, the information processing device 102 according to the sixth embodiment has a capture setting unit 1801 and a UI presentation unit 1802. The other configurations, excluding the capture setting unit 1801 and the UI presentation unit 1802, are the same as those in Figure 14, so their descriptions are omitted.

[0086] The capture settings unit 1801 changes the rendering settings during screen capture execution based on instructions from the user. The UI presentation unit 1802 generates a display image for display on the display unit 404, overlaying a user interface (UI) for the user to instruct on changing rendering settings during capture or starting screen capture.

[0087] Figure 19 is a flowchart showing the information processing flow, particularly that performed by the information processing device 102, in the image display system according to the sixth embodiment. The series of processes shown in the flowchart of Figure 19 are realized when the information processing program according to the sixth embodiment, stored in the ROM 303 or storage device 304, is loaded into the RAM 302 and executed by the CPU 301. In this way, the CPU 301 fulfills the role of each component shown in Figure 18. The processing in the flowchart of Figure 19 shows the processing flow per frame. Therefore, by repeatedly executing this process every frame, images can be displayed as a video on the HMD 101 for the user to view, and the screen-captured video can be saved as a file in the storage device 304. In the flowchart of Figure 19, the same processing steps as in Figure 15 are denoted by the same reference numerals as in Figure 15, and their explanations are omitted.

[0088] In the sixth embodiment, if the state determination unit 401 determines in S1502 that the current state of the information processing device is being checked or captured, the processing of the information processing device 102 proceeds to S1901. When proceeding to S1901, the capture setting unit 1801 changes the rendering setting to a setting for screen capture. In the sixth embodiment, the rendering setting indicates the frame rate and the rendering angle of view. Note that the rendering setting is not limited to this. Also, the setting for screen capture is, by default, for example, the frame rate is adjusted to the refresh rate of the display panel of the HMD 101, and the rendering angle of view is adjusted to the viewing angle of the HMD 101. The setting for screen capture can be changed according to an instruction from the user. The method of the instruction from the user will be described in S1905 which will be described later.

[0089] Next, in S1902, the capture setting unit 1801 calculates the high-resolution region of the resolution map required for the FFR process based on the rendering setting for screen capture set in S1901. The size of the high-resolution region is determined based on the increase amount of the processing load required according to the rendering setting. For example, let the default frame rate be f, the horizontal pixel number of the rendering be w, the vertical pixel number be h, the horizontal pixel number of the high-resolution region of the resolution map for FFR be w FFR , the vertical pixel number be h FFR , and the ratio of reducing the resolution by FFR be R FR . The number of rendering calculations N f at this time can be calculated by the following formula (4).

[0090] N f = w FFR × h FFR + (w × h - w FFR × h FFR ) × R FR Formula (4)

[0091] Here, considering the case where the frame rate f is changed to f', the ratio R f of the frame rate can be calculated as in the following formula (5).

[0092] R f = f' / f Formula (5)

[0093] Also, when the frame rate f is changed to f', the number of horizontal pixels in the high-resolution region of the resolution map is w'. FFR , the number of vertical pixels is h' FFR Therefore, the number of rendering calculations is N. f ' can be calculated as shown in equation (6) below.

[0094] N f '={w' FFR ×h' FFR +(w×hw' FFR ×h' FFR )×R FR}×R f Formula (6)

[0095] Then, using these, the number of pixels in the high-resolution region such that the number of rendering calculations does not change even when the frame rate is changed to f' can be calculated using the following equations (7) and (8).

[0096] N f =N f ' Equation (7) w' FFR ×h' FFR ={R FR ×(1-R f ) × w × h + (1 - R FR )×w FFR ×h FFR} / {R f ×(1-R FR )} Formula (8)

[0097] Assuming that the aspect ratio of the high-resolution region is maintained, the number of pixels in the high-resolution region can be calculated using equations (9) to (11) below.

[0098] w FFR :h FFR =w' FFR :h' FFR Formula (9) w' FFR =√[{(R FR ×(1-R f ) × w × h + (1 - R FR )×w FFR ×h FFR ) / (Rf ×(1-R FR ))}×{w FFR / h FFR}] Formula (10) h' FFR =√[{(R FR (1-R f ) × w × h + (1 - R FR )×w FFR ×h FFR ) / (R f ×(1-R FR ))}×{h FFR / w FFR}] Formula (11)

[0099] Here, an example of maintaining the aspect ratio has been described, but this is not the only way to set the high-resolution area, and the ratio of horizontal pixels to vertical pixels can be set arbitrarily. The capture setting unit 1801 creates a resolution map for each setting in advance using the required size of the high-resolution area as described above, and sets it appropriately. Then, after S1902, the processing of the information processing device 102 proceeds to S802 and beyond.

[0100] In the sixth embodiment, after S503, the processing of the information processing device 102 proceeds to S1903. At S1903, the rendering settings unit 402 changes the rendering settings to the default settings. The default settings are, for example, the frame rate is set to match the refresh rate of the HMD's display panel, the rendering field of view is set to match the HMD's field of view, and the ETFR resolution map is based on the line of sight. After S1903, the processing of the information processing device 102 proceeds to S504 and beyond.

[0101] In the sixth embodiment, if the state determination unit 401 determines in S1503 that the current state of the information processing device is being checked, the processing of the information processing device 102 proceeds to S1904. When the process proceeds to S1904, the UI display unit 1802 overlays a UI onto the rendered image for the user to instruct on changing rendering settings during capture or to start screen capture.

[0102] Figure 20 shows an example of a rendered image in which the UI is superimposed by the UI presentation unit 1802. Figure 20(a) shows an example of a rendered image with a UI for changing the frame rate superimposed. The UI for instructing the user to change the setting can be something like a pull-down menu 2001, which allows the user to select one of several predefined setting values. For example, the frame rate setting values ​​could be integer values ​​such as 1 / 2, 1 / 3, etc., based on the refresh rate of the HMD101 display panel. The value selected by this UI is applied to the rendering settings in S1901 and used for rendering. Note that the pull-down menu 2001 is just an example and is not limited to this; radio buttons, for example, could also be used. Furthermore, the rendering settings that can be changed are not limited to the frame rate; it may also be possible to change the rendering field of view.

[0103] Figure 20(b) shows an example where radio button 2011 is used as a UI for instructing a change in the rendering field of view setting. As a selection of the rendering field of view setting value, for example, an appropriate integer value can be used with the field of view of the HMD 101 as the maximum value. In addition, the UI presentation unit 1802 may also superimpose a button UI for instructing the start of screen capture, such as button 2002 in Figure 20(a) or button 2012 in Figure 20(b).

[0104] Next, in S1905, the information presentation unit 1402 generates a display image for display on the display unit 404 by superimposing information indicating the high-resolution area and rendering field of view during screen capture onto the image generated in S1904. In this embodiment, as explained in S1904, it is possible to change the rendering settings from several options based on user instructions. In this embodiment, the information presentation unit 1402 displays the high-resolution area for each option to show how the high-resolution area changes for each option.

[0105] Figure 20(a) shows an example of a display image when the frame rate can be changed. In this example, the frame rate can be selected from two values: 30 FPS (frames per second) and 60 FPS, and currently 60 FPS is selected. In this example, the high-resolution area at the current setting of 60 FPS is shown by a solid rectangle 2003, and the high-resolution area at the selectable value of 30 FPS is shown by a dotted rectangle 2004. As another example, Figure 20(b) shows an example of a display image when the rendering field of view can be changed. In this example, the rendering field of view can be selected from two values: 90° and 120°, and currently 90° is selected. In this case, the high-resolution area at the current setting of 90° is shown by a solid rectangle 2013, and the high-resolution area at the selectable value of 120° is shown by a dotted rectangle 2014. Furthermore, as information indicating the rendering field of view, the current rendering field of view is shown by a thick solid rectangle 2015, and the selectable rendering field of view of 120° is shown by a thick dotted rectangle 2016. After S1905, the processing of the information processing device 102 proceeds to S505, and then to S1906.

[0106] At step 1906, the status determination unit 401 determines whether the current state of the information processing device is viewing, capturing, or checking. If the status determination unit 401 determines that the device is capturing, the information processing device 102 proceeds to S507. If it determines that the device is checking, the information processing device 102 proceeds to S1907. On the other hand, if it determines that the device is viewing, the information processing device 102 terminates the process shown in the flowchart of Figure 19.

[0107] If the process proceeds to S1907, the capture setting unit 1801 retrieves the rendering settings set by the user using the instruction UI superimposed in S1904. Next, in S1908, the capture setting unit 1801 updates the screen capture settings used in S1901 with the rendering settings acquired in S1907. After S1907, the information processing device 102 terminates the processing of the flowchart in Figure 19.

[0108] According to the information processing device 102 of the sixth embodiment, the user can change the rendering settings applied during screen capture while confirming the image. This allows the user to change the rendering settings if they are not satisfied with the rendering field of view or the size of the high-resolution area confirmed on the confirmation screen.

[0109] <Modified examples of embodiments> In the fifth embodiment described above, information indicating the high-resolution area was superimposed only during verification, but information indicating the high-resolution area may also be superimposed during capture. Furthermore, in the fifth embodiment, if a screen capture start command is received while viewing, the current state of the information processing device is determined to be "checking," and if a screen capture start command is received while checking, the current state is determined to be "capturing." However, the method for determining the current state based on the presence or absence of a screen capture start command is not limited to this. For example, if a screen capture start command is received while viewing, the current state may be determined to be "capturing." In this case, the "checking" state is skipped, and screen capture is started immediately. In the sixth embodiment described above, an instruction UI was superimposed, and the user used it to give instructions to change rendering settings or to start screen capture. However, the method by which the user gives instructions is not limited to this. For example, instructions can be assigned to buttons or sticks on a controller, and the user can use them to give instructions. In that case, the superimposition process of the instruction UI in S1904 does not need to be performed.

[0110] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0111] This embodiment includes the following configurations, methods, and programs. (Composition 1) A setting means to configure the rendering resolution to differ between areas of the screen corresponding to the user's line of sight and other areas, A rendering means that renders an image to be displayed on the screen according to the rendering resolution set by the setting means, A state determination means for determining whether capturing is in progress for the image displayed on the aforementioned screen, It has, The setting means is characterized in that, when the state determination means determines that the capture is in progress, it stops setting the rendering resolution for the area corresponding to the user's gaze and the other areas, and sets the rendering resolution for a specific area of ​​the screen to a predetermined resolution. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the setting means sets the rendering resolution for areas other than the specific area to a resolution lower than the predetermined resolution. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the setting means is set to perform foveal rendering by eye tracking, which sets the rendering resolution to differ between the region corresponding to the user's gaze and the other regions. (Composition 4) The information processing device according to any one of configurations 1 to 3, characterized in that, when it is determined that the capture is in progress, the setting means is set to a predetermined resolution for rendering a specific area of ​​the screen, and is set to fixed foveal rendering. (Composition 5) The system further includes load acquisition means for acquiring the rendering processing load, The information processing device according to any one of configurations 1 to 4, characterized in that the setting means further sets the rendering field of view based on the rendering processing load when it is determined that the capture is in progress. (Composition 6) The information processing apparatus according to configuration 5, characterized in that the load acquisition means acquires the rendering processing load based on the number of rendering calculations. (Composition 7) The information processing apparatus according to configuration 5 or 6, characterized in that the setting means is set to narrow the rendering field of view in accordance with the increase in the rendering processing load. (Composition 8) The information processing device according to any one of configurations 5 to 7, characterized in that, when it is determined that the capture is in progress, the setting means sets a specific area of ​​the screen to be rendered in high resolution using fixed foveal rendering, and sets the range of the area to be rendered in high resolution using fixed foveal rendering based on the rendering processing load. (Composition 9) The system further includes a mode determination means for determining whether the capture mode for the image displayed on the screen is a video capture mode or a still image capture mode. The information processing apparatus according to any one of configurations 5 to 8, characterized in that the setting means sets the rendering field of view based on the rendering processing load when the mode determination means determines that the video is in capture mode. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, further comprising an output means for outputting an image rendered by the rendering means when the state determination means determines that the capture is in progress. (Composition 11) The information processing apparatus according to any one of configurations 1 to 10, further comprising a display means for generating a display image based on an image rendered by the rendering means. (Composition 12) The information processing apparatus according to any one of configurations 1 to 11, further comprising a state presentation means for superimposing information indicating that the capture is in progress onto the displayed image. (Composition 13) The information processing apparatus according to configuration 11 or 12, further comprising information presentation means for superimposing on the display image at least one of the information indicating the rendering angle during the capture and the information indicating the rendering resolution. (Composition 14) The state determination means further determines whether the user is viewing the image rendered by the rendering means, the user is reviewing the image to be captured, or the capture is in progress. The information display means is characterized in that, when the state determination means determines that the status is being checked, it further superimposes information indicating that the status is being checked onto the display image. (Composition 15) The information display means is characterized in that it further superimposes a plurality of confirmation pieces of information corresponding to rendering settings that can be set by the setting means onto the displayed image, as described in configuration 14. (Composition 16) It further has a receiving means for receiving instructions from the user, The information processing apparatus according to configuration 14 or 15, characterized in that the state determination means determines that the capture is in progress when the receiving means receives a capture start instruction from the user, and determines that the viewing is in progress when the receiving means receives a capture end instruction from the user. (Composition 17) It further has a receiving means for receiving instructions from the user, The information processing apparatus according to configuration 14 or 15, characterized in that the state determination means determines that the confirmation is in progress when the receiving means receives a capture start instruction from the user, determines that the capture is in progress when the receiving means receives a confirmation end instruction from the user, and determines that the viewing is in progress when the receiving means receives a capture end instruction from the user. (Composition 18) It further has a receiving means for receiving instructions from the user, The information processing apparatus according to configuration 14 or 15, further comprising a capture setting means that, when the status determination means determines that the confirmation is in progress, performs rendering settings during the capture in accordance with instructions from the user received by the receiving means. (Composition 19) The information processing device according to any one of configurations 14 to 18, further comprising a UI presentation means that superimposes on the displayed image a user interface for the user to instruct at least one of the following: start capturing or change rendering settings while capturing is in progress, when the state determination means determines that the state is being checked. (Method 1) A setting process to configure the rendering resolution to differ between areas of the screen corresponding to the user's line of sight and other areas, A rendering step which renders an image to be displayed on the screen according to the rendering resolution set in the setting step, A state determination step to determine whether capturing is in progress for the image displayed on the aforementioned screen, It has, The setting step is characterized in that, if the state determination step determines that the capture is in progress, the setting of the rendering resolution for the area corresponding to the user's gaze and the other areas is stopped, and the rendering resolution for a specific area of ​​the screen is set to a predetermined resolution. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 to 19. [Explanation of symbols]

[0112] 401: State determination unit, 402: Rendering setting unit, 403: Rendering unit, 404: Display unit, 405: Output unit

Claims

1. A setting means to configure the rendering resolution to differ between areas of the screen corresponding to the user's line of sight and other areas, A rendering means that renders an image to be displayed on the screen according to the rendering resolution set by the setting means, A state determination means for determining whether capturing is in progress for the image displayed on the aforementioned screen, It has, The setting means is characterized in that, when the state determination means determines that the capture is in progress, it stops setting the rendering resolution for the area corresponding to the user's gaze and the other areas, and sets the rendering resolution for a specific area of ​​the screen to a predetermined resolution.

2. The information processing apparatus according to claim 1, wherein the setting means sets the rendering resolution for areas other than the specific area to a resolution lower than the predetermined resolution.

3. The information processing apparatus according to claim 1, characterized in that the setting means is set to perform foveal rendering by eye tracking, which sets the rendering resolution to differ between the area corresponding to the user's gaze and the other areas.

4. The information processing device according to claim 1, characterized in that, when it is determined that the capture is in progress, the setting means is set to a predetermined resolution for rendering a specific area of ​​the screen, thereby setting it to fixed foveal rendering.

5. The system further includes load acquisition means for acquiring the rendering processing load, The information processing apparatus according to claim 1, characterized in that the setting means further sets the rendering field of view based on the rendering processing load when it is determined that the capture is in progress.

6. The information processing apparatus according to claim 5, characterized in that the load acquisition means acquires the rendering processing load based on the number of rendering calculations.

7. The information processing apparatus according to claim 5, characterized in that the setting means is set to narrow the rendering field of view in accordance with the increase in the rendering processing load.

8. The information processing apparatus according to claim 5, characterized in that, when it is determined that the capture is in progress, the setting means sets a specific area of ​​the screen to be rendered in high resolution using fixed foveal rendering, and sets the range of the area to be rendered in high resolution using fixed foveal rendering based on the rendering processing load.

9. The system further includes a mode determination means for determining whether the capture mode for the image displayed on the screen is a video capture mode or a still image capture mode. The information processing apparatus according to claim 5, wherein the setting means sets the rendering field of view based on the rendering processing load when the mode determination means determines that the video is in capture mode.

10. The information processing apparatus according to claim 1, further comprising an output means for outputting an image rendered by the rendering means when the state determination means determines that the capture is in progress.

11. The information processing apparatus according to claim 1, further comprising a display means for generating a display image based on an image rendered by the rendering means.

12. The information processing apparatus according to claim 11, further comprising a state presentation means for superimposing information indicating that the capture is in progress onto the displayed image.

13. The information processing apparatus according to claim 11, further comprising information presentation means for superimposing on the display image at least one of the information indicating the rendering angle during the capture and the information indicating the rendering resolution.

14. The state determination means further determines whether the user is viewing the image rendered by the rendering means, the user is reviewing the image to be captured, or the capture is in progress. The information display means further superimposes information indicating that it is being checked onto the display image when the state determination means determines that it is being checked.

15. The information display means further superimposes a plurality of confirmation pieces of information corresponding to rendering settings that can be set by the setting means onto the displayed image, as described in claim 14.

16. It further has a receiving means for receiving instructions from the user, The information processing apparatus according to claim 14, wherein the state determination means determines that the capture is in progress when the receiving means receives a capture start instruction from the user, and determines that the viewing is in progress when the receiving means receives a capture end instruction from the user.

17. It further has a receiving means for receiving instructions from the user, The information processing apparatus according to claim 14, characterized in that the state determination means determines that the confirmation is in progress when the receiving means receives a capture start instruction from the user, determines that the capture is in progress when the receiving means receives a confirmation end instruction from the user, and determines that the viewing is in progress when the receiving means receives a capture end instruction from the user.

18. It further has a receiving means for receiving instructions from the user, The information processing apparatus according to claim 14, further comprising a capture setting means that, when the state determination means determines that the confirmation is in progress, the receiving means performs rendering settings during the execution of the capture in accordance with instructions from the user received by the receiving means.

19. The information processing apparatus according to claim 14, further comprising a UI presentation means that superimposes on the displayed image a user interface for the user to instruct at least one of the following: to start capturing or to change the rendering settings while capturing is in progress, when the state determination means determines that the state determination means is in the process of confirming.

20. A setting process to configure the rendering resolution to differ between areas of the screen corresponding to the user's line of sight and other areas, A rendering step which renders an image to be displayed on the screen according to the rendering resolution set in the setting step, A state determination step to determine whether capturing is in progress for the image displayed on the aforementioned screen, It has, The setting step is characterized in that, if the state determination step determines that the capture is in progress, the setting of the rendering resolution for the area corresponding to the user's gaze and the other areas is stopped, and the rendering resolution for a specific area of ​​the screen is set to a predetermined resolution.

21. Computers, A setting means to configure the rendering resolution to differ between areas of the screen corresponding to the user's line of sight and other areas, A rendering means that renders an image to be displayed on the screen according to the rendering resolution set by the setting means, A state determination means for determining whether capturing is in progress for the image displayed on the aforementioned screen, It has, The setting means is a program that, when the state determination means determines that the capture is in progress, stops setting the rendering resolution for the area corresponding to the user's gaze and the other areas, and sets the rendering resolution for a specific area of ​​the screen to a predetermined resolution.

Images