Image processing system, image processing method, and program

Abstract

The present invention enhances, by using a small amount of calculation, apparent image quality of a virtual image. At least one processor acquires an initial virtual image (50) having a region of interest (501) and a peripheral region (502), acquires an image of interest (51) on the basis of the initial virtual image (50), acquires a first image (51h) by executing, on the image of interest (51), a first image quality enhancement process for enhancing image quality of images, and acquires an output image (53) having a first region (531) and a second region (532) on the basis of an image indicating the peripheral region (502) and the first image (51h).

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] In recent years, various technologies have been developed to improve the image quality of virtual images obtained by rendering three-dimensional data representing a virtual space. For example, temporal anti-aliasing (TAA) is known as a technology that improves the image quality of a current frame based on accumulated information from past frames. TAA is a so-called rule-based method, but machine learning-based methods such as NVIDIA's Deep Learning Super Sampling (DLSS) have also been attracting attention recently.

[0003] When improving the image quality of a virtual image, if the image quality improvement process is applied to all areas of the image, the amount of calculation becomes large.

[0004] The present invention has been made in consideration of the above-mentioned problems, and one of its objects is to provide an image processing system, an image processing method, and a program that can improve the apparent image quality of a virtual image with a small amount of calculation.

[0005] An image processing system according to one embodiment of the present invention is an image processing system having at least one processor, wherein the at least one processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial image quality and having a focus area and a peripheral area other than the focus area; a focus image acquisition step for acquiring a focus image representing the focus area based on the initial virtual image; a first image acquisition step for acquiring a first image having a first image quality higher than the initial image quality by performing a first image quality improvement process on the focus image to improve the image quality; and an output image acquisition step for acquiring an output image having a first area corresponding to the focus area and having the first image quality, and a second area corresponding to the peripheral area and having a second image quality lower than the first image quality, based on the first image and an image representing the peripheral area.

[0006] FIG. 1 is a diagram showing an example of the overall configuration of an image processing system according to the present embodiment; FIG. 2 is a diagram showing an example of the hardware configuration of a game device according to the present embodiment; FIG. 3 is a diagram showing an example of the hardware configuration of a head-mounted display according to the present embodiment; FIG. 4 is a functional block diagram showing an example of functions implemented in the image processing system according to the present embodiment; FIG. 5 is a diagram showing an overall view of an example of processing executed by the image processing system according to the present embodiment; and FIG. 6 is a flow chart showing an example of processing executed by the image processing system according to the present embodiment.

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0008] [1. Overall Configuration of Image Processing System] Fig. 1 is a diagram showing an example of the overall configuration of an image processing system 10 according to this embodiment. As shown in Fig. 1, the image processing system 10 according to this embodiment includes, as an example, a head-mounted display (hereinafter sometimes referred to as "HMD") 12, a game device 14, a relay device 16, a television 18, a camera / microphone unit 20, and a controller 22. Note that the overall configuration of the image processing system 10 shown in Fig. 1 is an example and is not limited to this. For example, the image processing system 10 does not have to include the television 18, the camera / microphone unit 20, or the relay device 16.

[0009] [1-1. Hardware Configuration of Game Device] Fig. 2 is a diagram showing an example of the hardware configuration of the game device 14 according to this embodiment. As shown in Fig. 2, the game device 14 includes, as an example, a processor 14a, a storage unit 14b, a communication unit 14c, and an input / output unit 14d.

[0010] The processor 14a is, for example, a program-controlled device such as a CPU that operates according to a program installed in the game device 14. The processor 14a also includes a GPU that draws images in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 14b includes, for example, a main storage device such as a ROM or RAM, and an auxiliary storage device such as a HDD or SSD. The storage unit 14b stores programs executed by the processor 14a. The storage unit 14b stores, for example, a game program (game software) in addition to programs for implementing various functions of the image processing system 10, which will be described later. The storage unit 14b also reserves a frame buffer area in which images are drawn by the GPU. The game device 14 may load and execute a program stored on a computer-readable information storage medium, such as an optical disc (e.g., a DVD-ROM or Blu-ray® disc) or a flash memory, into the storage unit 14b. Alternatively, the game device 14 may execute a program downloaded via a communication network, such as the Internet.

[0012] The communication unit 14c is a communication interface such as an Ethernet (registered trademark) module, a wireless LAN module, etc. The input / output unit 14d is an input / output port such as an HDMI (registered trademark) port, a USB port, etc. A controller 22 is connected to the input / output unit 14d as an operation unit that accepts input from a user.

[0013] The above-described hardware configurations of the game device 14 may also be included in the HMD 12 .

[0014] [1-2. Hardware Configuration of Head-Mounted Display] The HMD 12 is worn on the user's head and has a screen (display unit 12e) for displaying images. As shown in FIG. 1, the HMD 12 has, as an example, a main body and a wearing band. The wearing band is annular and surrounds the user's head as a whole. The main body houses the display unit 12e and the like and includes a housing that forms the exterior of the main body. The wearing band is connected to the top of the main body.

[0015] 3 is a diagram showing an example of the hardware configuration of the head-mounted display 12 according to the present embodiment. As shown in Fig. 3, the HMD 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an input / output unit 12d, a display unit 12e, a detection unit 12f, and a vibration unit 12g. The physical configurations of the processor 12a, the storage unit 12b, the communication unit 12c, and the input / output unit 12d may be similar to the physical configurations of the processor 14a, the storage unit 14b, the communication unit 14c, and the input / output unit 14d, respectively.

[0016] The display unit 12e is a screen such as a liquid crystal display or an organic EL display, and displays an image generated by the game device 14. The display unit 12e is placed in front of the eyes of the user when the user wears the HMD 12. For example, the display unit 12e receives a signal output by the game device 14 and relayed by the relay device 16, and outputs an image represented by the signal.

[0017] In this embodiment, the display unit 12e can display a three-dimensional image by displaying an image for the left eye and an image for the right eye. Note that the display unit 12e is not limited to a unit that displays three-dimensional images, and may be a unit that displays only two-dimensional images.

[0018] The detection unit 12f includes a gaze sensor that detects the user's gaze. The gaze sensor includes, for example, a camera or an infrared sensor. The gaze sensor may include a light-emitting unit that irradiates the user's pupil with infrared light or the like for observation. The gaze sensor is provided in the HMD 12 at a position where it can capture an image of the user's pupil, and may be provided, for example, inside the display unit 12e.

[0019] The detector 12f may also include a posture sensor that detects the posture of the HMD 12. The posture sensor is, for example, an IMU (Inertial Measurement Unit) that includes a gyro sensor, an acceleration sensor, etc. The detector 12f outputs the detection result to the processor 12a or the processor 14a.

[0020] The vibration unit 12g is, for example, a vibration device that constitutes a so-called eccentric motor. The vibration unit 12g is driven (vibrates) in accordance with instructions received from the game device 14. As shown in FIG. 1, the vibration unit 12g is provided, for example, at the front of the wearing band. Note that FIG. 1 shows the arrangement of the vibration unit 12g, and also shows its shape schematically. The vibration unit 12g is not limited to the arrangement shown in FIG. 1, as long as it is provided in the HMD 12 so as to vibrate the HMD 12.

[0021] The vibration unit 12g vibrates, for example, when an object or game character operated by the user takes some action or receives an impact. The vibration of the vibration unit 12g causes the wearing band to vibrate, and the vibration is transmitted to the user's head. This allows the user to experience a sense of realism.

[0022] 3 is an example, and is not intended to be limiting. For example, the HMD 12 does not necessarily have to include the orientation sensor or the vibration unit 12g.

[0023] 2. Functional Configuration of Image Processing System Functions implemented in the image processing system 10 according to this embodiment will be described below with reference to Fig. 4 and Fig. 5. Fig. 4 is a functional block diagram showing an example of functions implemented in the image processing system 10 according to this embodiment. Fig. 5 is a diagram showing an overall view of an example of processing executed in the image processing system 10 according to this embodiment.

[0024] [2-1. HMD] As shown in FIG. 4, the HMD 12 functionally includes a detection result acquisition unit 120 and an output unit 122.

[0025] The detection result acquisition unit 120 acquires the detection result from the gaze sensor. The detection result includes information indicating the user's gaze direction, such as the position of the user's pupil and the position of the light reflection point reflected in the user's pupil. The detection result acquired by the detection result acquisition unit 120 is transmitted to the game device 14 and is used for processing by the region-of-interest determination unit 143, which will be described later.

[0026] The output unit 122 displays on the display unit 12e an output image 53 transmitted from the game device 14. The output image 53 is acquired by an output image acquisition unit 148 of the game device 14, which will be described later.

[0027] 4, the game device 14 functionally includes a game processing unit 140, a rendering unit 141, an initial virtual image acquisition unit 142, a region of interest determination unit 143, a target image acquisition unit 144, a peripheral image acquisition unit 145, a first image acquisition unit 146, a second image acquisition unit 147, and an output image acquisition unit 148.

[0028] The game processing unit 140 executes various processes related to the game. For example, the game processing unit 140 executes processes such as placing game objects in a virtual space, operating or moving game objects, and changing the viewpoint from which the virtual space is viewed, in response to a game program executed by the processor 14a or user input received by the controller 22. Game objects are composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0029] The rendering unit 141 generates an initial virtual image 50 (see FIG. 5 ) by performing rendering (drawing processing) of three-dimensional data that represents a virtual space viewed from a predetermined viewpoint. The initial virtual image 50 is generated for each unit time (e.g., 1 / 120 seconds). In other words, the initial virtual image 50 is one of multiple frames that make up the game video. In the following description, the initial virtual image 50 may also be referred to as a frame.

[0030] The rendering unit 141 performs rendering based on the results of various processes executed by the game processing unit 140. Specifically, the rendering unit 141 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects placed in a virtual space. The vertex processing includes a coordinate transformation process (perspective projection) from a view coordinate system to a screen coordinate system, and a numerical value related to a change in viewpoint is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 141 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. In addition to the above processes, the rendering unit 141 may also perform processes to apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 141 may be appropriately set by a game software developer, etc.

[0031] The rendering unit 141 applies jitter when generating each frame. That is, the rendering unit 141 generates each frame by performing rendering so that the viewpoint varies from frame to frame. Here, even if the game processing unit 140 fixes the viewpoint at a predetermined position, the rendering unit 141 varies the viewpoint from frame to frame. As a result, the position of the displayed game object varies from frame to frame. Specifically, the rendering unit 141 varies the viewpoint from frame to frame by adding a numerical value corresponding to a sub-pixel size that differs from frame to frame to the perspective projection matrix. The rendering unit 141 varies the viewpoint from frame to frame according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0032] In this embodiment, in order to perform the first high-definition image processing (DLAA) and the second high-definition image processing (TAA) described later, a predetermined number of frames from the previous time generated by the rendering unit 141 are stored in the memory unit 14b.

[0033] The initial virtual image acquisition unit 142 acquires an initial virtual image 50 having an initial image quality (see FIG. 5 ). As shown in FIG. 5 , the initial virtual image 50 has a region of interest 501 and a peripheral region 502. The peripheral region 502 is a region of the initial virtual image 50 other than the region of interest 501. While FIG. 5 illustrates an example in which the region of interest 501 is rectangular, the region of interest 501 may have another shape, such as an ellipse.

[0034] The region-of-interest determination unit 143 determines a region of interest 501 in the initial virtual image 50 based on the detection result of the line-of-sight sensor (see FIG. 5 ). For example, the region-of-interest determination unit 143 identifies a pixel on the display unit 12e that the user is focusing on based on the detection result. The region-of-interest determination unit 143 then identifies a pixel on the initial virtual image 50 that corresponds to the pixel on the display unit 12e, and determines a predetermined range of area based on the pixel as the region of interest 501.

[0035] The image-of-interest acquisition unit 144 acquires an image of interest 51 indicating a region of interest 501 (see FIG. 5 ) based on the initial virtual image 50. For example, the pixel value of each pixel in the region of interest 501 is set for each pixel in the image of interest 51.

[0036] The peripheral image acquisition unit 145 acquires a peripheral image 52 indicating the peripheral region 502 based on the initial virtual image 50 (see FIG. 5 ). For example, the pixel value of each pixel in the peripheral image 52 is set for each pixel. In this embodiment, missing values ​​are set (null-filled) for pixels in a region in the peripheral image 52 that corresponds to the region of interest 501, but the pixel values ​​of pixels in the region of interest 501 in the initial virtual image 50 may also be set for the pixels in that region. That is, in this embodiment, the second image quality improvement process described below is not performed on the region in the peripheral image 52 that corresponds to the region of interest 501, but the second image quality improvement process may be performed on that region.

[0037] The first image acquisition unit 146 acquires a first image 51h (see FIG. 5) by performing a first image quality improvement process on the image of interest 51. The first image 51h has a first image quality that is higher than the initial image quality.

[0038] The first image quality improvement process is a process for improving the image quality. The first image quality improvement process may be, for example, a process for improving the image quality based on machine learning. In this embodiment, as an example, a case will be described in which the first image quality improvement process is NVIDIA's DLAA (Deep Learning Anti-Aliasing). However, the first image quality improvement process is not limited to this example and may be any process for improving the image quality.

[0039] For ease of explanation, the present embodiment illustrates a case where the first image quality improvement process does not increase the image resolution (does not increase the pixel density). That is, the resolution of the first image 51h is the same as the resolution of the region of interest 501, and therefore the initial virtual image 50. However, the first image quality improvement process may also be a process that increases the image resolution (for example, NVIDIA's DLSS).

[0040] In this specification, "image quality" refers to a high S / N ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed continuously), and other image qualities. For example, the image quality of a certain image A being higher than the image quality of another image B means that at least one of the S / N ratio, spatial frequency reproducibility, and temporal stability of image A is higher than at least one of the S / N ratio, spatial frequency reproducibility, and temporal stability of image B. Note that the evaluation index for image quality is not limited to the above-mentioned S / N ratio, spatial frequency reproducibility, and temporal stability, and any index related to image quality may be used. For example, the image quality of two images may be quantitatively compared using objective indexes such as PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity).

[0041] The second image acquisition unit 147 acquires a second image 52h (see FIG. 5) by performing a second image quality improvement process on the peripheral image 52. The second image 52h has a second image quality that is lower than the first image quality.

[0042] The second image quality improvement process is a process that improves the image quality with a smaller amount of calculation than the first image quality improvement process. The second image quality improvement process may be, for example, a rule-based process that improves the image quality. In this embodiment, as an example, a case where the second image quality improvement process is TAA will be described. However, the second image quality improvement process is not limited to this example, and may be any process that improves the image quality with a smaller amount of calculation than the first image quality improvement process.

[0043] Furthermore, in this embodiment, as with the first image quality improvement process, the second image quality improvement process does not increase the image resolution. That is, the resolution of the second image 52h is the same as the resolution of the surrounding area 502, and therefore the initial virtual image 50. However, the second image quality improvement process may increase the image resolution as long as it requires less calculation than the first image quality improvement process.

[0044] Furthermore, as described above, in this embodiment, the second image quality improvement process is not performed on the area in the peripheral image 52 that corresponds to the region of interest 501. That is, in this embodiment, similar to the peripheral image 52, missing values ​​are set for the pixels in the area in the second image 52h that corresponds to the region of interest 501.

[0045] The output image acquisition unit 148 acquires an output image 53 having a first region 531 and a second region 532 based on the first image 51h and the second image 52h (see FIG. 5). The first region 531 corresponds to the region of interest 501 and has a first image quality. The second region 532 corresponds to the surrounding region and has a second image quality.

[0046] Specifically, the output image acquisition unit 148 references each pixel of the first image 51h and draws a pixel in the first region 531 associated with that pixel, and also references each pixel of the second image 52h and draws a pixel in the second region 532 associated with that pixel, thereby acquiring the output image 53. In this embodiment, the resolution (pixel density) of the output image 53 is the same as the resolution of the initial virtual image 50. Each pixel in the first region 531 corresponds one-to-one with each pixel in the first image 51h, and each pixel in the second region 532 corresponds one-to-one with each pixel in the second image 52h (excluding the region corresponding to the region of interest 501). Note that the resolution of the output image 53 may be different from the resolution of the initial virtual image 50.

[0047] 5, the output image 53 has a shape corresponding to the shape (rounded rectangular shape) of the display unit 12e of the HMD 12. Therefore, in this embodiment, the output image 53 is output as is to the display unit 12e of the HMD 12 by the output unit 122. However, the output image 53 may be an ordinary rectangular image. In this case, the output image 53 needs to be converted into an image having a shape corresponding to the shape of the display unit 12e of the HMD 12.

[0048] The output image acquisition unit 148 may calculate the pixel values ​​of a boundary region by providing a boundary region between the first region 531 and the second region 532 and performing alpha blending based on the pixel values ​​of the pixels adjacent to the boundary region in the first region 531 and the pixel values ​​of the pixels adjacent to the boundary region in the second region 532. This smooths the boundary between the first region 531 and the second region 532, making the difference in image quality between these regions less noticeable.

[0049] 6 is a flow diagram showing an example of processing executed by the image processing system 10 according to this embodiment. The processing shown in FIG. 6 is executed by the processor 14a and the processor 12a operating in accordance with programs stored in the storage units 14b and 12b. Note that in this embodiment, the processing shown in FIG. 6 is executed each time the rendering unit 141 generates an initial virtual image 50 (frame).

[0050] 6, the processor 14a first acquires an initial virtual image 50 (S600). Next, the processor 14a determines a region of interest 501 in the initial virtual image 50 based on the detection result of the line-of-sight sensor (S601). The processor 14a then acquires a region of interest 51 based on the initial virtual image 50 (S602). The processor 14a also acquires a peripheral image 52 based on the initial virtual image 50 (S603).

[0051] The processor 14a executes a first image quality improvement process on the image of interest 51 to obtain a first image 51h (S604). The processor 14a also executes a second image quality improvement process on the peripheral image 52 to obtain a second image 52h (S605). The processor 14a acquires an output image 53 based on the first image 51h and the second image 52h (S606), transmits the output image 53 to the HMD 12, and ends the processing. The processor 12a outputs the output image 53 received from the HMD 12 to the display unit 12e (S607), and ends the processing.

[0052] 6 is an example and is not limited to this example. For example, the peripheral image 52 may be acquired first, followed by the image of interest 51. Similarly, the second image 52h may be acquired first, followed by the first image 51h. Alternatively, the processing may be performed in the following order: the image of interest 51, the first image 51h, the peripheral image 52, and the second image 52h.

[0053] [4. Summary] The image processing system 10 according to the present embodiment described above performs a first image quality improvement process on the target image 51 to improve the image quality, thereby obtaining a first image 51h having a first image quality higher than the initial image quality. Furthermore, the image processing system 10 performs a second image quality improvement process on the peripheral image 52 to improve the image quality with a smaller amount of calculation than the first image quality improvement process, thereby obtaining a second image 52h. The image processing system 10 then obtains an output image 53 based on the first image 51h and the second image 52h. This allows the apparent image quality of the initial virtual image 50 to be improved with a smaller amount of calculation than when the first image quality improvement process is performed on the entire region of the initial virtual image 50.

[0054] Furthermore, the image processing system 10 according to this embodiment determines the region of interest 501 of the initial virtual image 50 based on the detection result of the line-of-sight sensor. This allows the size of the region of interest 501 to be reduced compared to when the region of interest 501 is determined in advance. As a result, the size of the region on which the first image quality improvement process is to be performed can be reduced, and the apparent image quality of the initial virtual image 50 can be improved with a smaller amount of calculation.

[0055] [5. Modifications] The present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0056] For example, in this embodiment, an example is given of a case where the game processing unit 140, rendering unit 141, initial virtual image acquisition unit 142, focus area determination unit 143, focus image acquisition unit 144, peripheral image acquisition unit 145, first image acquisition unit 146, second image acquisition unit 147 and output image acquisition unit 148 are implemented in the game device 14, but at least one of these functional units may be implemented by a server connected to the game device 14 and HMD 12 via a network such as the Internet.

[0057] Furthermore, in the present embodiment, the image processing system 10 includes the peripheral image acquisition unit 145 and the second image acquisition unit 147, but the image processing system 10 does not need to include the peripheral image acquisition unit 145 and the second image acquisition unit 147. In other words, the image processing system 10 may be configured not to perform image quality improvement processing on the peripheral region 502. In this case, the output image acquisition unit 148 may refer to each pixel in the peripheral region 502 of the initial virtual image 50, and draw the pixels of the second region 532 associated with that pixel.

[0058] As described above, even if the image processing system 10 does not have the peripheral image acquisition unit 145 and the second image acquisition unit 147, it is natural that high-quality image processing is not performed on the peripheral area 502, but the apparent image quality of the initial virtual image 50 can be improved with a smaller amount of calculation than when the first high-quality image processing is performed on all areas of the initial virtual image 50.

[0059] Furthermore, the image processing system 10 does not need to include the region of interest determination unit 143. That is, in this embodiment, the region of interest 501 is determined for each frame by the region of interest determination unit 143 based on the detection result by the gaze sensor, but the region of interest 501 may also be an area of ​​a certain range that is determined in advance and is common to all frames.

[0060] The image processing system 10 may also apply a post-effect to at least one of the first image 51 h and the second image 52 h. Alternatively, the image processing system 10 may apply a post-effect to the display image 53.

[0061] In addition, in the present embodiment, the image processing system 10 is a game system including the game device 14 and the HMD 12, but the application of the present invention is not limited to game systems. For example, the present invention may also be applied to simulation systems for sports, vehicles, etc., or metaverse systems that provide a so-called metaverse space.

[0062] Furthermore, in this embodiment, an example has been given in which the initial virtual image 50 (and hence the output image 53) represents a so-called virtual reality (VR) space, but the initial virtual image 50 may also represent a mixed reality (MR) space.

[0063] Furthermore, in this embodiment, the initial virtual image 50 (and thus the output image 53) is one of multiple frames that make up a video (moving image), but the initial virtual image 50 may also be a still image.

[0064] [6. Supplementary Notes] (1) An image processing system including at least one processor, wherein the at least one processor executes: an initial virtual image acquisition step of acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial image quality and including a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step of acquiring an image of interest representing the region of interest based on the initial virtual image; a first image acquisition step of acquiring a first image having a first image quality higher than the initial image quality by performing a first image quality improvement process on the image of interest that improves image quality; and an output image acquisition step of acquiring an output image having a first region corresponding to the region of interest and having the first image quality, and a second region corresponding to the peripheral region and having a second image quality lower than the first image quality, based on the first image and an image representing the peripheral region. (2) The image processing system according to (1), wherein the at least one processor further executes: a peripheral image acquisition step of acquiring a peripheral image indicating the peripheral region based on the initial virtual image; and a second image acquisition step of acquiring the second image by executing a second image quality improvement process on the peripheral image that improves image quality with a smaller amount of calculation than the first image quality improvement process on the peripheral image, and in the output image acquisition step, acquires the output image based on the first image and the second image. (3) The image processing system according to (1) or (2), wherein the at least one processor, in the output image acquisition step, acquires the output image by referring to each pixel of the first image and drawing pixels of the first region associated with the pixel, and by referring to each pixel of the second image and drawing pixels of the second region associated with the pixel. (4) The image processing system according to any one of (1) to (3), further comprising a head-mounted display that is worn on a user's head and has a screen for displaying an image, and the at least one processor further executes an output step of outputting an image representing the output image to the screen.(5) The image processing system according to (4), wherein the head-mounted display further includes a gaze sensor that detects the user's gaze, and the at least one processor further executes a region-of-interest determination step that determines the region of interest based on a detection result by the gaze sensor.

Claims

1. An image processing system having at least one processor, wherein the at least one processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial image quality and including a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step for acquiring an image of interest representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first image quality higher than the initial image quality by performing a first image quality improvement process on the image of interest to improve the image quality; and an output image acquisition step for acquiring an output image having a first region corresponding to the region of interest and having the first image quality, and a second region corresponding to the peripheral region and having a second image quality lower than the first image quality, based on the first image and an image representing the peripheral region.

2. The image processing system of claim 1, wherein the at least one processor further executes: a peripheral image acquisition step for acquiring a peripheral image showing the peripheral area based on the initial virtual image; and a second image acquisition step for acquiring the second image by performing a second image quality improvement process on the peripheral image, the second image quality improvement process improving the image quality with a smaller amount of calculation than the first image quality improvement process; and wherein in the output image acquisition step, the output image is acquired based on the first image and the second image.

3. The image processing system of claim 1 or 2, wherein, in the output image acquisition step, the at least one processor acquires the output image by referencing each pixel of the first image and drawing the pixels of the first region associated with that pixel, and by referencing each pixel of the second image and drawing the pixels of the second region associated with that pixel.

4. The image processing system according to claim 1 or 2, further comprising a head-mounted display that is worn on the user's head and has a screen for displaying an image, and the at least one processor further performs an output step of outputting an image representing the output image to the screen.

5. The image processing system of claim 4, wherein the head-mounted display further comprises a gaze sensor that detects the user's gaze, and the at least one processor further executes a region of interest determination step that determines the region of interest based on the detection result by the gaze sensor.

6. An image processing method in which a processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial image quality and including a region of interest and a peripheral region other than the region of interest; a target image acquisition step for acquiring a region of interest representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first image quality higher than the initial image quality by performing a first image quality improvement process on the region of interest to improve the image quality; and an output image acquisition step for acquiring an output image based on the first image and an image representing the peripheral region, the output image having a first region corresponding to the region of interest and having the first image quality, and a second region corresponding to the peripheral region and having a second image quality lower than the first image quality.

7. A program that causes a computer to execute the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data that represents a virtual space, the initial virtual image having an initial image quality and having a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step for acquiring an image of interest that represents the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first image quality higher than the initial image quality by performing a first image quality improvement process on the image of interest that improves the image quality; and an output image acquisition step for acquiring an output image having a first region that corresponds to the region of interest and has the first image quality, and a second region that corresponds to the peripheral region and has a second image quality lower than the first image quality, based on the first image and an image that represents the peripheral region.

Images