Image processing system, image processing method, and program

Abstract

The present invention enhances the apparent definition of a virtual image with small memory consumption. At least one processor acquires an initial virtual image (50) having an initial resolution and having a region (501) of interest and a peripheral region (502), acquires an image (51) of interest on the basis of the initial virtual image (50), executes up-scaling processing on the image (51) of interest, thereby acquiring a first image (51h), generates a first region (531) corresponding to the region (501) of interest on the basis of the first image (51h), and generates a second region (532) corresponding to the peripheral region (502) on the basis of a second image (52h), thereby acquiring an output image (53) having the first region (531) and the second region (532).

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] In recent years, various technologies have been developed to improve the resolution of virtual images obtained by rendering three-dimensional data representing a virtual space. For example, machine learning-based techniques such as NVIDIA's Deep Learning Super Sampling (DLSS) have been attracting attention in recent years.

[0003] When increasing the resolution of an image, an upscaling process is performed on the original image to increase the resolution, but performing the upscaling process on all areas of the virtual image results in a large amount of memory (frame buffer) being consumed.

[0004] The present invention has been made in consideration of the above-mentioned problems, and one of its objects is to provide an image processing system, an image processing method, and a program that can increase the apparent resolution of a virtual image with small memory consumption.

[0005] An image processing system according to one embodiment of the present invention is an image processing system having at least one processor, wherein the at least one processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial resolution and having a region of interest and a peripheral region other than the region of interest; a target image acquisition step for acquiring a region of interest having the initial resolution and representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first resolution higher than the initial resolution by performing an upscaling process on the region of interest; and an output image acquisition step for acquiring an output image having the first region and the second region by generating a first region corresponding to the region of interest based on the first image and generating a second region corresponding to the peripheral region based on a second image having a second resolution lower than the first resolution.

[0006] FIG. 1 is a diagram showing an example of the overall configuration of an image processing system according to the present embodiment; FIG. 2 is a diagram showing an example of the hardware configuration of a game device according to the present embodiment; FIG. 3 is a diagram showing an example of the hardware configuration of a head-mounted display according to the present embodiment; FIG. 4 is a functional block diagram showing an example of functions implemented in the image processing system according to the present embodiment; FIG. 5 is a diagram showing an overall view of an example of processing executed by the image processing system according to the present embodiment; and FIG. 6 is a flow chart showing an example of processing executed by the image processing system according to the present embodiment.

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0008] [1. Overall Configuration of Image Processing System] Fig. 1 is a diagram showing an example of the overall configuration of an image processing system 10 according to this embodiment. As shown in Fig. 1, the image processing system 10 according to this embodiment includes, as an example, a head-mounted display (hereinafter sometimes referred to as "HMD") 12, a game device 14, a relay device 16, a television 18, a camera / microphone unit 20, and a controller 22. Note that the overall configuration of the image processing system 10 shown in Fig. 1 is an example and is not limited to this. For example, the image processing system 10 does not have to include the television 18, the camera / microphone unit 20, or the relay device 16.

[0009] [1-1. Hardware Configuration of Game Device] Fig. 2 is a diagram showing an example of the hardware configuration of the game device 14 according to this embodiment. As shown in Fig. 2, the game device 14 includes, as an example, a processor 14a, a storage unit 14b, a communication unit 14c, and an input / output unit 14d.

[0010] The processor 14a is, for example, a program-controlled device such as a CPU that operates according to a program installed in the game device 14. The processor 14a also includes a GPU that draws images in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 14b includes, for example, a main storage device such as a ROM or RAM, and an auxiliary storage device such as a HDD or SSD. The storage unit 14b stores programs executed by the processor 14a. The storage unit 14b stores, for example, a game program (game software) in addition to programs for implementing various functions of the image processing system 10, which will be described later. The storage unit 14b also reserves a frame buffer area in which images are drawn by the GPU. The game device 14 may load and execute a program stored on a computer-readable information storage medium, such as an optical disc (e.g., a DVD-ROM or Blu-ray® disc) or a flash memory, into the storage unit 14b. Alternatively, the game device 14 may execute a program downloaded via a communication network, such as the Internet.

[0012] The communication unit 14c is a communication interface such as an Ethernet (registered trademark) module, a wireless LAN module, etc. The input / output unit 14d is an input / output port such as an HDMI (registered trademark) port, a USB port, etc. A controller 22 is connected to the input / output unit 14d as an operation unit that accepts input from a user.

[0013] The above-described hardware configurations of the game device 14 may also be included in the HMD 12 .

[0014] [1-2. Hardware Configuration of Head-Mounted Display] The HMD 12 is worn on the user's head and has a screen (display unit 12e) for displaying images. As shown in FIG. 1, the HMD 12 has, as an example, a main body and a wearing band. The wearing band is annular and surrounds the user's head as a whole. The main body houses the display unit 12e and the like and includes a housing that forms the exterior of the main body. The wearing band is connected to the top of the main body.

[0015] 3 is a diagram showing an example of the hardware configuration of the head-mounted display 12 according to the present embodiment. As shown in Fig. 3, the HMD 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an input / output unit 12d, a display unit 12e, a detection unit 12f, and a vibration unit 12g. The physical configurations of the processor 12a, the storage unit 12b, the communication unit 12c, and the input / output unit 12d may be similar to the physical configurations of the processor 14a, the storage unit 14b, the communication unit 14c, and the input / output unit 14d, respectively.

[0016] The display unit 12e is a screen such as a liquid crystal display or an organic EL display, and displays an image generated by the game device 14. The display unit 12e is placed in front of the eyes of the user when the user wears the HMD 12. For example, the display unit 12e receives a signal output by the game device 14 and relayed by the relay device 16, and outputs an image represented by the signal.

[0017] In this embodiment, the display unit 12e can display a three-dimensional image by displaying an image for the left eye and an image for the right eye. Note that the display unit 12e is not limited to a unit that displays three-dimensional images, and may be a unit that displays only two-dimensional images.

[0018] The detection unit 12f includes a gaze sensor that detects the user's gaze. The gaze sensor includes, for example, a camera or an infrared sensor. The gaze sensor may include a light-emitting unit that irradiates the user's pupil with infrared light or the like for observation. The gaze sensor is provided in the HMD 12 at a position where it can capture an image of the user's pupil, and may be provided, for example, inside the display unit 12e.

[0019] The detector 12f may also include a posture sensor that detects the posture of the HMD 12. The posture sensor is, for example, an IMU (Inertial Measurement Unit) that includes a gyro sensor, an acceleration sensor, etc. The detector 12f outputs the detection result to the processor 12a or the processor 14a.

[0020] The vibration unit 12g is, for example, a vibration device that constitutes a so-called eccentric motor. The vibration unit 12g is driven (vibrates) in accordance with instructions received from the game device 14. As shown in FIG. 1, the vibration unit 12g is provided, for example, at the front of the wearing band. Note that FIG. 1 shows the arrangement of the vibration unit 12g, and also shows its shape schematically. The vibration unit 12g is not limited to the arrangement shown in FIG. 1, as long as it is provided in the HMD 12 so as to vibrate the HMD 12.

[0021] The vibration unit 12g vibrates, for example, when an object or game character operated by the user takes some action or receives an impact. The vibration of the vibration unit 12g causes the wearing band to vibrate, and the vibration is transmitted to the user's head. This allows the user to experience a sense of realism.

[0022] 3 is an example, and is not intended to be limiting. For example, the HMD 12 does not necessarily have to include the orientation sensor or the vibration unit 12g.

[0023] 2. Functional Configuration of Image Processing System Functions implemented in the image processing system 10 according to this embodiment will be described below with reference to Fig. 4 and Fig. 5. Fig. 4 is a functional block diagram showing an example of functions implemented in the image processing system 10 according to this embodiment. Fig. 5 is a diagram showing an overall view of an example of processing executed in the image processing system 10 according to this embodiment.

[0024] [2-1. HMD] As shown in FIG. 4, the HMD 12 functionally includes a detection result acquisition unit 120 and an output unit 122.

[0025] The detection result acquisition unit 120 acquires the detection result from the gaze sensor. The detection result includes information indicating the user's gaze direction, such as the position of the user's pupil and the position of the light reflection point reflected in the user's pupil. The detection result acquired by the detection result acquisition unit 120 is transmitted to the game device 14 and is used for processing by the region-of-interest determination unit 143, which will be described later.

[0026] The output unit 122 displays on the display unit 12e an output image 53 transmitted from the game device 14. The output image 53 is acquired by an output image acquisition unit 148 of the game device 14, which will be described later.

[0027] 4, the game device 14 functionally includes a game processing unit 140, a rendering unit 141, an initial virtual image acquisition unit 142, a region of interest determination unit 143, a target image acquisition unit 144, a peripheral image acquisition unit 145, a first image acquisition unit 146, a second image acquisition unit 147, and an output image acquisition unit 148.

[0028] The game processing unit 140 executes various processes related to the game. For example, the game processing unit 140 executes processes such as placing game objects in a virtual space, operating or moving game objects, and changing the viewpoint from which the virtual space is viewed, in response to a game program executed by the processor 14a or user input received by the controller 22. Game objects are composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0029] The rendering unit 141 generates an initial virtual image 50 (see FIG. 5 ) by performing rendering (drawing processing) of three-dimensional data that represents a virtual space viewed from a predetermined viewpoint. The initial virtual image 50 is generated for each unit time (e.g., 1 / 120 seconds). In other words, the initial virtual image 50 is one of multiple frames that make up the game video. In the following description, the initial virtual image 50 may also be referred to as a frame.

[0030] The rendering unit 141 performs rendering based on the results of various processes executed by the game processing unit 140. Specifically, the rendering unit 141 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects placed in a virtual space. The vertex processing includes a coordinate transformation process (perspective projection) from a view coordinate system to a screen coordinate system, and a numerical value related to a change in viewpoint is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 141 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. In addition to the above processes, the rendering unit 141 may also perform processes to apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 141 may be appropriately set by a game software developer, etc.

[0031] The rendering unit 141 applies jitter when generating each frame. That is, the rendering unit 141 generates each frame by performing rendering so that the viewpoint varies from frame to frame. Here, even if the game processing unit 140 fixes the viewpoint at a predetermined position, the rendering unit 141 varies the viewpoint from frame to frame. As a result, the position of the displayed game object varies from frame to frame. Specifically, the rendering unit 141 varies the viewpoint from frame to frame by adding a numerical value corresponding to a sub-pixel size that differs from frame to frame to the perspective projection matrix. The rendering unit 141 varies the viewpoint from frame to frame according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0032] In this embodiment, in order to perform the upscaling process (DLSS) and high-quality image processing (TAA) described below, a predetermined number of frames from the previous time generated by the rendering unit 141 are stored in the memory unit 14b.

[0033] The initial virtual image acquisition unit 142 acquires an initial virtual image 50 having an initial resolution (see FIG. 5 ). As shown in FIG. 5 , the initial virtual image 50 has a region of interest 501 and a peripheral region 502. The peripheral region 502 is a region of the initial virtual image 50 other than the region of interest 501. While FIG. 5 illustrates an example in which the region of interest 501 is rectangular, the region of interest 501 may have another shape, such as an ellipse.

[0034] The region-of-interest determination unit 143 determines a region of interest 501 in the initial virtual image 50 based on the detection result of the line-of-sight sensor (see FIG. 5 ). For example, the region-of-interest determination unit 143 identifies a pixel on the display unit 12e that the user is focusing on based on the detection result. The region-of-interest determination unit 143 then identifies a pixel on the initial virtual image 50 that corresponds to the pixel on the display unit 12e, and determines a predetermined range of area based on the pixel as the region of interest 501.

[0035] The image-of-interest acquisition unit 144 acquires an image of interest 51 indicating a region of interest 501 based on the initial virtual image 50 (see FIG. 5 ). For example, the pixel value of each pixel in the image of interest 51 is set to each pixel in the region of interest 501. The image of interest 51 has an initial resolution.

[0036] The peripheral image acquisition unit 145 acquires a peripheral image 52 indicating the peripheral region 502 based on the initial virtual image 50 (see FIG. 5 ). For example, the pixel value of each pixel in the peripheral image 52 is set for each pixel. In this embodiment, missing values ​​are set (null-filled) for pixels in a region in the peripheral image 52 that corresponds to the region of interest 501, but the pixel values ​​of pixels in the region of interest 501 in the initial virtual image 50 may also be set for the pixels in that region. That is, in this embodiment, the image quality improvement process described below is not performed on the region in the peripheral image 52 that corresponds to the region of interest 501, but the image quality improvement process may be performed on that region.

[0037] The first image acquisition unit 146 acquires a first image 51h by performing an upscaling process on the image of interest 51 (see FIG. 5). The first image 51h has a first resolution that is higher than the initial resolution. The first image acquisition unit 146 also stores the first image 51h in a first frame buffer. The first frame buffer is a memory area in the memory of the storage unit 14b.

[0038] Upscaling processing is processing that increases the resolution of an image. Upscaling processing can also be referred to as processing that increases the number of pixels (pixel density) of an image. In this embodiment, as an example, a case will be described in which the upscaling processing is NVIDIA DLSS. Note that DLSS not only increases the resolution of an image but also increases the image quality (interpolation processing, etc.), but it is also possible not to increase the image quality of the image of interest 51. Furthermore, the upscaling processing is not limited to this example, and may be any processing that increases the resolution of an image.

[0039] In this specification, "image quality" refers to a high S / N ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed continuously), and other image qualities. For example, the image quality of a certain image A being higher than the image quality of another image B means that at least one of the S / N ratio, spatial frequency reproducibility, and temporal stability of image A is higher than at least one of the S / N ratio, spatial frequency reproducibility, and temporal stability of image B. Note that the evaluation index for image quality is not limited to the above-mentioned S / N ratio, spatial frequency reproducibility, and temporal stability, and any index related to image quality may be used. For example, the image quality of two images may be quantitatively compared using objective indexes such as PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity).

[0040] The second image acquisition unit 147 acquires a second image 52h based on the peripheral image 52 (see FIG. 5). In this embodiment, the second image 52h is acquired by performing image quality improvement processing on the peripheral image 52. The second image 52h has a higher image quality than the image quality of the initial virtual image 50. The image quality improvement processing is a process for improving the image quality. In this embodiment, as an example, a case will be described in which the image quality improvement processing is TAA. However, the image quality improvement processing is not limited to this example and may be any type. It should be noted that the image quality improvement processing does not necessarily have to be performed on the peripheral image 52 (peripheral region 502).

[0041] The second image acquisition unit 147 also stores the second image 52h in a second frame buffer. The second frame buffer is a memory area in the memory of the storage unit 14b that is different from the first frame buffer. Because the size of the second image 52h is smaller than the size of the first image 51h, the size of the second frame buffer is smaller than the size of the first frame buffer.

[0042] As described above, in this embodiment, the image quality improvement process is not performed on the area in the peripheral image 52 that corresponds to the region of interest 501. That is, in this embodiment, similar to the peripheral image 52, missing values ​​are set for the pixels in the area in the second image 52h that corresponds to the region of interest 501.

[0043] The output image acquisition unit 148 generates a first region 531 corresponding to the region of interest 501 based on the first image 51h, and generates a second region 532 corresponding to the peripheral region 502 based on the second image 52h, thereby acquiring an output image 53 having the first region 531 and the second region 532 (see Figure 5).

[0044] Specifically, the output image acquisition unit 148 acquires the output image 53 by referring to the first frame buffer to draw pixels of the first region 531 that correspond to each pixel of the first image 51h, and by referring to the second frame buffer to draw pixels of the second region 532 that correspond to each pixel of the second image 52h.

[0045] The output image 53 has the first resolution in all regions. Therefore, each pixel in the first region 531 corresponds one-to-one with each pixel in the first image 51h. On the other hand, each pixel in the second region 532 corresponds to multiple pixels in the second image 52h (excluding the region corresponding to the region of interest 501). Therefore, the resolution of the second region 532 is lower than the resolution of the first region 531.

[0046] In this specification, definition refers to the degree to which the details of an object are clearly displayed in an image. Definition can be said to be one element of image quality. If the definition of image A is lower than the definition of image B, there will be a lot of noise such as jaggies even if the resolution of image A and the resolution of image B are the same. For example, the aforementioned indicators such as the signal-to-noise ratio, PSNR, and SSIM can be used to evaluate definition.

[0047] The output image 53 may have a resolution different from the first resolution, that is, the output image 53 only needs to have a resolution corresponding to the first resolution in all areas.

[0048] 5, the output image 53 has a shape corresponding to the shape (rounded rectangular shape) of the display unit 12e of the HMD 12. Therefore, in this embodiment, the output image 53 is output as is to the display unit 12e of the HMD 12 by the output unit 122. However, the output image 53 may be an ordinary rectangular image. In this case, the output image 53 needs to be converted into an image having a shape corresponding to the shape of the display unit 12e of the HMD 12.

[0049] The output image acquisition unit 148 may calculate the pixel values ​​of a boundary region by providing a boundary region between the first region 531 and the second region 532 and performing alpha blending based on the pixel values ​​of the pixels adjacent to the boundary region in the first region 531 and the pixel values ​​of the pixels adjacent to the boundary region in the second region 532. This smooths the boundary between the first region 531 and the second region 532, making the difference in image quality between these regions less noticeable.

[0050] 6 is a flow diagram showing an example of processing executed by the image processing system 10 according to this embodiment. The processing shown in FIG. 6 is executed by the processor 14a and the processor 12a operating in accordance with programs stored in the storage units 14b and 12b. Note that in this embodiment, the processing shown in FIG. 6 is executed each time the rendering unit 141 generates an initial virtual image 50 (frame).

[0051] 6, the processor 14a first acquires an initial virtual image 50 (S600). Next, the processor 14a determines a region of interest 501 in the initial virtual image 50 based on the detection result of the line-of-sight sensor (S601). The processor 14a then acquires a region of interest 51 based on the initial virtual image 50 (S602). The processor 14a also acquires a peripheral image 52 based on the initial virtual image 50 (S603).

[0052] The processor 14a performs upscaling processing on the image of interest 51 to obtain a first image 51h (S604) and stores the first image 51h in the first frame buffer (S605). The processor 14a also performs image quality improvement processing on the peripheral image 52 to obtain a second image 52h (S606) and stores the second image 52h in the second frame buffer (S607). The processor 14a generates a first region 531 corresponding to the region of interest 501 based on the first image 51h, and generates a second region 532 corresponding to the peripheral region 502 based on the second image 52h, thereby obtaining an output image 53 (S608), transmitting the output image 53 to the HMD 12, and ending the processing. The processor 12a outputs the output image 53 received from the HMD 12 to the display unit 12e (S609), and ending the processing.

[0053] 6 is an example and is not limited to this example. For example, the peripheral image 52 may be acquired first, followed by the image of interest 51. Similarly, the second image 52h may be acquired first, followed by the first image 51h. Alternatively, the processing may be performed in the following order: the image of interest 51, the first image 51h, the peripheral image 52, and the second image 52h.

[0054] [4. Summary] The image processing system 10 according to the present embodiment described above performs an upscaling process on the image of interest 51 to obtain a first image 51h having a first resolution higher than the initial resolution. The image processing system 10 then generates a first region 531 corresponding to the region of interest 501 based on the first image 51h, and generates a second region 532 corresponding to the peripheral region 502 based on a second image 52h having a second resolution lower than the first resolution, thereby obtaining an output image 53 having the first region 531 and the second region 532. This makes it possible to increase the apparent definition of the initial virtual image 50 with less memory consumption than when the upscaling process is performed on all regions of the initial virtual image 50.

[0055] Furthermore, the image processing system 10 according to this embodiment determines the region of interest 501 of the initial virtual image 50 based on the detection result of the line-of-sight sensor. This allows the size of the region of interest 501 to be smaller than when the region of interest 501 is determined in advance. As a result, the size of the region on which upscaling processing is to be performed can be reduced, and the apparent definition of the initial virtual image 50 can be increased with less memory consumption.

[0056] [5. Modifications] The present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0057] For example, in this embodiment, an example is given of a case where the game processing unit 140, rendering unit 141, initial virtual image acquisition unit 142, focus area determination unit 143, focus image acquisition unit 144, peripheral image acquisition unit 145, first image acquisition unit 146, second image acquisition unit 147 and output image acquisition unit 148 are implemented in the game device 14, but at least one of these functional units may be implemented by a server connected to the game device 14 and HMD 12 via a network such as the Internet.

[0058] Furthermore, in the present embodiment, the image processing system 10 includes the peripheral image acquisition unit 145 and the second image acquisition unit 147, but the image processing system 10 does not need to include the peripheral image acquisition unit 145 and the second image acquisition unit 147. In this case, the output image acquisition unit 148 may refer to the frame buffer in which the initial virtual image 50 is stored, and draw the pixels of the second region 532 associated with each pixel of the peripheral region 502.

[0059] As described above, even if the image processing system 10 does not have the peripheral image acquisition unit 145 and the second image acquisition unit 147, it is natural that the upscaling process is not performed on the peripheral area 502, but the apparent resolution of the initial virtual image 50 can be increased with less memory consumption than when the upscaling process is performed on all areas of the initial virtual image 50.

[0060] In the present embodiment, the resolution of the second image 52h (second resolution) is the same as the initial resolution, but the resolution of the second image 52h may be higher than the initial resolution as long as it is lower than the first resolution. The resolution of the second image 52h may also be lower than the initial resolution. By lowering the resolution of the second image 52h than the initial resolution, the apparent definition of the initial virtual image 50 can be increased with even less memory consumption.

[0061] Furthermore, the image processing system 10 does not need to include the region of interest determination unit 143. That is, in this embodiment, the region of interest 501 is determined for each frame by the region of interest determination unit 143 based on the detection result by the gaze sensor, but the region of interest 501 may also be an area of ​​a certain range that is determined in advance and is common to all frames.

[0062] The image processing system 10 may also apply a post-effect to at least one of the first image 51 h and the second image 52 h. Alternatively, the image processing system 10 may apply a post-effect to the display image 53.

[0063] In addition, in the present embodiment, the image processing system 10 is a game system including the game device 14 and the HMD 12, but the application of the present invention is not limited to game systems. For example, the present invention may also be applied to simulation systems for sports, vehicles, etc., or metaverse systems that provide a so-called metaverse space.

[0064] Furthermore, in this embodiment, an example has been given in which the initial virtual image 50 (and hence the output image 53) represents a so-called virtual reality (VR) space, but the initial virtual image 50 may also represent a mixed reality (MR) space.

[0065] Furthermore, in this embodiment, the initial virtual image 50 (and thus the output image 53) is one of multiple frames that make up a video (moving image), but the initial virtual image 50 may also be a still image.

[0066] [6. Supplementary Notes] (1) An image processing system including at least one processor, wherein the at least one processor executes: an initial virtual image acquisition step of acquiring an initial virtual image obtained by rendering three-dimensional data indicating a virtual space, the initial virtual image having an initial resolution and having a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step of acquiring an image of interest having the initial resolution and indicating the region of interest based on the initial virtual image; a first image acquisition step of acquiring a first image having a first resolution higher than the initial resolution by performing an upscaling process on the image of interest; and an output image acquisition step of acquiring an output image having the first region and the second region by generating a first region corresponding to the region of interest based on the first image and generating a second region corresponding to the peripheral region based on a second image having a second resolution lower than the first resolution. (2) The image processing system according to (1), wherein the at least one processor further executes: a peripheral image acquisition step of acquiring a peripheral image having the initial resolution and showing the peripheral region based on the initial virtual image; and a second image acquisition step of acquiring the second image based on the peripheral image. (3) The image processing system according to (1) or (2), wherein the image processing system further includes at least one memory, and the at least one processor stores the first image in a first frame buffer that is a memory area in the at least one memory, and stores the second image in a second frame buffer that is a memory area in the at least one memory different from the first frame buffer, and acquires the output image by referring to the first frame buffer to draw pixels in the first region corresponding to each pixel of the first image, and by referring to the second frame buffer to draw pixels in the second region corresponding to each pixel of the second image. (4) The image processing system according to any one of (1) to (3), wherein the output image has a resolution according to the first resolution in all regions.(5) The image processing system according to any one of (1) to (4), further comprising a head-mounted display worn on a user's head and having a screen for displaying an image, wherein the at least one processor further executes an output step of outputting an image representing the output image to the screen. (6) The image processing system according to (5), further comprising a gaze sensor that detects the user's gaze, wherein the at least one processor further executes a region-of-interest determination step of determining the region of interest based on a detection result by the gaze sensor.

Claims

1. An image processing system having at least one processor, wherein the at least one processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial resolution and having a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step for acquiring an image of interest having the initial resolution and representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first resolution higher than the initial resolution by performing an upscaling process on the image of interest; and an output image acquisition step for acquiring an output image having the first region and the second region by generating a first region corresponding to the region of interest based on the first image and generating a second region corresponding to the peripheral region based on a second image having a second resolution lower than the first resolution.

2. The image processing system of claim 1, wherein the at least one processor further executes a peripheral image acquisition step of acquiring a peripheral image having the initial resolution and showing the peripheral region based on the initial virtual image, and a second image acquisition step of acquiring the second image based on the peripheral image.

3. The image processing system according to claim 1 or 2, further comprising at least one memory, wherein the at least one processor stores the first image in a first frame buffer which is a memory area in the at least one memory, and stores the second image in a second frame buffer which is a memory area in the at least one memory different from the first frame buffer, and in the output image acquisition step, acquires the output image by referring to the first frame buffer to draw pixels in the first area corresponding to each pixel of the first image, and by referring to the second frame buffer to draw pixels in the second area corresponding to each pixel of the second image.

4. The image processing system according to claim 1 or 2, wherein the output image has a resolution corresponding to the first resolution in all areas.

5. The image processing system according to claim 1 or 2, further comprising a head-mounted display that is worn on the user's head and has a screen for displaying an image, and the at least one processor further performs an output step of outputting an image representing the output image to the screen.

6. The image processing system according to claim 5, wherein the head-mounted display further comprises a gaze sensor that detects the user's gaze, and the at least one processor further executes a region of interest determination step that determines the region of interest based on the detection result by the gaze sensor.

7. An image processing method in which a processor executes the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data representing a virtual space, the initial virtual image having an initial resolution and having a region of interest and a peripheral region other than the region of interest; a target image acquisition step for acquiring a region of interest having the initial resolution and representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first resolution higher than the initial resolution by performing an upscaling process on the image of interest; and an output image acquisition step for acquiring an output image having the first region and the second region by generating a first region corresponding to the region of interest based on the first image and generating a second region corresponding to the peripheral region based on a second image having a second resolution lower than the resolution of the first image.

8. A program that causes a computer to execute the following steps: an initial virtual image acquisition step for acquiring an initial virtual image obtained by rendering three-dimensional data that represents a virtual space, the initial virtual image having an initial resolution and having a region of interest and a peripheral region other than the region of interest; an image of interest acquisition step for acquiring an image of interest having the initial resolution and representing the region of interest based on the initial virtual image; a first image acquisition step for acquiring a first image having a first resolution higher than the initial resolution by performing an upscaling process on the image of interest; and an output image acquisition step for acquiring an output image having the first region and the second region by generating a first region corresponding to the region of interest based on the first image and generating a second region corresponding to the peripheral region based on a second image that has a second resolution lower than the resolution of the first image.

Images