Image processing program, image processing system, image processing method, and image processing apparatus

Abstract

Provided is an image processing program capable of displaying reflection even near the edge of the screen. 【Solution means】An image processing system according to an embodiment renders an object in a virtual space onto a frame buffer, sets, as an incident direction, a direction from a virtual camera toward a position in the virtual space corresponding to a target pixel among the drawn images, and calculates a reflection direction based on the incident direction and a normal direction. The image processing system calculates, as a ray tracing direction, a direction obtained by adding an in-screen correction to the calculated reflection direction, traces a ray along the ray tracing direction, and adds the color of the collision position of the ray to the color of the target pixel.

Description

Technical Field

[0001] The present invention relates to an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of expressing reflection based on a drawn image.

Background Art

[0002] Conventionally, there is a method of expressing reflection by performing processing on a drawn image (for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the above prior art is processing on a drawn image, for example, the reflection may be interrupted near the edge of the screen.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of displaying reflection even near the edge of the screen.

Means for Solving the Problems

[0006] To solve the above problems, the present invention adopts the following configuration.

[0007] (First Configuration) The image processing program of the first configuration causes a computer of an information processing apparatus to perform a first depth test and update the first depth buffer for an object in a virtual space using the first depth buffer, and cause drawing on a frame buffer based on the result of the first depth test. Further, the image processing program causes the computer, for each pixel of the frame buffer on which drawing has been performed, with the pixel as a target pixel, based on the depth of the first depth buffer, the direction from a virtual camera toward the position in the virtual space corresponding to the target pixel as an incident direction, the position as a reflection position, to calculate, as a ray tracing direction, a direction obtained by adding correction toward the inside of the screen to the reflection direction based on the incident direction and the normal direction of the reflection position, trace the ray along the ray tracing direction, determine a collision position where the ray collides with an object in the virtual space based on the first depth buffer, and when the collision position is determined within a range where the trace distance of the ray is within a first distance, determine, as a reflected color to be added to the color of the pixel in the frame buffer corresponding to the collision position, a color based on the color of the pixel in the frame buffer corresponding to the collision position.

[0008] According to the above, ray tracing can be performed in the drawn image, and the reflected color can be determined based on the drawn pixels even near the edge of the screen.

[0009] (Second configuration) In the second configuration, in the above first configuration, a correction may be made to move the screen coordinates of a terminal position, with a position at a second distance in the ray tracing direction from the reflection position as the terminal position, toward the inside of the screen.

[0010] According to the above, by performing correction to move the terminal position, the ray tracing direction can be calculated.

[0011] (Third configuration) In the third configuration, in the above second configuration, the second distance may be the first distance.

[0012] According to the above, by setting the position where the ray travels the maximum distance in the reflection direction as the end position and performing correction to move the end position, the ray tracing direction can be calculated.

[0013] (Fourth configuration) In the fourth configuration, in any of the first to third configurations above, the correction may be a correction that moves the screen coordinates of the end position closer to the screen coordinates of the reflection position by a predetermined degree.

[0014] According to the above, by bringing the screen coordinates of the end position closer to the screen coordinates of the reflection position, for example, the ray tracing direction can be made closer to the direction along the edge of the screen.

[0015] (Fifth configuration) In the fifth configuration, in the fourth configuration above, the degree may be higher as the screen coordinates of the reflection position are closer to the edge of the screen.

[0016] According to the above, since the degree of correction increases as it gets closer to the edge of the screen, it is possible to more easily keep the collision position of the ray within the range of the drawn image.

[0017] (Sixth configuration) In the sixth configuration, in any of the first to fourth configurations above, the correction may be a correction that is performed only when the screen coordinates of the reflection position are included in a predetermined range close to the edge of the screen.

[0018] According to the above, correction can be performed only in the range close to the edge of the screen, and unnatural reflections can be avoided.

[0019] (Seventh configuration) In the seventh configuration, in any of the first to fourth configurations above, the correction may be a correction that is performed only when the reflection direction is facing outside the screen.

[0020] According to the above, when the reflection direction faces the inside of the screen, no correction is performed, so unnatural reflections can be avoided.

[0021] (Eighth configuration) In the eighth configuration, in any of the first to seventh configurations described above, the correction may be performed only on the horizontal component of the screen coordinates.

[0022] According to the above, for example, correction suitable for games where the virtual space spreads horizontally can be performed.

[0023] Further, other embodiments may be an image processing system that performs the above image processing, an image processing apparatus, or an image processing method.

Advantages of the Invention

[0024] According to the present invention, the reflected color can be determined based on the pixels drawn even near the edge of the screen.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Mode for Carrying Out the Invention

[0026] (System Configuration) Hereinafter, a game system according to an example of the present embodiment will be described. An example of the game system 1 in the present embodiment includes a main body device (information processing device; functioning as a game device main body in the present embodiment) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are each detachable from the main body device 2. That is, the game system 1 can be used as a device in which the left controller 3 and the right controller 4 are each attached to the main body device 2 and integrated. Also, the game system 1 can be used with the main body device 2, the left controller 3, and the right controller 4 as separate entities. Hereinafter, the hardware configuration of the game system 1 of the present embodiment will be described, and then the control of the game system 1 of the present embodiment will be described.

[0027] FIG. 1 is a diagram showing an example of a state in which the left controller 3 and the right controller 4 are attached to the main body device 2. As shown in FIG. 1, the left controller 3 and the right controller 4 are each attached to the main body device 2 and integrated. The main body device 2 is a device that executes various processes (for example, game processes) in the game system 1. The main body device 2 includes a display 12. The left controller 3 and the right controller 4 are devices that include an operation unit for the user to input.

[0028] The left controller 3 and the right controller 4 are detachable from the main body device 2. In the following, the left controller 3 and the right controller 4 may be collectively referred to as the "controller".

[0029] The main body device 2 alone or the integrated device with the left controller 3 and the right controller 4 attached to the main body device 2 may be a portable device. Also, the main body device 2 or the integrated device may be a hand-held device. Further, the main body device 2 or the integrated device may be a transportable device.

[0030] Also, the main body device 2 is provided with a touch panel 13 on the screen of the display 12. In the present embodiment, the touch panel 13 is of a type capable of multi-touch input (for example, a capacitance type). However, the touch panel 13 may be of any type, for example, a type capable of single-touch input (for example, a resistive film type).

[0031] FIG. 2 is a block diagram showing an example of the internal configuration of the main body device 2.

[0032] The main body device 2 includes a processor 81. The processor 81 is an information processing unit that executes various information processes executed in the main body device 2, and includes one or more CPUs (Central Processing Units) and one or more GPUs (Graphics Processing Units). The processor 81 may be composed of an SoC (System-on-a-chip) including a plurality of functions such as a CPU function and a GPU function. Note that the CPU and the GPU may be configured as separate processors. Also, one or more memories for temporarily storing data are provided inside the processor 81. The processor 81 executes various information processes by executing an information processing program (for example, a game program) stored in a storage unit (specifically, an internal storage medium such as a flash memory 84 or an external storage medium mounted on the slot 23).

[0033] The main body device 2 includes a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85 as an example of an internal storage medium built therein. The flash memory 84 and the DRAM 85 are connected to the processor 81. The flash memory 84 is mainly a memory used to store various data (which may be programs) stored in the main body device 2. The DRAM 85 is a memory used to temporarily store various data used in information processing.

[0034] The main body device 2 includes a slot interface (hereinafter abbreviated as "I / F") 91. The slot I / F 91 is connected to the processor 81. The slot I / F 91 is connected to the slot 23 and reads and writes data to and from a predetermined type of storage medium (for example, a dedicated memory card) mounted on the slot 23 according to the instructions of the processor 81.

[0035] The processor 81 appropriately reads and writes data between the flash memory 84 and the DRAM 85, and each of the above storage media, and executes the above information processing.

[0036] The main body device 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates (specifically, wirelessly communicates) with an external device via a network. In this embodiment, the network communication unit 82 connects to a wireless LAN and communicates with an external device by a method compliant with the Wi-Fi standard as a first communication mode. Also, the network communication unit 82 performs wireless communication with another main body device 2 of the same type by a predetermined communication method (for example, communication using a proprietary protocol or infrared communication) as a second communication mode.

[0037] The main body device 2 includes a controller communication unit 83. The controller communication unit 83 is connected to a processor 81. The controller communication unit 83 performs wireless communication with the left controller 3 and / or the right controller 4. The communication method between the main body device 2 and the left controller 3 and the right controller 4 is arbitrary. However, in the present embodiment, the controller communication unit 83 performs communication according to the Bluetooth (registered trademark) standard between the left controller 3 and the right controller 4.

[0038] The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 performs wired communication with the left controller 3, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. Also, when the processor 81 performs wired communication with the right controller 4, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. Further, when the processor 81 communicates with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in the present embodiment, the main body device 2 can perform both wired communication and wireless communication with the left controller 3 and the right controller 4 respectively. Also, when the left controller 3 and the right controller 4 are attached to the main body device 2 as an integrated device or when the main body device 2 alone is attached to the cradle, the main body device 2 can output data (for example, image data and audio data) to a stationary monitor or the like via the cradle.

[0039] The main body device 2 includes a touch panel controller 86 which is a circuit for controlling the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. The touch panel controller 86 generates data indicating, for example, the position where a touch input is made based on a signal from the touch panel 13 and outputs it to the processor 81.

[0040] The main body device 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Also, although not shown, the power control unit 97 is connected to each part of the main body device 2 (specifically, each part that receives power supply from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls the power supply from the battery 98 to each of the above parts based on a command from the processor 81.

[0041] Also, the battery 98 is connected to the lower terminal 27. When an external charging device (for example, a cradle) is connected to the lower terminal 27 and power is supplied to the main body device 2 via the lower terminal 27, the supplied power is charged to the battery 98.

[0042] (Overview of Image Processing) Next, the image processing of this embodiment will be described. In the game system 1 (an example of an image processing system) of this embodiment, a plurality of objects are arranged in a three-dimensional virtual space and a game is played.

[0043] FIG. 3 is a diagram showing an example of a plurality of objects arranged in the virtual space during the execution of the game of this embodiment. When the game of this embodiment is started, a three-dimensional virtual space defined by the xyz orthogonal coordinate system is set. For example, the y-axis is the upward axis of the virtual space, the x-axis is the rightward axis, and the z-axis is the depthwise axis.

[0044] As shown in FIG. 3, in the virtual space, as a plurality of objects, a ground object 30 is arranged. The ground object 30 is an object imitating the ground and is, for example, a plane parallel to the xz plane. Note that the ground object 30 may have irregularities or slopes. On the ground object 30, a tree object 32 and a cylindrical object 34 are arranged. Also, for example, a mountain object 36 is arranged at a predetermined position in the z-axis direction of the virtual space. These ground object 30, tree object 32, and cylindrical object 34 are fixed in the virtual space.

[0045] In addition, character objects 41 and 43 are arranged in the virtual space. The character objects 41 and 43 are flat objects, and are plate-shaped objects whose length in the thickness direction is smaller than their lengths in the up-down, left-right directions. For example, the character objects 41 and 43 are 3D objects each including a planar mesh constituting the front surface and a planar mesh constituting the back surface. The character object 41 is a player character controlled by a player. The character object 41 performs actions according to inputs to the controller within the virtual space. For example, as actions, the character object 41 moves on the ground object 30, moves its limbs within the virtual space, jumps, or changes the direction of its face. When the character object 41 moves in the left-right direction, the screen is scrolled in the left-right direction. Also, the character object 43 is a non-player character that moves according to the movement of the character object 41 and is automatically controlled by the processor 81. Note that the character object 43 may be controlled by a player.

[0046] A virtual camera is arranged in the virtual space, a game image of the virtual space viewed from the virtual camera is generated, and the generated game image is displayed on the display 12 or a stationary monitor (hereinafter, these are referred to as "display devices").

[0047] FIG. 4 is a diagram showing an example of a game image displayed on the display device. Images of the respective objects (30, 32, 34, 36, 41, 43) are displayed on the display device. For example, the ground object 30 is a reflective surface, and the respective objects are reflected on the ground object 30. In the present embodiment, reflections of the respective objects are generated by a method called SSR (Screen Space Reflection).

[0048] For example, as shown in FIG. 4, on the ground object 30, a reflection 52 of the tree object 32 and a reflection 54 of the cylindrical object 34 are displayed. Also, on the ground object 30, a reflection 61 of the character object 41 and a reflection 63 of the character object 43 are displayed. On the other hand, a reflection of the mountain object 36 is not displayed on the ground object 30.

[0049] In this embodiment, the tree object 32 and the cylindrical object 34 are set in advance as specific objects, and are set to be reflected on the ground object even when they are a certain distance away from the virtual camera. On the other hand, the mountain object 36 is not set as a specific object.

[0050] The specific object is set in units of meshes (aggregates of polygons). For example, each of the plurality of meshes constituting the tree object 32 may be set as a specific object. Also, any one of the plurality of meshes constituting the tree object 32 may be set as a specific object. For example, when the tree object 32 is composed of a first mesh constituting the trunk portion and a second mesh constituting the leafy portion, only the first mesh of the first mesh and the second mesh may be set as a specific object. That is, the "specific object" does not necessarily have to be an object that appears as one in appearance, and may be a mesh that constitutes a part of an object that appears as one in appearance. For example, a terrain object in which a ground parallel to the xz plane and a wall surface perpendicular to the xz plane are continuously formed may appear to form one terrain having a step in appearance, but the ground and the wall surface may be composed of different meshes. In such a terrain object, the wall surface may be set as a specific object, and the ground does not have to be set as a specific object.

[0051] In this embodiment, a tree object 32 and a column object 34 set as specific objects are reflected on the ground object. Hereinafter, image processing related to the reflection of these objects will be described.

[0052] FIG. 5 is a diagram showing an overview of image processing related to the reflection of objects other than the character object.

[0053] As shown in FIG. 5, the processor 81 first performs a first depth test (step S1). Here, for all objects (30, 32, 34, 36) other than the character object, a depth test is performed on a pixel-by-pixel basis, and the result of the depth test is stored in the first depth buffer.

[0054] FIG. 6 is a diagram showing an example of the first depth buffer. As shown in FIG. 6, the first depth buffer stores the depth value for each pixel for all objects (30, 32, 34, 36) other than the character object. Note that in FIG. 6, the images of the respective objects are displayed, but this conceptually represents the depth values of the respective pixels stored in the first depth buffer. The position of each pixel is represented by the coordinate values of the screen coordinate system. For example, the Sx axis of the screen coordinate system is the axis in the right direction in the image (screen), and the Sy axis is the axis in the upward direction in the image. The origin of the screen coordinate system may be set, for example, at the center of the image or at the lower left of the image.

[0055] Returning to FIG. 5, after step S1, the processor 81 performs rendering to the frame buffer based on the result of the first depth test (step S2). In step S2, an image is drawn on the frame buffer based on the depth stored in the first depth buffer. In the present embodiment, as a rendering method, deferred rendering (also referred to as deferred shading) is used. In step S2, the normal buffer storing the normal information for each pixel is updated, and an image is drawn on the frame buffer. Note that, as a rendering method, forward rendering may be used instead of deferred rendering.

[0056] FIG. 7 is a diagram showing an example of an image drawn on the frame buffer after the rendering by step S2. As shown in FIG. 7, in step S2, images of all objects (30, 32, 34, 36) other than the character object are drawn on the frame buffer. For each pixel, color information is stored. The color information includes, for example, RGB values representing three colors and an alpha value representing transparency (or opacity). Note that although the shadow is omitted in FIG. 7, in step S2, an image with a shadow is generated.

[0057] Returning to FIG. 5, after step S2, the processor 81 further performs a second depth test on a specific object among the plurality of objects arranged in the virtual space (step S3). Specifically, among the plurality of objects (30, 32, 34, 36), a second depth test is further performed on the tree object 32 and the cylinder object 34 set as the specific objects, and the result of the second depth test is stored in the second depth buffer. As a result, in the second depth buffer, the depth value for each pixel of only the tree object 32 and the cylinder object 34 among the plurality of objects (30, 32, 34, 36) included in the imaging range of the virtual camera is stored.

[0058] FIG. 8 is a diagram showing an example of the second depth buffer. As shown in FIG. 8, the second depth buffer stores depth values for each pixel of the tree object 32 and the cylinder object 34. On the other hand, the depth value of the mountain object 36 is not stored in the second depth buffer. Note that FIG. 8 conceptually represents the depth values of the respective pixels stored in the second depth buffer.

[0059] Returning to FIG. 5, after step S3, the processor 81 performs ray tracing using the first depth buffer or the second depth buffer (step S4). Here, the processing is performed for each pixel (hereinafter referred to as the "target pixel"). Specifically, the processor 81 sets the target pixel, calculates the reflection position of the ray when a ray (virtual light ray) is emitted in the direction from the position of the virtual camera toward the target pixel, and performs a collision determination as to whether the ray reflected from the reflection position collides with an object.

[0060] Next, the processor 81 calculates the color of the pixel in the frame buffer corresponding to the collision position of the ray as the color that is reflected onto the target pixel (step S5). The processing of step S5 is also performed for each pixel.

[0061] Hereinafter, the processing of steps S4 and S5 when each of the pixels PIXa to PIXc shown in FIG. 7 is set as the target pixel will be specifically described.

[0062] For example, the processor 81 sets the pixel PIXa shown in FIG. 7 as the target pixel. The pixel PIXa is a pixel on the virtual camera side of the cylinder object 34. The processor 81 calculates the position of the target pixel PIXa in the virtual space as the ray reflection position RPa based on the position of the target pixel PIXa in the screen coordinate system and the depth value of the position stored in the first depth buffer.

[0063] FIG. 9 is a diagram showing how ray tracing is performed, and is a diagram showing how a ray reflected from the reflection position RPa collides with the cylindrical object 34. The right direction in FIG. 9 is the depth direction of the screen when the game image generated based on the virtual camera VC is displayed on the screen.

[0064] As shown in FIG. 9, the processor 81 calculates the direction from the position of the virtual camera VC to the reflection position RPa as the incident direction, and the direction of the ray reflected from the reflection position RPa as the ray tracing direction RTDa. Specifically, the processor 81 calculates the reflection direction of the ray based on the incident direction and the normal line of the reflection position RPa. The incident angle of the incident direction is equal to the reflection angle of the reflection direction. Then, the processor 81 calculates the reflection direction or the direction obtained by correcting the reflection direction as the ray tracing direction RTDa. Note that the correction of the reflection direction will be described later.

[0065] The processor 81 advances the ray in the ray tracing direction RTDa from the reflection position RPa, and performs a collision determination as to whether the ray collides with an object. The processor 81 performs the collision determination using the first depth buffer when the trace distance of the ray is less than La. Here, the trace distance is the distance along the ray tracing direction from the reflection position.

[0066] FIG. 10 is a diagram showing a state in which a collision determination of a ray reflected from a reflection position RPa is performed using a first depth buffer. First, the processor 81 calculates a position in the virtual space that has advanced by a maximum distance Lb in the ray tracing direction RTDa from the reflection position RPa in the virtual space, and calculates the ray tracing end position in the screen coordinate system by converting the calculated position into the screen coordinate system. The processor 81 extends a ray from the reflection position RPa (the pixel of interest RPa) as the ray tracing start position to the ray tracing end position in the screen coordinate system, and determines whether the tip of the ray collides with an object. Specifically, the processor 81 advances the ray by a predetermined distance, and determines whether the ray collides with an object based on the depth of the tip of the ray and the depths of the respective positions stored in the first depth buffer. In the examples shown in FIGS. 9 and 10, it is determined that the ray reflected at the reflection position RPa collides at the collision position CPa.

[0067] When it is determined that the ray collides with an object at the collision position CPa, in step S5, the color of the pixel of the frame buffer corresponding to the collision position CPa is calculated as the color that appears in the pixel of interest RPa.

[0068] Next, a case where the pixel PIXb shown in FIG. 7 is set as the pixel of interest and a case where the pixel PIXc is set as the pixel of interest will be described. The pixel PIXb is a pixel on the virtual camera side of the tree object 32. The pixel PIXc is a pixel on the virtual camera side of the mountain object 36.

[0069] FIG. 11 is a diagram showing a state in which ray tracing is performed, and shows a state in which a ray reflected from a reflection position RPb collides with the tree object 32. FIG. 12 is a diagram showing a state in which ray tracing is performed, and shows a state in which a ray reflected from a reflection position RPc does not collide with an object. FIG. 13 is a diagram showing a state in which a collision determination of a ray is performed using a second depth buffer.

[0070] As shown in FIG. 11, when the processor 81 sets the pixel PIXb as the pixel of interest, in the same manner as above, the reflection position RPb corresponding to the pixel of interest PIXb is calculated, and the direction of the ray reflected from the reflection position RPb is calculated as the ray tracing direction RTDb. The processor 81 advances the ray in the ray tracing direction RTDb and performs a collision determination as to whether the ray collides with an object. When the trace distance is less than La, the processor 81 performs the collision determination of the ray using the first depth buffer. However, when the trace distance is less than La, the ray does not collide with an object. Therefore, the processor 81 further advances the ray and performs the collision determination of the ray using the second depth buffer (FIG. 13). As shown in FIGS. 11 and 13, when the trace distance is equal to or greater than La and equal to or less than the maximum distance Lb, it is determined that the ray collides with the tree object 32 at the collision position CPb.

[0071] On the other hand, as shown in FIG. 12, when the processor 81 sets the pixel PIXc as the pixel of interest, in the same manner as above, the reflection position RPc corresponding to the pixel of interest PIXc is calculated, the direction of the ray reflected from the reflection position RPc is calculated as the ray tracing direction RTDc, and the ray is advanced in the ray tracing direction RTDb. When the trace distance is less than La, the processor 81 performs the collision determination of the ray using the first depth buffer. When the trace distance is equal to or greater than La and equal to or less than the maximum distance Lb, the processor 81 performs the collision determination of the ray using the second depth buffer. Here, as shown in FIG. 13, the depth of the mountain object 36 is not stored in the second depth buffer. Therefore, as shown in FIGS. 12 and 13, the processor 81 determines that the ray does not collide with an object even if the ray is advanced to the maximum distance Lb.

[0072] In addition, when the collision position of a ray is calculated based on the first depth buffer and when the collision position of a ray is calculated based on the second depth buffer, the parameters for adding the color of the pixel at the collision position to the color of the pixel of interest may be made different. For example, when the collision position is calculated based on the second depth buffer, the parameters may be changed so that the reflected color becomes darker than when the collision position is calculated based on the first depth buffer.

[0073] By performing the processes of steps S4 and S5 described above for each pixel of interest, the color that is reflected onto each pixel of interest is determined. Then, by adding the determined reflected color to the color of each pixel of interest, the reflections of each object are drawn on the frame buffer.

[0074] FIG. 14 is a diagram showing an example of an image drawn on the frame buffer when the color that is reflected onto each pixel is added.

[0075] As shown in FIG. 14, for pixel RPa, the color of the pixel at the collision position CPa calculated as described above is added. By performing the same process for each pixel included in the area in front of the cylindrical object 34 (the area on the virtual camera side), the reflection 54 of the cylindrical object 34 is drawn on the frame buffer. Also, for pixel RPb, the color of the pixel at the collision position CPb is added. By performing the same process for each pixel included in the area in front of the wooden object 32, the reflection 52 of the wooden object 32 is drawn on the frame buffer. On the other hand, for pixel RPc, the collision position is not calculated. The same applies when the process is performed for each pixel included in the area in front of the mountain object 36. Therefore, the reflection of the mountain object 36 is not drawn on the frame buffer.

[0076] As described above, in this embodiment, the results of performing the first depth test on a plurality of objects are stored in the first depth buffer, and the results of performing the second depth test on a specific object among the plurality of objects are stored in the second depth buffer. When the trace distance is less than La, a ray collision determination is made based on the first depth buffer. When the trace distance is greater than or equal to La and less than or equal to the maximum distance Lb, a ray collision determination is made based on the second depth buffer. Thereby, using SSR, for a specific object among a plurality of objects, even when the object is far from the virtual camera, the reflection can be displayed, and for objects other than the specific object, the reflection can be prevented from being displayed. By designating the object to be reflected as the specific object, a desired object can be reflected, for example, on the ground. Conversely, by not designating the object that is not desired to be reflected as the specific object, the desired object can be prevented from being reflected.

[0077] In addition, the parameters for reflecting the color of the pixel at the collision position to the target pixel are made different between the case where the collision position of the ray is calculated based on the first depth buffer and the case where the collision position of the ray is calculated based on the second depth buffer. Thereby, for example, a specific object can be reflected more darkly and clearly.

[0078] (Correction of the reflection direction) Next, the correction of the reflection direction will be described. As described above, the direction from the virtual camera VC toward the reflection position is calculated as the incident direction, and based on the incident direction and the normal direction of the reflection position, the reflection direction is calculated. In this embodiment, when the correction condition is satisfied, the corrected direction of the reflection direction is set as the ray tracing direction. When the correction condition is not satisfied, the reflection direction is set as the ray tracing direction. Hereinafter, the correction condition for correcting the reflection direction and the method of correction will be specifically described.

[0079] FIG. 15 is a diagram showing an example of the ray tracing direction with the reflection direction corrected. As shown in FIG. 15, for example, when ray tracing is performed for a target pixel (reflection position RP1) close to the left end of the screen, the reflection direction RD1 calculated based on the incident direction and the normal direction is corrected, and the corrected direction is set as the ray tracing direction RTD1. Also, when ray tracing is performed for a target pixel (reflection position RP2) on the right side of the screen relative to the reflection position RP1, the reflection direction RD2 is corrected, and the corrected direction is set as the ray tracing direction RTD2.

[0080] The correction condition is satisfied when both the condition regarding the reflection position and the condition regarding the reflection direction are met. The condition regarding the reflection position is that the reflection position is within a predetermined range from the edge of the screen. Also, the condition regarding the reflection direction is that the reflection direction faces outside the screen. When either the condition regarding the reflection position or the condition regarding the reflection direction is not met, the correction condition is not satisfied and the reflection direction is not corrected. For example, when the reflection position is within the range from the left end of the screen to 1 / 4 of the length of the screen in the left - right direction and the reflection direction faces the left direction of the screen, the correction condition is satisfied. Also, when the reflection position is within the range from the right end of the screen to 1 / 4 and the reflection direction faces the right direction of the screen, the correction condition is satisfied.

[0081] The degree of correction is greater for the reflection position RP1 closer to the left end of the screen than for the reflection position RP2. Specifically, the degree of correction CR is determined based on the position of the reflection position in the Sx - axis direction of the screen coordinate system. The closer the position of the reflection position in the Sx - axis direction is to the left end or the right end of the screen, the greater the degree of correction CR. Also, the greater the angle between the reflection direction and the Sy - axis direction in the screen coordinate system, the smaller the degree of correction CR.

[0082] FIG. 16 is a diagram for explaining details of correction with respect to the reflection direction. As shown in FIG. 16, for example, when the angle between the reflection direction RD and the upward direction (Sy-axis direction) of the screen is 45 degrees or less, the degree of correction CR is set to a value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP. CR(Sx) is set, for example, in the range of 0 to 1, and becomes larger as the reflection position gets closer to the left or right end of the screen. When the angle between the reflection direction RD and the upward direction of the screen exceeds 45 degrees, the value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP becomes smaller according to the exceeded angle. When the angle between the reflection direction RD and the upward direction of the screen exceeds a threshold value (for example, 60 degrees to 65 degrees), the degree of correction CR is set to "0".

[0083] According to the degree of correction CR, correction toward the inside of the screen is applied in the reflection direction. Specifically, the position obtained by converting the position advanced by the maximum distance Lb from the reflection position RP in the virtual space in the reflection direction RD into the screen coordinate system is calculated as the ray tracing end position EP. The Sx coordinate value of this ray tracing end position EP is corrected to move in the inward direction of the screen according to the degree of correction CR. For example, by linear interpolation, the Sx coordinate value of the ray tracing end position EP is corrected to approach the Sx coordinate value of the reflection position RP. For example, when the reflection position RP is located within a predetermined range from the left end of the screen, the ray tracing end position EP is moved in the right direction (positive Sx-axis direction) according to the degree of correction CR. The Sx coordinate value of the ray tracing end position EP does not move beyond the Sx coordinate value of the reflection position RP. Also, the Sy coordinate value of the ray tracing end position EP does not change before and after the correction. Then, the direction from the reflection position EP to the moved ray tracing end position EP' is calculated as the ray tracing direction RTD.

[0084] Thus, in the area near the edge of the screen, the direction with the corrected reflection direction is calculated as the ray tracing direction. As a result, at the edge of the screen, it is possible to generate the reflection by SSR. For example, as shown in FIG. 15, for the pixel at the position RP1 at the left end of the screen, when ray tracing is performed in the reflection direction RD1, the collision position of the ray may be outside the screen, and the color of the pixel at the collision position cannot be added to the pixel at the position RP1. Therefore, if no correction is made to the reflection direction, the edge part of the screen may have a different color from other parts, resulting in an image with a sense of incongruity.

[0085] In this embodiment, since the reflection direction based on the incident direction and the normal direction is corrected and the corrected direction is set as the ray tracing direction, it is possible to easily keep the collision position of the ray within the screen, and it is possible to generate the reflection based on the rendered image. The Sx coordinate value of the ray tracing end position EP is moved so as to approach the Sx coordinate value of the reflection position RP. Therefore, even if the maximum correction is performed on the reflection direction facing the outside of the screen, it does not become the direction facing the inside of the screen, but becomes the upward direction of the screen. Therefore, for example, the color of the pixel on the right side of the pixel at the left end position of the screen is not reflected. As a result, it is possible to prevent the object from being unnaturally distorted and reflected on the ground.

[0086] Also, in this embodiment, the degree of correction increases as the reflection position approaches the edge of the screen. Therefore, even in the vicinity of the edge of the screen, it is possible to generate the reflection based on the pixels within the screen.

[0087] Also, for example, when the reflection direction is facing the inside of the screen, if the reflection direction is corrected, the reflection may be far from the actual reflection. In this embodiment, when the reflection direction is not facing the outside of the screen, the reflection direction is not corrected, so it is possible to prevent the generation of a reflection far from the actual reflection.

[0088] Also, when the angle between the reflection direction and the upward direction of the screen is relatively large (for example, exceeding 45 degrees), the degree of correction CR becomes small. When the angle between the reflection direction and the upward direction of the screen exceeds the threshold value, the degree of correction becomes 0. Thereby, for example, when the ground is inclined, it is possible to prevent the degree of correction from being too large and generating unnatural reflections. Also, it is possible to suppress the correction for reflections on a wall surface where the reflection direction is horizontal.

[0089] Also, in this embodiment, the correction of the reflection direction is performed only for the horizontal component in the screen coordinate system. Therefore, for example, it is suitable for a game in which a virtual space extends in the left - right direction of the screen and is scrolled horizontally.

[0090] (Generation of reflection of character object) Next, a method for generating the reflections of the character objects 41 and 43 will be described. FIG. 17 is a diagram showing an example of a game image when the character object 41 does not exist in front of the cylindrical object 34. FIG. 18 is a diagram showing an example of a game image when the character object 41 moves and the character object 41 exists in front of the cylindrical object 34.

[0091] As shown in FIG. 17, when the character object 41 does not exist in front of the cylindrical object 34, for example, a ray trace is performed from the front - side reflection position RPd, the collision position CPd is calculated, and the color of the collision position CPd is added to the color of the reflection position RPd. By performing the same processing for each pixel, the reflection 54 of the cylindrical object 34 is displayed.

[0092] On the one hand, in FIG. 18, a character object 41 exists in front of the cylindrical object 34, and the head of the character object 41 exists at the same position CPd as in FIG. 17. In this case, the color of the pixel at the collision position CPd of the cylindrical object 34 shown in FIG. 17 cannot be added to the pixel at the reflection position RPd, and a part of the cylindrical object 34 is not reflected in the reflection position RPd. Thus, for example, when the character object 41 is located in front of the cylindrical object 34, there may be a case where a natural reflection according to the positional relationship of the original objects cannot be expressed.

[0093] In this embodiment, in order to display the reflection of the character object and the object behind it, image processing is performed by the method shown in FIG. 19. FIG. 19 is a diagram showing an outline of image processing for generating the reflection of the character object and the object behind it.

[0094] As shown in FIG. 19, first, the processor 81 performs rendering of objects other than the character object (step S10). Here, a process of drawing a plurality of objects 30, 32, 34, 36 arranged in the virtual space on the frame buffer is performed. The process of step S10 is the process of steps S1 to S2 in FIG. 5.

[0095] Next, the processor 81 renders the character object in a separate buffer and creates a character plane with the rendered image set as a texture (step S11).

[0096] FIG. 20 is a diagram showing an example of a character plane created in step S11. As shown in FIG. 20, a character object 41 is rendered in a separate buffer, and an image obtained by rendering the character object 41 is set as a texture for a two-dimensional planar object, thereby creating a character plane 65. Similarly, a character plane 66 is created by setting an image obtained by rendering the character object 43 as a texture. In the present embodiment, an image obtained by rendering each character object is generated every frame. Note that an image obtained by rendering each character object may be prepared in advance. For example, images corresponding to respective actions of the character object 41 may be stored in a storage medium in advance, and a character plane 65 may be created by setting the pre-stored images as textures.

[0097] Next, the processor 81 calculates the reflection color of an object other than the character object (step S12). Here, the reflection colors of a plurality of objects 30, 32, 34, and 36 arranged in the virtual space are calculated. Specifically, the processor 81 performs the processes of steps S3 to S5 in FIG. 5 described above. At this point, since the character object is not drawn in the frame buffer, even if the character object exists in front of the object as viewed from the virtual camera VC in the virtual space, the reflection color of the object can be calculated. The calculated reflection color of an object other than the character object (hereinafter referred to as "first reflection color") is stored in the reflection buffer. For example, in the reflection buffer, an RGB value and an alpha value are stored as color information indicating the first reflection color.

[0098] Next, the processor 81 calculates the reflection color of the character (step S13). Here, the processor 81 calculates the reflection color of the character object when the character plane created in step S11 is arranged in the virtual space.

[0099] FIG. 21 is a diagram for explaining the calculation of the color of the reflection of the character in step S13. As shown in FIG. 21, the processor 81 calculates the intersection position of the character plane 65 and the ray when the character plane 65 is arranged in the virtual space according to the position and orientation of the character object 41 in the virtual space. Specifically, the processor 81 calculates the depth when the character plane 65 is arranged in the virtual space, and calculates the intersection position CRPe of the ray from the reflection position RPe in the ray tracing direction RTDe and the character plane 65. The color of the pixel at the intersection position CRPe is calculated as the color of the reflection of the character (hereinafter referred to as the "second reflection color"). Here, among the character plane 65, the portion other than the image of the character object 41 is set to be transparent and does not intersect with the ray. For example, the ray from the reflection position RPf in the ray tracing direction RTDf does not intersect with the image of the character object 41 in the character plane 65. In this case, the intersection position is not calculated. The calculated second reflection color is stored in the reflection buffer. For example, in the reflection buffer, the RGB value and the alpha value are stored as the color information indicating the second reflection color.

[0100] Returning to FIG. 19, the processor 81 then calculates the color reflected on the pixel of interest based on the results of step S12 and the results of step S13 (step S14). For example, by adding the second reflection color to the first reflection color, the color reflected on the pixel of interest is calculated. Here, "adding the second color to the first color" may mean mixing the two colors. For example, "adding the second color to the first color" may mean performing alpha blending with the first color as the background color and the second color as the foreground color. Also, "adding the second color to the first color" may mean performing alpha blending with the second color as the background color and the first color as the foreground color. Also, "adding the second color to the first color" may mean mixing the first color and the second color based on their respective alpha values. Also, "adding the second color to the first color" may mean that the second color is prioritized over the first color and the first color is overwritten with the second color.

[0101] Next, the processor 81 renders the result calculated in step S14 to the frame buffer (step S15). Here, the processor 81 adds the reflected color calculated in step S14 to the color of the pixel of interest stored in the frame buffer. For example, the processor 81 performs alpha blending with the color of the pixel of interest as the background color and the reflected color calculated in step S14 as the foreground color. As a result, the reflections of the character object and objects other than the character object are drawn in the frame buffer.

[0102] After the process of step S15 is performed, the processor 81 renders the character objects 41 and 43 to the frame buffer (step S16). As a result, a game image as shown in FIG. 22 is generated and displayed on the display device.

[0103] FIG. 22 is a diagram showing an example of a game image displayed when the image processing of FIG. 19 is performed. As shown in FIG. 22, the tree object 32 and its reflection 52 are displayed. Also, the character object 43 and its reflection 63 are displayed. Further, the cylindrical object 34 and its reflection 54 are displayed. There is a character object 41 in front of the cylindrical object 34, and the character object 41 and its reflection 61 are displayed. A part of the reflection 54 of the cylindrical object 34 and the reflection 61 of the character object 41 overlap, but these are all in shapes corresponding to the actual shapes and positional relationships, and a natural reflection can be expressed.

[0104] In this embodiment, the character object is a flat object. As described above, when generating the reflection of the character object based on the character plane in which the image of the character object is set as a texture, the reflection of the character object may look flat. If the character object itself is a three-dimensional object having a certain thickness, the reflection may look flat and the image may be uncomfortable. However, in this embodiment, since the character object itself is a flat object, an image without discomfort can be obtained.

[0105] (Details of Image Processing) Next, the details of the above-described image processing will be described. First, the data stored in the memory of the main body device 2 (the memory in the processor 81, the DRAM 85, the flash memory 84, or an external storage medium, etc.) will be described. FIG. 23 is a diagram showing an example of the data stored in the memory of the main body device 2.

[0106] As shown in FIG. 23, the memory of the main body device 2 stores a program, operation data, object data, character data, and character plane data. Further, the memory of the main body device 2 stores a first depth buffer, a second depth buffer, a normal buffer, a reflection buffer, and a frame buffer.

[0107] The program is a program for executing the main processing described later and includes an image processing program for performing the above-described image processing related to reflection. The program is stored in advance in an external storage medium or the flash memory 84 attached to the slot 23 and is read into the DRAM 85 when the game is executed. Note that the program may be acquired from another device via a network (for example, the Internet).

[0108] The operation data is data regarding operations acquired from the left controller 3 and the right controller 4. For example, the operation data is transmitted from the left controller 3 and the right controller 4 to the main body device 2 at a predetermined time interval (for example, at intervals of 1 / 200 second) and stored in the memory.

[0109] The object data is data regarding each object (30, 32, 34, 36) other than the character objects arranged in the virtual space. The object data includes data regarding the position and orientation of each object in the virtual space, data regarding the shape, and data regarding the texture. Further, the object data includes information regarding whether each object is set as a specific object. For example, the tree object 32 and the column object 34 are set as specific objects. On the other hand, the mountain object 36 is not set as a specific object.

[0110] The character data is data regarding the character objects 41, 43. The character data includes data regarding the position and orientation of each character in the virtual space, data regarding the shape, and data regarding the texture. In the present embodiment, each character object is formed as a 3D model, but is formed as a planar object that is thinner in the thickness direction than in the vertical and horizontal directions.

[0111] The character plane data is a two-dimensional object and is data regarding a planar object in which an image obtained by rendering a character object is set as a texture. The character plane data includes data regarding the character plane 65 in which the image of the character object 41 is set as a texture and data regarding the character plane 66 in which the image of the character object 43 is set as a texture.

[0112] The first depth buffer is a buffer that stores the depth value for each pixel generated and updated based on the result of the first depth test. In the first depth buffer, the depth values of all objects other than the character object included in the imaging range of the virtual camera VC are stored.

[0113] The second depth buffer is a buffer that stores the depth value for each pixel generated and updated based on the result of the second depth test. In the second depth buffer, the depth values of specific objects included in the imaging range of the virtual camera VC are stored.

[0114] The normal buffer is a buffer that stores the normal information of each pixel.

[0115] The reflection buffer is a buffer for storing the color of the first reflection (the reflection color of objects other than the character object) and the color of the second reflection (the reflection color of the character object). In the reflection buffer, in the ray tracing process described later, when the ray collides with an object other than the character object, the color of the first reflection is stored, and when the ray intersects with the character plane, the color of the second reflection is stored. Although the details will be described later, there may be a plurality of reflection colors stored in the reflection buffer.

[0116] The frame buffer is a buffer for storing the game image to be displayed. The image stored in the frame buffer is output to the display device at a predetermined timing and is displayed on the display device.

[0117] (Details of the game processing in the main device 2) Next, with reference to FIGS. 24 to 27, the details of the processing performed in the main device 2 will be described. FIG. 24 is a flowchart showing an example of the main processing executed by the processor 81 of the main device 2. The processing shown in FIGS. 24 to 27 is performed by the CPU or GPU of the processor 81.

[0118] As shown in FIG. 24, the processor 81 first executes initial processing (step S100). Specifically, the processor 81 sets up a three-dimensional virtual space and arranges in the virtual space each object (30, 32, 34, 36), each character object (41, 43), the virtual camera VC, the light source, and various other objects used in the game. After the processor 81 executes the initial processing, it repeatedly executes the processing of the following steps S101 to S108 at a predetermined frame time interval (for example, at intervals of 1 / 60 second).

[0119] In step S101, the processor 81 acquires operation data from the controller.

[0120] Next, the processor 81 performs game processing based on the acquired operation data (step S102). For example, based on the operation data, the processor 81 moves the character object 41 within the virtual space or causes the character object 41 to perform a predetermined action (such as a jump action, an attack action, etc.). Also, the processor 81 moves the character object 43 within the virtual space or causes the character object 43 to perform a predetermined action according to a predetermined algorithm. Further, the processor 81 controls enemy objects other than the character objects within the virtual space or moves obstacle objects that obstruct the character objects within the virtual space.

[0121] Next, the processor 81 performs rendering processing on objects other than character objects (step S103). Specifically, the processor 81 performs a first depth test on each object other than the character objects 41 and 43 included in the imaging range of the virtual camera VC, and updates the first depth buffer. Further, the processor 81 draws each object on the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering is used as the rendering method. In step S103, the normal buffer is also updated. Note that forward rendering may be used as the rendering method.

[0122] Next, the processor 81 renders the character object in a separate buffer to create a character plane (step S104). Specifically, the processor 81 renders the character object 41 in a separate buffer, and creates a character plane 65 by setting the image of the character object 41 rendered in the separate buffer as a texture in a plane object. Similarly, the processor 81 creates a character plane 66 by setting the image of the character object 43 as a texture in a plane object.

[0123] After step S104, the processor 81 performs SSR processing (step S105). The SSR processing is a process for drawing the reflections of all objects including the character object on the frame buffer. The details of the SSR processing will be described below.

[0124] (SSR Processing) FIG. 25 is a flowchart showing an example of the SSR processing in step S105.

[0125] As shown in FIG. 25, the processor 81 performs a second depth test on a specific object and updates the second depth buffer (step S121). For example, the processor 81 performs a depth test on the tree object 32 and the cylinder object 34 that are set in advance as specific objects, and updates the second depth buffer. Note that the setting of the specific object is performed in mesh units.

[0126] Next, the processor 81 performs ray tracing processing (step S122). The ray tracing processing is performed on each target pixel. Here, the color of the reflection onto each target pixel is calculated. Note that the ray tracing processing may be performed for all pixels, or may be performed only for pixels within a specific range. For example, the load may be reduced by excluding parts where the model is not drawn, parts where SSR is set to invalid in model or mesh units, parts where the normal direction is outside a specific range, and the like. Details of the ray tracing processing will be described below.

[0127] (Ray Tracing Processing) FIG. 26 is a flowchart showing an example of the ray tracing processing in step S122.

[0128] As shown in FIG. 26, the processor 81 calculates the position in the virtual space of the target pixel as the reflection position and calculates the reflection direction (step S131). Specifically, the processor 81 calculates the position in the virtual space of the target pixel as the reflection position based on the position of the target pixel in the screen coordinate system and the depth of the target pixel stored in the first depth buffer. Further, the processor 81 sets the direction from the position of the virtual camera VC to the reflection position as the incident direction, and calculates the reflection direction based on the incident direction and the normal direction of the reflection position.

[0129] Next, the processor 81 sets the calculated reflection direction or the corrected direction of the reflection direction as the ray tracing direction (step S132). Here, when the correction condition is satisfied, the corrected direction of the reflection direction is set as the ray tracing direction, and when the correction condition is not satisfied, the reflection direction is set as the ray tracing direction. Also, the ray tracing end position is set. The correction condition and the correction method are as described above.

[0130] After step S132, the processor 81 sets the first depth buffer as the depth buffer for determination and starts ray tracing from the pixel of interest (step S133). The ray tracing is performed in the screen coordinate system. Next, the processor 81 advances the ray by a predetermined distance (step S134).

[0131] Subsequently, the processor 81 determines whether the trace distance has exceeded the maximum distance Lb (step S135). Specifically, the processor 81 determines whether the tip of the ray in the screen coordinate system has reached the ray tracing end position.

[0132] When it is determined that the trace distance has exceeded Lb (step S135: YES), the processor 81 then performs the process of step S141.

[0133] On the other hand, when the trace distance is less than or equal to Lb (step S135: NO), the processor 81 determines whether the ray has collided with an object (step S136). Specifically, the processor 81 uses the depth buffer for determination to determine whether the tip of the ray has collided with an object.

[0134] When it is determined that the ray has not collided with an object (step S136: NO), the processor 81 determines whether the trace distance is greater than or equal to La (step S137).

[0135] When the trace distance is greater than or equal to La (step S137: YES), the processor 81 changes the determination depth buffer to the second depth buffer (step S138) and executes the process of step S134 again. On the other hand, when the trace distance is less than La (step S137: NO), the processor 81 executes the process of step S134 again while maintaining the determination depth buffer.

[0136] On the other hand, when it is determined that the ray has collided with an object (step S136: YES), the processor 81 calculates the color of the first specular reflection from the color at the collision position of the ray (step S139). For example, the processor 81 may calculate the color of the pixel at the collision position stored in the frame buffer as the color of the first specular reflection, or may calculate the color obtained by applying a predetermined process to the color of the pixel at the collision position as the color of the first specular reflection. The color of the first specular reflection is different between the case where the collision is determined based on the first depth buffer and the case where the collision is determined based on the second depth buffer. For example, when the first depth buffer is set as the depth buffer for determination, depending on the trace distance, the color of the pixel at the collision position may be changed to be lighter (the alpha value is made smaller), and the changed color may be calculated as the color of the first specular reflection. Further, when the second depth buffer is set as the depth buffer for determination, regardless of the trace distance, the color of the pixel at the collision position may be calculated as the color of the first specular reflection. Further, when the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position is changed to be lighter depending on the trace distance, but is changed to be darker than when the first depth buffer is set as the depth buffer for determination, and the changed color may be calculated as the color of the first specular reflection. Then, the processor 81 stores the calculated color of the first specular reflection in the specular buffer. Here, the color of the first specular reflection stored in the specular buffer when it is determined that the ray has collided based on the first depth buffer is denoted as "the color of the first specular reflection (first depth buffer)". Also, the color of the first specular reflection stored in the specular buffer when it is determined that the ray has collided based on the second depth buffer is denoted as "the color of the first specular reflection (second depth buffer)".

[0137] Next, the processor 81 determines whether the depth buffer for determination is the first depth buffer and whether the color of the first reflection calculated in step S139 is translucent (step S140). For example, when the trace distance is less than or equal to La and the color of the first reflection is translucent, further ray tracing is performed deeper. Being translucent means that the alpha value of the reflection is less than 1.0, and the alpha value is calculated independently based on the distance La, the collision position, and other various parameters.

[0138] If the determination in step S140 is NO, the processor 81 proceeds to step S141. If the determination in step S140 is YES, the processor 81 proceeds to step S138.

[0139] In step S141, the processor 81 calculates the color of the reflection of the character object (the color of the second reflection). The details of the process in step S141 will be described below.

[0140] FIG. 27 is a flowchart showing an example of the process for calculating the color of the reflection of the character in step S141.

[0141] As shown in FIG. 27, the processor 81 calculates the intersection position of the ray and the character plane (step S151). Specifically, the processor 81 calculates the depth when the character plane is arranged according to the position and orientation in the virtual space of the character object, and determines whether the ray used in the ray tracing intersects the character plane. When the ray intersects the character plane, the processor 81 calculates the intersection position.

[0142] Next, the processor 81 stores the color of the intersection position as the color of the second reflection in the reflection buffer (step S152). Specifically, the processor 81 stores, as the color of the second reflection, a color based on the color of the pixel of the texture image of the character object corresponding to the intersection position. Note that the color of the second reflection may be the color of the pixel of the texture image of the character object, or may be a color calculated by applying a predetermined process to the color of the pixel. When the processor 81 performs the process of step S152, the processor ends the process shown in FIG. 27 and returns the process to FIG. 26. When the ray intersects a plurality of character planes, a plurality of colors of the second reflection are stored in the reflection buffer. In this case, the plurality of colors of the second reflection may be stored in the reflection buffer in the order of proximity to the virtual camera of the intersection positions.

[0143] By performing the processes of step S139 and step S152, a plurality of colors of reflection are stored in the reflection buffer from 0. For example, in the reflection buffer, the colors of reflection corresponding to the collision position or the intersection position of the ray may be stored in the order of proximity to the virtual camera. For example, when it is determined that the ray has collided at a first position where the trace distance is less than La and it is also determined that the ray has collided at a second position where the trace distance is equal to or greater than La and less than Lb, and when the ray intersects the character plane between the first position and the second position, the reflection buffer stores, in the order of proximity to the virtual camera, the first color of reflection (first depth buffer), the second color of reflection, and the first color of reflection (second depth buffer) (see FIG. 23). Note that when the ray does not collide with any object and does not intersect the character plane, no color of reflection is stored in the reflection buffer.

[0144] Returning to FIG. 26, after the process of step S141, the processor 81 calculates the color to be reflected onto the target pixel based on the reflected color stored in the reflection buffer (step S142). When a plurality of reflected colors are stored in the reflection buffer, the processor 81 may calculate the color to be reflected onto the target pixel by alpha-blending the plurality of reflected colors in the order from the farthest to the virtual camera. For example, as shown in FIG. 23, when the first reflected color (first depth buffer), the second reflected color, and the first reflected color (second depth buffer) are stored in the reflection buffer in the order from the closest to the virtual camera, the processor 81 calculates the color obtained by alpha-blending the second reflected color with the first reflected color (second depth buffer), and further calculates the color to be reflected onto the target pixel by alpha-blending the calculated color with the first reflected color (first depth buffer). Further, for example, when the second reflected color and the first reflected color are stored in the reflection buffer in the order from the closest to the virtual camera (that is, when there is no object in front of the character as seen from the virtual camera and there is an object behind the character), the processor 81 may calculate the color to be reflected onto the target pixel by alpha-blending the first reflected color with the second reflected color. For example, when the second reflected color on the front side is opaque, the second reflected color is set as the color to be reflected onto the target pixel. When the second reflected color on the front side is semi-transparent, the color obtained by blending the first reflected color and the second reflected color is calculated as the color to be reflected onto the target pixel. The processor 81 stores the calculated reflected color in the memory.

[0145] When the process of step S142 is performed, the processor 81 ends the process of FIG. 26 and returns the process to FIG. 25.

[0146] Returning to FIG. 25, after step S122, the processor 81 performs rendering to the frame buffer based on the result of the ray tracing process (step S123). For example, the processor 81 adds the reflected color calculated in step S142 to the color of the pixel of interest stored in the frame buffer. Thereby, the reflected color calculated in step S142 is reflected in the color of the pixel of interest.

[0147] Next, the processor 81 determines whether the processes of steps S122 and S123 have been performed for all pixels (step S124). If the processor 81 determines NO in step S124, it executes the process of step S122 again. By performing the processes of steps S122 and S123 for all pixels, the reflections (52, 54) of objects other than the character objects arranged in the virtual space and the reflections (61, 63) of the character objects are drawn in the frame buffer. If the processor 81 determines YES in step S124, it ends the process of FIG. 25 and returns the process to FIG. 24.

[0148] Returning to FIG. 24, after step S105, the processor 81 further renders the character objects 41 and 43 to the frame buffer (step S106).

[0149] Next, the processor 81 outputs the image stored in the frame buffer to the display device (step S107). Thereby, the game image is displayed.

[0150] Next, the processor 81 determines whether to end the game (step S108). For example, when the player instructs the end of the game, the processor 81 ends the game process shown in FIG. 24. If the processor 81 determines not to end the game (step S108: NO), it executes the process of step S101 again. Thus, the description of the main process shown in FIG. 24 ends.

[0151] Note that the order, content, values used for determination, etc. of the processes in the above flowchart are merely examples, and these may be changed as appropriate.

[0152] As described above, in the present embodiment, when the trace distance is less than La, the ray collision determination is performed based on the first depth buffer, and when the trace distance is greater than or equal to La, the ray collision determination is performed based on the second depth buffer (steps S135 to S140). Thereby, even when the trace distance is long for a specific object, the reflection can be displayed. Also, for objects other than the specific object, the reflection can be prevented from being displayed, and the desired object can be made to reflect or not reflect. Further, the reflection is generated based on the first depth buffer and the second depth buffer, and when based on the first depth buffer, the color of the reflection is made lighter according to the trace distance, so that the specific object can be reflected largely and clearly, and the other objects can be reflected small.

[0153] Also, in the present embodiment, when the pixel of interest (reflection position) is close to the edge of the screen, the direction obtained by correcting the reflection direction based on the incident direction and the normal direction at the reflection position is set as the ray tracing direction (step S132). Thereby, the color of the reflection can be calculated based on the pixels drawn in the frame buffer, and the reflection can be generated even at the edge of the screen.

[0154] Also, in the present embodiment, the objects other than the character object are drawn in the frame buffer first, the reflection of the object is drawn, then the reflection of the character object is drawn, and finally the character object is drawn. Thereby, the reflection of the object behind the character object as seen from the virtual camera can be displayed, and the reflection of the character object can also be displayed.

[0155] In this embodiment, the color of the collision position calculated based on the first depth buffer, the color of the collision position calculated based on the second depth buffer, and the color of the intersection position with the character plane are calculated, and these colors are alpha-blended in order from the back. As a result, even when objects overlap when viewed from the virtual camera, the reflection of each object can be displayed.

[0156] (Modification example) As described above, the image processing of this embodiment has been explained. However, the above embodiment is merely an example, and for example, the following modifications may be added.

[0157] For example, in the above embodiment, deferred rendering is used as the rendering method. However, in other embodiments, forward rendering may be used. When forward rendering is used, in the above SSR process, for example, the normal direction of the reflection position corresponding to the pixel of interest is estimated based on the depths of a plurality of pixels around the pixel of interest, and the reflection direction may be calculated based on the incident direction and the normal direction. Alternatively, the posture of the mesh in the virtual space may be calculated, the normal direction of the reflection position corresponding to the pixel of interest may be calculated, and the reflection direction may be calculated based on the normal direction.

[0158] Also, in the above embodiment, the reflection direction is corrected for the pixels in the region within a predetermined range from the left end or the right end of the screen. However, in other embodiments, the reflection direction may also be corrected for the pixels in the region within a predetermined range from the upper end or the lower end of the screen. In the above embodiment, in the correction of the reflection direction, the position in the Sx-axis direction of the ray tracing end position in the screen coordinate system is made closer to the position in the Sx-axis direction of the reflection position. In other embodiments, the position in the Sy-axis direction of the ray tracing end position may also be moved in the positive or negative direction of the Sy-axis to correct the reflection direction. The reflection direction may be corrected so that the collision position of the ray falls within the screen by moving the ray tracing end position in the horizontal and / or vertical directions.

[0159] In the above embodiment, ray tracing is performed in the screen coordinate system. However, in other embodiments, ray tracing may be performed in the xyz coordinate system of the virtual space. That is, in the xyz coordinate system of the virtual space, a ray may be extended, the collision position of the ray may be calculated, and the color of the pixel at the position corresponding to the collision position may be calculated as the reflected color.

[0160] In the above embodiment, a character plane is created by setting an image of a character object as a texture on a planar object, and the reflected color of the character object is calculated using the character plane. In other embodiments, an image of a character object may be set as a texture on a three-dimensional object, and the reflected color of the character object may be calculated using the three-dimensional object. Also, in the above embodiment, the character object is an object with a flat shape, but in other embodiments, the character object may be a three-dimensional object having a thickness.

[0161] In the above embodiment, the case where an object is reflected on the ground with the ground as the reflecting surface is described. However, a surface other than the ground (for example, a wall surface or a ceiling surface) may be used as the reflecting surface, and the object may be reflected by the above-described processing.

[0162] In the above embodiment, a game image is generated. However, the above-described image processing may be used not only for game images but also when generating any image.

[0163] Also, the above hardware configuration is merely an example, and the above image processing may be performed on any other hardware. For example, the above processing may be executed on any information processing device such as a personal computer, a tablet terminal, a smartphone, a server on the Internet, etc. Further, the above image processing may be executed in an information processing system including a plurality of devices. The plurality of devices may execute the above image processing in a distributed manner. Also, the device that performs the above image processing and the device that displays the image may be different. For example, a first device (e.g., a server) may generate an image by executing part or all of the above image processing, and transmit the generated image to a second device via a network (e.g., the Internet or a LAN), and the second device may display the image.

[0164] Also, the configurations according to the above embodiments and their modifications can be arbitrarily combined as long as they do not conflict with each other. Also, the above is merely an exemplification of the present invention, and various improvements and modifications other than the above may be added.

Explanation of Reference Numerals

[0165] 1 Game system 2 Main body device 81 Processor 32 Tree object 34 Cylindrical object 36 Mountain object 41, 43 Character object

Claims

1. The computer of the information processing device About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; causing a frame buffer to perform drawing based on the results of the first depth test; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, the position as a reflection position, and a direction obtained by further correcting the incident direction and the normal direction of the reflection position as a ray tracing direction, the direction being a ray tracing direction obtained by correcting the incident direction and the normal direction of the reflection position as a ray tracing direction; tracing the ray along the ray tracing direction and determining a collision position at which the ray collides with an object in the virtual space based on the first depth buffer; When the collision position is determined within a range where the tracing distance of the ray is within a first distance, an image processing program determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as a reflection color to be added to the color of the pixel of interest.

2. 2 . The image processing program according to claim 1 , wherein the correction is a correction for moving a screen coordinate of an end position toward the inside of the screen, the end position being a position that is a second distance from the reflection position in the ray tracing direction.

3. The image processing program according to claim 2 , wherein the second distance is the first distance.

4. 3. The image processing program according to claim 2, wherein the correction is a correction for moving the screen coordinates of the end position so as to approach the screen coordinates of the reflection position by a predetermined degree.

5. 5. The image processing program according to claim 4, wherein the degree becomes higher as the screen coordinates of the reflection position are closer to an edge of the screen.

6. 5. The image processing program according to claim 4, wherein the correction is performed only when the screen coordinates of the reflection position are within a predetermined range near an edge of the screen.

7. The image processing program according to claim 4 , wherein the correction is performed only when the reflection direction faces toward the outside of the screen.

8. 8. The image processing program according to claim 2, wherein the correction is performed only on left and right components of screen coordinates.

9. 1. An image processing system comprising a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on the results of the first depth test; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest is defined as an incident direction, and the position is defined as a reflection position, and a direction obtained by further correcting the incident direction and the normal direction of the reflection position toward the inside of the screen is calculated as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position at which the ray collides with an object in the virtual space based on the first depth buffer; An image processing system that, when the collision position is determined within a range where the tracing distance of the ray is within a first distance, determines a color based on the color of the pixel in the frame buffer corresponding to the collision position as the reflection color to be added to the color of the pixel of interest.

10. 10. The image processing system according to claim 9, wherein the correction is a correction for moving the screen coordinates of an end position toward the inside of the screen, the end position being a position that is a second distance from the reflection position in the ray tracing direction.

11. The image processing system of claim 10 , wherein the second distance is the first distance.

12. 11. The image processing system according to claim 10, wherein the correction is a correction for moving the screen coordinates of the end position so as to approach the screen coordinates of the reflection position by a predetermined degree.

13. The image processing system according to claim 12 , wherein the degree is higher as the screen coordinates of the reflection position are closer to an edge of the screen.

14. 13. The image processing system according to claim 12, wherein the correction is performed only when the screen coordinates of the reflection position are within a predetermined range near an edge of the screen.

15. The image processing system according to claim 12 , wherein the correction is performed only when the reflection direction faces outside the screen.

16. 16. The image processing system according to claim 10, wherein the correction is performed only on left-right components of screen coordinates.

17. 1. An image processing method, comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on a result of the first depth test; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, the position as a reflection position, and a direction obtained by further correcting the incident direction and the normal direction of the reflection position, toward the inside of the screen, as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position at which the ray collides with an object in the virtual space based on the first depth buffer; When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of a pixel in the frame buffer corresponding to the collision position is determined as a reflection color to be added to the color of the pixel of interest.

18. 18. The image processing method according to claim 17, wherein the correction is a correction for moving the screen coordinates of an end position toward the inside of the screen, the end position being a position that is a second distance from the reflection position in the ray tracing direction.

19. The image processing method of claim 18 , wherein the second distance is the first distance.

20. 20. The image processing method according to claim 18, wherein the correction is a correction for moving the screen coordinates of the end position so as to approach the screen coordinates of the reflection position by a predetermined degree.

21. The image processing method according to claim 20 , wherein the degree becomes higher as the screen coordinates of the reflection position are closer to an edge of the screen.

22. 21. The image processing method according to claim 20, wherein the correction is performed only when the screen coordinates of the reflection position are within a predetermined range near an edge of the screen.

23. The image processing method according to claim 20 , wherein the correction is performed only when the reflection direction faces outside the screen.

24. 24. The image processing method according to claim 18, wherein the correction is performed only on horizontal components of screen coordinates.

25. An image processing device comprising a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on the results of the first depth test; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest is defined as an incident direction, and the position is defined as a reflection position, and a direction obtained by further correcting the incident direction and the normal direction of the reflection position toward the inside of the screen is calculated as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position at which the ray collides with an object in the virtual space based on the first depth buffer; An image processing device that, when the collision position is determined within a range where the tracing distance of the ray is within a first distance, determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as a reflection color to be added to the color of the pixel of interest.

Images