Image processing program, image processing system, image processing method, and image processing apparatus

Abstract

Provided is an image processing program capable of varying the way of reflection for each object. 【Solution means】An image processing system according to an embodiment updates a first depth buffer by performing a first depth test on an object in a virtual space, and performs rendering on a frame buffer based on the first depth buffer. The image processing system updates a second depth buffer by performing a second depth test on a specific object among the objects. When the trace distance is less than a second distance that is less than a first distance, the image processing system performs ray tracing based on the first depth buffer, and when the trace distance is greater than or equal to the second distance, the image processing system performs ray tracing based on the second depth buffer, and calculates the color of a pixel in the frame buffer corresponding to the collision position of the ray as the reflection color.

Description

Technical Field

[0001] The present invention relates to an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of expressing reflection based on a drawn image.

Background Art

[0002] Conventionally, there has been a method of expressing reflection by performing processing on a drawn image (for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the above prior art is processing on a drawn image, it has been difficult to adjust the way of reflection for each object.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of making the way of reflection different for each object.

Means for Solving the Problems

[0006] In order to solve the above problems, the present invention adopts the following configuration.

[0007] (First Configuration) The image processing program of the first configuration causes the computer of the information processing apparatus to perform a first depth test and update the first depth buffer for an object in the virtual space using the first depth buffer, cause drawing to be performed on the frame buffer based on the result of the first depth test, and further cause the computer to perform a second depth test and update the second depth buffer for an object of the first type among the objects using the second depth buffer. Further, the image processing program causes the computer to, for each pixel of the frame buffer on which drawing has been performed, with the pixel being the pixel of interest, calculate, based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest as the incident direction, the direction of the ray reflected with the position as the reflection position as the ray tracing direction, trace the ray along the ray tracing direction, determine the collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer when the trace distance of the ray is less than a second distance smaller than a first distance and based on the depth of the second depth buffer when the trace distance is equal to or greater than the second distance, and determine, when the collision position is determined within the range where the trace distance is within the first distance, the color based on the color of the pixel in the frame buffer corresponding to the collision position as the reflected color to be added to the color of the pixel of interest.

[0008] According to the above, by switching the depth buffer to be referred to according to the trace distance, for example, it is possible to reflect an object far away and make the reflection method different for each object.

[0009] (Second Configuration) In the second configuration, in the first configuration described above, the computer may further determine the reflected color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position based on different parameters between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

[0010] According to the above, depending on the depth buffer to be referred to, based on different parameters, the color of the pixel corresponding to the collision position can be added to the color of the pixel of interest.

[0011] (Third configuration) In the third configuration, in the above first or second configuration, the computer further arranges a planar object, which is set as a texture with an image rendered without drawing to the frame buffer for a second type of object, at the position of the second type of object in the virtual space. When the ray intersects the planar object, it is determined as the reflected color to be further added to the color of the pixel of interest based on the color of the texture at the intersection position, the reflected color is reflected in the frame buffer, and after reflecting the reflected color, the second type of object may be drawn to the frame buffer.

[0012] According to the above, it is possible to further generate a reflection of the second type of object. For example, even when the second type of object exists in front of the object, it is possible to generate the reflection of the object and the reflection of the second type of object.

[0013] (Fourth configuration) In the fourth configuration, in any of the above first to third configurations, the computer may further calculate a direction obtained by adding a correction toward the inside of the screen to the reflection direction based on the incident direction and the normal direction of the reflection position as the ray tracing direction.

[0014] According to the above, ray tracing can be performed based on a direction obtained by correcting the reflection direction based on the normal direction.

[0015] (Fifth configuration) In the fifth configuration, in the above third configuration, the second type of object may be a flat object.

[0016] According to the above, the second type of object is a flat object, and since the reflection also appears planar, an image without a sense of incongruity can be obtained.

[0017] Further, other embodiments may be an image processing system that performs the above image processing, an image processing apparatus, or an image processing method.

Advantages of the Invention

[0018] According to the present invention, the way of reflection can be made different for each object.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Mode for Carrying Out the Invention

[0020] (System Configuration) Hereinafter, a game system according to an example of the present embodiment will be described. An example of the game system 1 in the present embodiment includes a main body device (information processing device; which functions as a game device main body in the present embodiment) 2, a left controller 3, and a right controller 4. The main body device 2 is detachable from the left controller 3 and the right controller 4 respectively. That is, the game system 1 can be used as a device in which the left controller 3 and the right controller 4 are respectively attached to the main body device 2 and integrated. Also, the game system 1 can be used with the main body device 2, the left controller 3, and the right controller 4 as separate entities. Hereinafter, the hardware configuration of the game system 1 of the present embodiment will be described, and then the control of the game system 1 of the present embodiment will be described.

[0021] FIG. 1 is a diagram showing an example of a state in which the left controller 3 and the right controller 4 are attached to the main body device 2. As shown in FIG. 1, the left controller 3 and the right controller 4 are respectively attached to the main body device 2 and integrated. The main body device 2 is a device that executes various processes (for example, game processes) in the game system 1. The main body device 2 includes a display 12. The left controller 3 and the right controller 4 are devices that include an operation unit for the user to input.

[0022] The left controller 3 and the right controller 4 are detachable from the main body device 2. In the following, the left controller 3 and the right controller 4 may be collectively referred to as "controller".

[0023] The main body device 2 alone or the integrated device with the left controller 3 and the right controller 4 attached to the main body device 2 may be a portable device. Also, the main body device 2 or the integrated device may be a hand-held device. Further, the main body device 2 or the integrated device may be a transportable device.

[0024] Also, the main body device 2 includes a touch panel 13 on the screen of the display 12. In the present embodiment, the touch panel 13 is of a type (for example, a capacitance type) capable of multi-touch input. However, the touch panel 13 may be of any type, for example, a type (for example, a resistive film type) capable of single-touch input.

[0025] FIG. 2 is a block diagram showing an example of the internal configuration of the main body device 2.

[0026] The main body device 2 includes a processor 81. The processor 81 is an information processing unit that executes various information processes executed in the main body device 2, and includes one or more CPUs (Central Processing Units) and one or more GPUs (Graphics Processing Units). The processor 81 may be composed of a SoC (System-on-a-chip) including a plurality of functions such as a CPU function and a GPU function. Note that the CPU and the GPU may be configured as separate processors. Also, one or more memories for temporarily storing data are provided inside the processor 81. The processor 81 executes various information processes by executing an information processing program (for example, a game program) stored in a storage unit (specifically, an internal storage medium such as a flash memory 84 or an external storage medium attached to the slot 23).

[0027] The main body device 2 includes a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85 as an example of an internal storage medium built therein. The flash memory 84 and the DRAM 85 are connected to the processor 81. The flash memory 84 is mainly a memory used to store various data (which may be programs) stored in the main body device 2. The DRAM 85 is a memory used to temporarily store various data used in information processing.

[0028] The main body device 2 includes a slot interface (hereinafter abbreviated as "I / F") 91. The slot I / F 91 is connected to the processor 81. The slot I / F 91 is connected to the slot 23 and reads and writes data to and from a predetermined type of storage medium (for example, a dedicated memory card) mounted on the slot 23 in accordance with instructions from the processor 81.

[0029] The processor 81 appropriately reads and writes data between the flash memory 84 and the DRAM 85 and each of the above storage media to execute the above information processing.

[0030] The main body device 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates (specifically, wirelessly) with an external device via a network. In the present embodiment, the network communication unit 82 connects to a wireless LAN and communicates with an external device by a method compliant with the Wi-Fi standard as a first communication mode. Further, the network communication unit 82 performs wireless communication with another main body device 2 of the same type by a predetermined communication method (for example, communication using a proprietary protocol or infrared communication) as a second communication mode.

[0031] The main body device 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 performs wireless communication with the left controller 3 and / or the right controller 4. The communication method between the main body device 2 and the left controller 3 and the right controller 4 is arbitrary. However, in the present embodiment, the controller communication unit 83 performs communication according to the Bluetooth (registered trademark) standard between the left controller 3 and the right controller 4.

[0032] The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 performs wired communication with the left controller 3, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. Also, when the processor 81 performs wired communication with the right controller 4, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. Further, when the processor 81 communicates with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in the present embodiment, the main body device 2 can perform both wired communication and wireless communication with the left controller 3 and the right controller 4, respectively. Also, when the left controller 3 and the right controller 4 are attached to the main body device 2 as an integrated device or when the main body device 2 alone is attached to the cradle, the main body device 2 can output data (for example, image data or audio data) to a stationary monitor or the like via the cradle.

[0033] The main body device 2 includes a touch panel controller 86 which is a circuit for controlling the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. The touch panel controller 86 generates data indicating, for example, the position where a touch input is made based on a signal from the touch panel 13 and outputs it to the processor 81.

[0034] The main body device 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Also, although not shown, the power control unit 97 is connected to each part of the main body device 2 (specifically, each part that receives power supply from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls the power supply from the battery 98 to each of the above parts based on a command from the processor 81.

[0035] Also, the battery 98 is connected to the lower terminal 27. When an external charging device (for example, a cradle) is connected to the lower terminal 27 and power is supplied to the main body device 2 via the lower terminal 27, the supplied power is charged to the battery 98.

[0036] (Overview of Image Processing) Next, the image processing of the present embodiment will be described. In the game system 1 (an example of an image processing system) of the present embodiment, a plurality of objects are arranged in a three-dimensional virtual space and a game is played.

[0037] FIG. 3 is a diagram showing an example of a plurality of objects arranged in the virtual space during the execution of the game of the present embodiment. When the game of the present embodiment is started, a three-dimensional virtual space defined by an xyz orthogonal coordinate system is set. For example, the y-axis is the upward axis of the virtual space, the x-axis is the rightward axis, and the z-axis is the depth axis.

[0038] As shown in FIG. 3, a ground object 30 is arranged in the virtual space as a plurality of objects. The ground object 30 is an object imitating the ground and is, for example, a plane parallel to the xz plane. Note that the ground object 30 may have unevenness or a slope. On the ground object 30, a tree object 32 and a cylindrical object 34 are arranged. Also, for example, a mountain object 36 is arranged at a predetermined position in the z-axis direction of the virtual space. These ground object 30, tree object 32, and cylindrical object 34 are fixed in the virtual space.

[0039] In the virtual space, character objects 41 and 43 are arranged. The character objects 41 and 43 are flat objects, and are plate-shaped objects whose length in the thickness direction is smaller than the lengths in the up-down, left-right directions. For example, the character objects 41 and 43 are 3D objects each including a planar mesh forming the front surface and a planar mesh forming the back surface. The character object 41 is a player character controlled by the player. The character object 41 performs actions according to the input to the controller within the virtual space. For example, as actions, the character object 41 moves on the ground object 30, moves its limbs within the virtual space, jumps, or changes the direction of its face. When the character object 41 moves in the left-right direction, the screen is scrolled in the left-right direction. Also, the character object 43 is a non-player character that moves according to the movement of the character object 41 and is automatically controlled by the processor 81. Note that the character object 43 may be controlled by the player.

[0040] A virtual camera is arranged in the virtual space, a game image of the virtual space viewed from the virtual camera is generated, and the generated game image is displayed on the display 12 or a stationary monitor (hereinafter, these are referred to as "display device").

[0041] FIG. 4 is a diagram showing an example of a game image displayed on the display device. Images of the respective objects (30, 32, 34, 36, 41, 43) are displayed on the display device. For example, the ground object 30 is a reflecting surface, and the respective objects are reflected on the ground object 30. In the present embodiment, the reflection of each object is generated by a method called SSR (Screen Space Reflection).

[0042] For example, as shown in FIG. 4, on the ground object 30, the reflection 52 of the tree object 32 and the reflection 54 of the cylindrical object 34 are displayed. Also, on the ground object 30, the reflection 61 of the character object 41 and the reflection 63 of the character object 43 are displayed. On the other hand, the reflection of the mountain object 36 is not displayed on the ground object 30.

[0043] In this embodiment, the tree object 32 and the cylindrical object 34 are set in advance as specific objects, and are set to be reflected on the ground object even when they are a certain distance away from the virtual camera. On the other hand, the mountain object 36 is not set as a specific object.

[0044] The specific object is set in units of meshes (aggregates of polygons). For example, each of the plurality of meshes constituting the tree object 32 may be set as a specific object. Also, any one of the plurality of meshes constituting the tree object 32 may be set as a specific object. For example, when the tree object 32 is composed of a first mesh constituting the trunk portion and a second mesh constituting the leafy portion, only the first mesh among the first mesh and the second mesh may be set as a specific object. That is, the "specific object" does not necessarily have to be an object that appears as one in appearance, and may be a mesh that constitutes a part of an object that appears as one in appearance. For example, a terrain object in which a ground parallel to the xz plane and a wall surface perpendicular to the xz plane are continuously formed may appear to form one terrain with a step in appearance, but the ground and the wall surface may be composed of different meshes. In such a terrain object, the wall surface may be set as a specific object, and the ground does not have to be set as a specific object.

[0045] In this embodiment, a tree object 32 and a cylinder object 34 set as specific objects are reflected onto the ground object. Hereinafter, image processing related to the reflection of these objects will be described.

[0046] FIG. 5 is a diagram showing an outline of image processing related to the reflection of objects other than character objects.

[0047] As shown in FIG. 5, the processor 81 first performs a first depth test (step S1). Here, for all objects (30, 32, 34, 36) other than the character object, a depth test is performed in pixel units, and the result of the depth test is stored in the first depth buffer.

[0048] FIG. 6 is a diagram showing an example of the first depth buffer. As shown in FIG. 6, in the first depth buffer, depth values for each pixel are stored for all objects (30, 32, 34, 36) other than the character object. In FIG. 6, although the images of each object are displayed, this conceptually represents the depth values of each pixel stored in the first depth buffer. The position of each pixel is represented by coordinate values in the screen coordinate system. For example, the Sx axis of the screen coordinate system is the axis in the right direction in the image (screen), and the Sy axis is the axis in the upward direction in the image. The origin of the screen coordinate system may be set, for example, at the center of the image or at the lower left of the image.

[0049] Returning to FIG. 5, after step S1, the processor 81 performs rendering to the frame buffer based on the result of the first depth test (step S2). In step S2, an image is drawn on the frame buffer based on the depth stored in the first depth buffer. In the present embodiment, as a rendering method, deferred rendering (also referred to as deferred shading) is used. In step S2, the normal buffer storing the normal information for each pixel is updated, and an image is drawn on the frame buffer. Note that, as a rendering method, forward rendering may be used instead of deferred rendering.

[0050] FIG. 7 is a diagram showing an example of an image drawn on the frame buffer after the rendering by step S2. As shown in FIG. 7, in step S2, images of all objects (30, 32, 34, 36) other than the character object are drawn on the frame buffer. Color information is stored for each pixel. The color information includes, for example, RGB values representing three colors and an alpha value representing transparency (or opacity). Although the shadow is omitted in FIG. 7, in step S2, an image with a shadow is generated.

[0051] Returning to FIG. 5, after step S2, the processor 81 further performs a second depth test on a specific object among the plurality of objects arranged in the virtual space (step S3). Specifically, among the plurality of objects (30, 32, 34, 36), a second depth test is further performed on the tree object 32 and the cylinder object 34 set as the specific object, and the result of the second depth test is stored in the second depth buffer. Thereby, in the second depth buffer, the depth value for each pixel of only the tree object 32 and the cylinder object 34 among the plurality of objects (30, 32, 34, 36) included in the imaging range of the virtual camera is stored.

[0052] FIG. 8 is a diagram showing an example of the second depth buffer. As shown in FIG. 8, depth values for each pixel of the tree object 32 and the cylinder object 34 are stored in the second depth buffer. On the other hand, the depth value of the mountain object 36 is not stored in the second depth buffer. Note that FIG. 8 conceptually represents the depth values of the respective pixels stored in the second depth buffer.

[0053] Returning to FIG. 5, after step S3, the processor 81 performs ray tracing using the first depth buffer or the second depth buffer (step S4). Here, processing is performed for each pixel (hereinafter referred to as the "target pixel"). Specifically, the processor 81 sets the target pixel, calculates the reflection position of a ray (virtual light ray) when the ray is emitted from the position of the virtual camera in the direction toward the target pixel, and performs a collision determination as to whether the ray reflected from the reflection position collides with an object.

[0054] Next, the processor 81 calculates the color of the pixel in the frame buffer corresponding to the collision position of the ray as the color reflected in the target pixel (step S5). The processing in step S5 is also performed for each pixel.

[0055] Hereinafter, the processing of steps S4 and S5 when each of the pixels PIXa to PIXc shown in FIG. 7 is set as the target pixel will be specifically described.

[0056] For example, the processor 81 sets the pixel PIXa shown in FIG. 7 as the target pixel. The pixel PIXa is a pixel on the virtual camera side of the cylinder object 34. The processor 81 calculates the position of the target pixel PIXa in the virtual space as the ray reflection position RPa based on the position of the target pixel PIXa in the screen coordinate system and the depth value of the position stored in the first depth buffer.

[0057] FIG. 9 is a diagram showing how ray tracing is performed, and is a diagram showing how a ray reflected from a reflection position RPa collides with a cylindrical object 34. The right direction in FIG. 9 is the depth direction of the screen when a game image generated based on the virtual camera VC is displayed on the screen.

[0058] As shown in FIG. 9, the processor 81 calculates the direction from the position of the virtual camera VC to the reflection position RPa as the incident direction, and calculates the direction of the ray reflected from the reflection position RPa as the ray tracing direction RTDa. Specifically, the processor 81 calculates the reflection direction of the ray based on the incident direction and the normal line of the reflection position RPa. The incident angle of the incident direction is equal to the reflection angle of the reflection direction. Then, the processor 81 calculates the reflection direction or the direction obtained by correcting the reflection direction as the ray tracing direction RTDa. The correction of the reflection direction will be described later.

[0059] The processor 81 advances the ray in the ray tracing direction RTDa from the reflection position RPa, and performs a collision determination as to whether the ray collides with an object. When the trace distance of the ray is less than La, the processor 81 performs the collision determination using the first depth buffer. Here, the trace distance is the distance along the ray tracing direction from the reflection position.

[0060] FIG. 10 is a diagram showing a state in which a collision determination of a ray reflected from a reflection position RPa is performed using a first depth buffer. First, the processor 81 calculates a position in the virtual space that has advanced by the maximum distance Lb in the ray tracing direction RTDa from the reflection position RPa in the virtual space, and calculates the ray tracing end position in the screen coordinate system by converting the calculated position into the screen coordinate system. The processor 81 extends a ray from the reflection position RPa (the pixel of interest RPa) as the ray tracing start position to the ray tracing end position in the screen coordinate system, and determines whether the tip of the ray collides with an object. Specifically, the processor 81 advances the ray by a predetermined distance, and determines whether the ray collides with an object based on the depth of the tip of the ray and the depths of the respective positions stored in the first depth buffer. In the examples shown in FIGS. 9 and 10, it is determined that the ray reflected at the reflection position RPa collides at the collision position CPa.

[0061] When it is determined that the ray collides with an object at the collision position CPa, in step S5, the color of the pixel in the frame buffer corresponding to the collision position CPa is calculated as the color reflected in the pixel of interest RPa.

[0062] Next, a case where the pixel PIXb shown in FIG. 7 is set as the pixel of interest and a case where the pixel PIXc is set as the pixel of interest will be described. The pixel PIXb is a pixel on the virtual camera side of the tree object 32. The pixel PIXc is a pixel on the virtual camera side of the mountain object 36.

[0063] FIG. 11 is a diagram showing a state in which ray tracing is performed, and shows a state in which a ray reflected from a reflection position RPb collides with a tree object 32. FIG. 12 is a diagram showing a state in which ray tracing is performed, and shows a state in which a ray reflected from a reflection position RPc does not collide with an object. FIG. 13 is a diagram showing a state in which a collision determination of a ray is performed using a second depth buffer.

[0064] As shown in FIG. 11, when the processor 81 sets the pixel PIXb as the pixel of interest, in the same manner as above, the reflection position RPb corresponding to the pixel of interest PIXb is calculated, and the direction of the ray reflected from the reflection position RPb is calculated as the ray tracing direction RTDb. The processor 81 advances the ray in the ray tracing direction RTDb and performs a collision determination as to whether the ray collides with an object. When the trace distance is less than La, the processor 81 performs a collision determination of the ray using the first depth buffer. However, when the trace distance is less than La, the ray does not collide with an object. Therefore, the processor 81 further advances the ray and performs a collision determination of the ray using the second depth buffer (FIG. 13). As shown in FIGS. 11 and 13, when the trace distance is equal to or greater than La and equal to or less than the maximum distance Lb, it is determined that the ray collides with the tree object 32 at the collision position CPb.

[0065] On the other hand, as shown in FIG. 12, when the processor 81 sets the pixel PIXc as the pixel of interest, in the same manner as above, the reflection position RPc corresponding to the pixel of interest PIXc is calculated, the direction of the ray reflected from the reflection position RPc is calculated as the ray tracing direction RTDc, and the ray is advanced in the ray tracing direction RTDb. When the trace distance is less than La, the processor 81 performs a collision determination of the ray using the first depth buffer. When the trace distance is equal to or greater than La and equal to or less than the maximum distance Lb, the processor 81 performs a collision determination of the ray using the second depth buffer. Here, as shown in FIG. 13, the depth of the mountain object 36 is not stored in the second depth buffer. Therefore, as shown in FIGS. 12 and 13, the processor 81 determines that the ray does not collide with an object even if the ray is advanced to the maximum distance Lb.

[0066] In addition, when the collision position of the ray is calculated based on the first depth buffer and when the collision position of the ray is calculated based on the second depth buffer, the parameters for adding the color of the pixel at the collision position to the color of the pixel of interest may be different. For example, when the collision position is calculated based on the second depth buffer, the parameters may be changed so that the reflected color is darker than when the collision position is calculated based on the first depth buffer.

[0067] By performing the processes of steps S4 and S5 described above for each pixel of interest, the color reflected in each pixel of interest is determined. Then, by adding the determined reflected color to the color of each pixel of interest, the reflection of each object is drawn on the frame buffer.

[0068] FIG. 14 is a diagram showing an example of an image drawn on the frame buffer when the color reflected in each pixel is added.

[0069] As shown in FIG. 14, for pixel RPa, the color of the pixel at the collision position CPa calculated as described above is added. By performing the same process for each pixel included in the area in front of the cylindrical object 34 (the area on the virtual camera side), the reflection 54 of the cylindrical object 34 is drawn on the frame buffer. Also, for pixel RPb, the color of the pixel at the collision position CPb is added. By performing the same process for each pixel included in the area in front of the wooden object 32, the reflection 52 of the wooden object 32 is drawn on the frame buffer. On the other hand, for pixel RPc, the collision position is not calculated. The same applies when the process is performed for each pixel included in the area in front of the mountain object 36. Therefore, the reflection of the mountain object 36 is not drawn on the frame buffer.

[0070] As described above, in this embodiment, the results of performing the first depth test on a plurality of objects are stored in the first depth buffer, and the results of performing the second depth test on a specific object among the plurality of objects are stored in the second depth buffer. When the trace distance is less than La, a ray collision determination is made based on the first depth buffer. When the trace distance is greater than or equal to La and less than or equal to the maximum distance Lb, a ray collision determination is made based on the second depth buffer. Thereby, by using SSR, for a specific object among the plurality of objects, even when it is far from the virtual camera, a reflection can be displayed, and for objects other than the specific object, a reflection can be prevented from being displayed. By designating the object to be reflected as a specific object, a desired object can be reflected, for example, on the ground. Conversely, by not designating the object that is not desired to be reflected as a specific object, a desired object can be prevented from being reflected.

[0071] In addition, the parameters for reflecting the color of the pixel at the collision position to the target pixel are made different between the case where the collision position of the ray is calculated based on the first depth buffer and the case where the collision position of the ray is calculated based on the second depth buffer. Thereby, for example, a specific object can be reflected more darkly and clearly.

[0072] (Correction of the reflection direction) Next, the correction of the reflection direction will be described. As described above, the direction from the virtual camera VC toward the reflection position is calculated as the incident direction, and the reflection direction is calculated based on the incident direction and the normal direction of the reflection position. In this embodiment, when the correction condition is satisfied, the corrected direction of the reflection direction is set as the ray tracing direction. When the correction condition is not satisfied, the reflection direction is set as the ray tracing direction. Hereinafter, the correction condition for correcting the reflection direction and the method of correction will be specifically described.

[0073] FIG. 15 is a diagram showing an example of the ray tracing direction with the reflection direction corrected. As shown in FIG. 15, for example, when ray tracing is performed for a target pixel (reflection position RP1) near the left end of the screen, the reflection direction RD1 calculated based on the incident direction and the normal direction is corrected, and the corrected direction is set as the ray tracing direction RTD1. Also, when ray tracing is performed for a target pixel (reflection position RP2) on the right side of the screen relative to the reflection position RP1, the reflection direction RD2 is corrected, and the corrected direction is set as the ray tracing direction RTD2.

[0074] The correction condition is satisfied when both the condition regarding the reflection position and the condition regarding the reflection direction are met. The condition regarding the reflection position is that the reflection position is within a predetermined range from the edge of the screen. Also, the condition regarding the reflection direction is that the reflection direction faces outside the screen. When either the condition regarding the reflection position or the condition regarding the reflection direction is not met, the correction condition is not satisfied and the reflection direction is not corrected. For example, when the reflection position is within the range from the left end of the screen to 1 / 4 of the length of the screen in the left-right direction and the reflection direction faces the left direction of the screen, the correction condition is satisfied. Also, when the reflection position is within the range from the right end of the screen to 1 / 4 and the reflection direction faces the right direction of the screen, the correction condition is satisfied.

[0075] The reflection position RP1 closer to the left end of the screen has a greater degree of correction than the reflection position RP2. Specifically, the degree of correction CR is determined based on the position of the reflection position in the Sx-axis direction of the screen coordinate system. The closer the position of the reflection position in the Sx-axis direction is to the left end or the right end of the screen, the greater the degree of correction CR becomes. Also, the greater the angle between the reflection direction and the Sy-axis direction in the screen coordinate system, the smaller the degree of correction CR becomes.

[0076] FIG. 16 is a diagram for explaining the details of correction with respect to the reflection direction. As shown in FIG. 16, for example, when the angle between the reflection direction RD and the upward direction of the screen (Sy-axis direction) is 45 degrees or less, the degree of correction CR is set to a value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP. CR(Sx) is set, for example, in the range of 0 to 1, and becomes larger as the reflection position is closer to the left or right end of the screen. When the angle between the reflection direction RD and the upward direction of the screen exceeds 45 degrees, the value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP becomes smaller according to the exceeded angle. When the angle between the reflection direction RD and the upward direction of the screen exceeds a threshold value (for example, 60 degrees to 65 degrees), the degree of correction CR is set to "0".

[0077] According to the degree of correction CR, a correction toward the inside of the screen is applied in the reflection direction. Specifically, the position obtained by converting the position advanced by the maximum distance Lb in the reflection direction RD from the reflection position RP in the virtual space into the screen coordinate system is calculated as the ray tracing end position EP. The Sx coordinate value of this ray tracing end position EP is corrected to move in the direction of the inside of the screen according to the degree of correction CR. For example, by linear interpolation, the Sx coordinate value of the ray tracing end position EP is corrected to approach the Sx coordinate value of the reflection position RP. For example, when the reflection position RP is located within a predetermined range from the left end of the screen, the ray tracing end position EP is moved in the right direction (positive Sx-axis direction) according to the degree of correction CR. The Sx coordinate value of the ray tracing end position EP does not move beyond the Sx coordinate value of the reflection position RP. Also, the Sy coordinate value of the ray tracing end position EP does not change before and after the correction. Then, the direction from the reflection position EP to the moved ray tracing end position EP' is calculated as the ray tracing direction RTD.

[0078] Thus, in the area near the edge of the screen, the direction obtained by correcting the reflection direction is calculated as the ray tracing direction. As a result, at the edge of the screen, it is possible to generate the reflection by SSR. For example, as shown in FIG. 15, for the pixel at the position RP1 at the left end of the screen, when ray tracing is performed in the reflection direction RD1, the collision position of the ray may be outside the screen, and the color of the pixel at the collision position cannot be added to the pixel at the position RP1. Therefore, if no correction is made to the reflection direction, the edge portion of the screen may have a different color from other portions, resulting in an image that gives an uncomfortable feeling.

[0079] In this embodiment, since the reflection direction based on the incident direction and the normal direction is corrected and the corrected direction is set as the ray tracing direction, it is possible to easily keep the collision position of the ray within the screen, and it is possible to generate the reflection based on the rendered image. The Sx coordinate value of the ray tracing end position EP is moved so as to approach the Sx coordinate value of the reflection position RP. For this reason, even if the maximum correction is made to the reflection direction facing the outside of the screen, it does not become a direction facing the inside of the screen, but becomes the upward direction of the screen. Therefore, for example, the color of the pixel on the right side of the pixel at the left end position of the screen is not reflected. As a result, it is possible to prevent the object from being unnaturally distorted and reflected on the ground.

[0080] Also, in this embodiment, the degree of correction increases as the reflection position approaches the edge of the screen. Therefore, even in the vicinity of the edge of the screen, it is possible to generate the reflection based on the pixels within the screen.

[0081] Also, for example, when the reflection direction faces the inside of the screen, if the reflection direction is corrected, the reflection may be far from the actual reflection. In this embodiment, when the reflection direction does not face the outside of the screen, the reflection direction is not corrected, so it is possible to prevent the generation of a reflection far from the actual reflection.

[0082] Also, when the angle between the reflection direction and the upward direction of the screen is relatively large (for example, exceeding 45 degrees), the correction degree CR becomes small. When the angle between the reflection direction and the upward direction of the screen exceeds the threshold value, the correction degree becomes 0. Thereby, for example, when the ground is inclined, it is possible to prevent the correction degree from being too large and generating unnatural reflections. Also, it is possible to suppress the correction for reflections on a wall surface where the reflection direction is horizontal.

[0083] Also, in this embodiment, the correction of the reflection direction is performed only for the horizontal component in the screen coordinate system. Therefore, for example, it is suitable for a game in which a virtual space extends in the left - right direction of the screen and is scrolled horizontally.

[0084] (Generation of reflection of character object) Next, a method for generating the reflections of the character objects 41 and 43 will be described. FIG. 17 is a diagram showing an example of a game image when the character object 41 does not exist in front of the cylindrical object 34. FIG. 18 is a diagram showing an example of a game image when the character object 41 moves and the character object 41 exists in front of the cylindrical object 34.

[0085] As shown in FIG. 17, when the character object 41 does not exist in front of the cylindrical object 34, for example, a ray trace is performed from the front - side reflection position RPd, the collision position CPd is calculated, and the color of the collision position CPd is added to the color of the reflection position RPd. By performing the same processing for each pixel, the reflection 54 of the cylindrical object 34 is displayed.

[0086] On the other hand, in FIG. 18, the character object 41 exists in front of the cylindrical object 34, and the head of the character object 41 exists at the same position CPd as in FIG. 17. In this case, the color of the pixel at the collision position CPd of the cylindrical object 34 shown in FIG. 17 cannot be added to the pixel at the reflection position RPd, and a part of the cylindrical object 34 is not reflected in the reflection position RPd. Thus, for example, when the character object 41 is located in front of the cylindrical object 34, it may not be possible to express a natural reflection according to the positional relationship of the original objects.

[0087] In the present embodiment, in order to display the reflection of the character object and the object behind it, image processing is performed by the method shown in FIG. 19. FIG. 19 is a diagram showing an outline of image processing for generating the reflection of the character object and the object behind it.

[0088] As shown in FIG. 19, first, the processor 81 renders an object other than the character object (step S10). Here, a process of drawing a plurality of objects 30, 32, 34, 36 arranged in the virtual space in the frame buffer is performed. The process of step S10 is the process of steps S1 to S2 in FIG. 5.

[0089] Next, the processor 81 renders the character object in a separate buffer and creates a character plane in which the rendered image is set as a texture (step S11).

[0090] FIG. 20 is a diagram showing an example of a character plane created in step S11. As shown in FIG. 20, a character object 41 is rendered in a separate buffer, and an image obtained by rendering the character object 41 is set as a texture in a two-dimensional plane object, thereby creating a character plane 65. Similarly, a character plane 66 is created by setting, as a texture, an image obtained by rendering the character object 43. In the present embodiment, an image obtained by rendering each character object is generated every frame. Note that an image obtained by rendering each character object may be prepared in advance. For example, an image corresponding to each action of the character object 41 may be stored in a storage medium in advance, and a character plane 65 may be created by setting the pre-stored image as a texture.

[0091] Next, the processor 81 calculates the reflection color of an object other than the character object (step S12). Here, the reflection color is calculated for a plurality of objects 30, 32, 34, 36 arranged in the virtual space. Specifically, the processor 81 performs the processes of steps S3 to S5 in FIG. 5 described above. At this point, since the character object is not drawn in the frame buffer, even if the character object exists on the front side of the object as viewed from the virtual camera VC in the virtual space, the reflection color of the object can be calculated. The calculated reflection color of an object other than the character object (hereinafter referred to as "the first reflection color") is stored in the reflection buffer. For example, RGB values and alpha values are stored in the reflection buffer as color information indicating the first reflection color.

[0092] Next, the processor 81 calculates the reflection color of the character (step S13). Here, the processor 81 calculates the reflection color of the character object when the character plane created in step S11 is arranged in the virtual space.

[0093] FIG. 21 is a diagram for explaining the calculation of the color of the reflection of the character in step S13. As shown in FIG. 21, the processor 81 calculates the intersection position of the character plane 65 and the ray when the character plane 65 is arranged in the virtual space according to the position and orientation of the character object 41 in the virtual space. Specifically, the processor 81 calculates the depth when the character plane 65 is arranged in the virtual space, and calculates the intersection position CRPe of the ray from the reflection position RPe in the ray tracing direction RTDe and the character plane 65. The color of the pixel at the intersection position CRPe is calculated as the color of the reflection of the character (hereinafter referred to as the "second reflection color"). Here, among the character plane 65, the portions other than the image of the character object 41 are set to be transparent and do not intersect with the ray. For example, the ray from the reflection position RPf in the ray tracing direction RTDf does not intersect with the image of the character object 41 in the character plane 65. In this case, the intersection position is not calculated. The calculated second reflection color is stored in the reflection buffer. For example, in the reflection buffer, the RGB value and the alpha value are stored as the color information indicating the second reflection color.

[0094] Returning to FIG. 19, the processor 81 then calculates the color reflected on the pixel of interest based on the results of step S12 and the results of step S13 (step S14). For example, the color reflected on the pixel of interest is calculated by adding the second reflection color to the first reflection color. Here, "adding the second color to the first color" may mean mixing the two colors. For example, "adding the second color to the first color" may mean performing alpha blending with the first color as the background color and the second color as the foreground color. Also, "adding the second color to the first color" may mean performing alpha blending with the second color as the background color and the first color as the foreground color. Also, "adding the second color to the first color" may mean mixing the first color and the second color based on their respective alpha values. Also, "adding the second color to the first color" may mean that the second color is given priority over the first color and the first color is overwritten with the second color.

[0095] Next, the processor 81 renders the result calculated in step S14 in the frame buffer (step S15). Here, the processor 81 adds the reflected color calculated in step S14 to the color of the pixel of interest stored in the frame buffer. For example, the processor 81 performs alpha blending with the color of the pixel of interest as the background color and the reflected color calculated in step S14 as the foreground color. As a result, the reflections of the character object and objects other than the character object are drawn in the frame buffer.

[0096] After the process of step S15 is performed, the processor 81 renders the character objects 41 and 43 in the frame buffer (step S16). As a result, a game image as shown in FIG. 22 is generated and displayed on the display device.

[0097] FIG. 22 is a diagram showing an example of a game image displayed when the image processing of FIG. 19 is performed. As shown in FIG. 22, the tree object 32 and its reflection 52 are displayed. Also, the character object 43 and its reflection 63 are displayed. Further, the cylinder object 34 and its reflection 54 are displayed. There is a character object 41 in front of the cylinder object 34, and the character object 41 and its reflection 61 are displayed. A part of the reflection 54 of the cylinder object 34 and the reflection 61 of the character object 41 overlap, but both of them have a shape according to the actual shape and positional relationship, and a natural reflection can be expressed.

[0098] In this embodiment, the character object is a flat object. As described above, when generating the reflection of the character object based on the character plane where the image of the character object is set as the texture, the reflection of the character object may look flat. If the character object itself is a three-dimensional object having a certain thickness, the reflection may look flat and the image may be uncomfortable. However, in this embodiment, since the character object itself is a flat object, an image without discomfort can be obtained.

[0099] (Details of Image Processing) Next, the details of the above-described image processing will be described. First, the data stored in the memory of the main body device 2 (the memory in the processor 81, the DRAM 85, the flash memory 84, or an external storage medium, etc.) will be described. FIG. 23 is a diagram showing an example of the data stored in the memory of the main body device 2.

[0100] As shown in FIG. 23, the memory of the main body device 2 stores a program, operation data, object data, character data, and character plane data. Further, the memory of the main body device 2 stores a first depth buffer, a second depth buffer, a normal buffer, a reflection buffer, and a frame buffer.

[0101] The program is a program for executing the main process described later and includes an image processing program for performing the above-described image processing related to reflection. The program is stored in advance in an external storage medium or the flash memory 84 attached to the slot 23 and is read into the DRAM 85 when the game is executed. Note that the program may be acquired from another device via a network (for example, the Internet).

[0102] The operation data is data related to operations acquired from the left controller 3 and the right controller 4. For example, the operation data is transmitted from the left controller 3 and the right controller 4 to the main body device 2 at a predetermined time interval (for example, at intervals of 1 / 200 seconds) and stored in the memory.

[0103] The object data is data related to each object (30, 32, 34, 36) other than the character objects arranged in the virtual space. The object data includes data related to the position and orientation of each object in the virtual space, data related to the shape, and data related to the texture. Further, the object data includes information regarding whether each object is set as a specific object. For example, the tree object 32 and the cylinder object 34 are set as specific objects. On the other hand, the mountain object 36 is not set as a specific object.

[0104] The character data is data related to the character objects 41, 43. The character data includes data related to the position and orientation of each character in the virtual space, data related to the shape, and data related to the texture. In the present embodiment, each character object is formed as a 3D model, but is formed as a planar object that is thinner in the thickness direction than in the vertical and horizontal directions.

[0105] The character plane data is a two-dimensional object and is data related to a planar object in which an image obtained by rendering a character object is set as a texture. The character plane data includes data related to the character plane 65 in which an image of the character object 41 is set as a texture, and data related to the character plane 66 in which an image of the character object 43 is set as a texture.

[0106] The first depth buffer is a buffer that stores the depth value for each pixel generated and updated based on the result of the first depth test. In the first depth buffer, the depth values of all objects other than the character object included in the imaging range of the virtual camera VC are stored.

[0107] The second depth buffer is a buffer that stores the depth value for each pixel generated and updated based on the result of the second depth test. In the second depth buffer, the depth values of specific objects included in the imaging range of the virtual camera VC are stored.

[0108] The normal buffer is a buffer that stores the normal information of each pixel.

[0109] The reflection buffer is a buffer for storing the color of the first reflection (the reflection color of objects other than the character object) and the color of the second reflection (the reflection color of the character object). In the reflection buffer, in the ray tracing process described later, when the ray collides with an object other than the character object, the color of the first reflection is stored, and when the ray intersects with the character plane, the color of the second reflection is stored. Although the details will be described later, there may be a plurality of reflection colors stored in the reflection buffer.

[0110] The frame buffer is a buffer for storing the game image to be displayed. The image stored in the frame buffer is output to the display device at a predetermined timing and is displayed on the display device.

[0111] (Details of the game process in the main device 2) Next, with reference to FIGS. 24 to 27, the details of the process performed in the main device 2 will be described. FIG. 24 is a flowchart showing an example of the main process executed by the processor 81 of the main device 2. The processes shown in FIGS. 24 to 27 are performed by the CPU or GPU of the processor 81.

[0112] As shown in FIG. 24, the processor 81 first executes initial processing (step S100). Specifically, the processor 81 sets up a three-dimensional virtual space and arranges in the virtual space each object (30, 32, 34, 36), each character object (41, 43), the virtual camera VC, a light source, and various other objects used in the game. After executing the initial processing, the processor 81 repeatedly executes the processing of the following steps S101 to S108 at a predetermined frame time interval (for example, at intervals of 1 / 60 second).

[0113] In step S101, the processor 81 acquires operation data from the controller.

[0114] Next, the processor 81 performs game processing based on the acquired operation data (step S102). For example, based on the operation data, the processor 81 moves the character object 41 within the virtual space or causes the character object 41 to perform a predetermined action (for example, a jump action, an attack action, etc.). Also, the processor 81 moves the character object 43 within the virtual space or causes the character object 43 to perform a predetermined action according to a predetermined algorithm. Further, the processor 81 controls enemy objects other than the character objects within the virtual space or moves obstacle objects that obstruct the character objects within the virtual space.

[0115] Next, the processor 81 performs rendering processing on objects other than character objects (step S103). Specifically, the processor 81 performs a first depth test on each object other than the character objects 41 and 43 included in the imaging range of the virtual camera VC, and updates the first depth buffer. Further, the processor 81 draws each object on the frame buffer based on the depth stored in the first depth buffer. In the present embodiment, deferred rendering is used as the rendering method. In step S103, the normal buffer is also updated. Note that forward rendering may be used as the rendering method.

[0116] Next, the processor 81 renders the character object in a separate buffer to create a character plane (step S104). Specifically, the processor 81 renders the character object 41 in a separate buffer, and creates a character plane 65 by setting the image of the character object 41 rendered in the separate buffer as a texture in a plane object. Similarly, the processor 81 creates a character plane 66 in which the image of the character object 43 is set as a texture in a plane object.

[0117] After step S104, the processor 81 performs SSR processing (step S105). The SSR processing is a process for drawing the reflections of all objects including the character object on the frame buffer. The details of the SSR processing will be described below.

[0118] (SSR Processing) FIG. 25 is a flowchart showing an example of the SSR processing in step S105.

[0119] As shown in FIG. 25, the processor 81 performs a second depth test on a specific object and updates the second depth buffer (step S121). For example, the processor 81 performs a depth test on the tree object 32 and the cylinder object 34 that are set in advance as specific objects, and updates the second depth buffer. Note that the setting of the specific object is performed in mesh units.

[0120] Next, the processor 81 performs ray tracing processing (step S122). The ray tracing processing is performed on each target pixel. Here, the color of the reflection into each target pixel is calculated. Note that the ray tracing processing may be performed for all pixels, or may be performed only for pixels in a specific range. For example, the load may be reduced by excluding a portion where the model is not drawn, a portion where SSR is set to invalid in model or mesh units, a portion where the normal direction is outside a specific range, and the like. The details of the ray tracing processing will be described below.

[0121] (Ray Tracing Processing) FIG. 26 is a flowchart showing an example of the ray tracing processing in step S122.

[0122] As shown in FIG. 26, the processor 81 calculates the position in the virtual space of the target pixel as the reflection position and calculates the reflection direction (step S131). Specifically, the processor 81 calculates the position in the virtual space of the target pixel as the reflection position based on the position of the target pixel in the screen coordinate system and the depth of the target pixel stored in the first depth buffer. Further, the processor 81 sets the direction from the position of the virtual camera VC to the reflection position as the incident direction, and calculates the reflection direction based on the incident direction and the normal direction of the reflection position.

[0123] Next, the processor 81 sets the calculated reflection direction or the corrected direction of the reflection direction as the ray tracing direction (step S132). Here, when the correction condition is satisfied, the corrected direction of the reflection direction is set as the ray tracing direction, and when the correction condition is not satisfied, the reflection direction is set as the ray tracing direction. Also, the ray tracing end position is set. The correction condition and the method of correction are as described above.

[0124] After step S132, the processor 81 sets the first depth buffer as the depth buffer for determination and starts ray tracing from the pixel of interest (step S133). Ray tracing is performed in the screen coordinate system. Next, the processor 81 advances the ray by a predetermined distance (step S134).

[0125] Subsequently, the processor 81 determines whether the trace distance has exceeded the maximum distance Lb (step S135). Specifically, the processor 81 determines whether the tip of the ray in the screen coordinate system has reached the ray tracing end position.

[0126] If it is determined that the trace distance has exceeded Lb (step S135: YES), the processor 81 then performs the process of step S141.

[0127] On the other hand, when the trace distance is less than or equal to Lb (step S135: NO), the processor 81 determines whether the ray has collided with an object (step S136). Specifically, the processor 81 uses the depth buffer for determination to determine whether the tip of the ray has collided with an object.

[0128] If it is determined that the ray has not collided with an object (step S136: NO), the processor 81 determines whether the trace distance is greater than or equal to La (step S137).

[0129] When the trace distance is equal to or greater than La (step S137: YES), the processor 81 changes the determination depth buffer to the second depth buffer (step S138) and executes the process of step S134 again. On the other hand, when the trace distance is less than La (step S137: NO), the processor 81 executes the process of step S134 again while maintaining the determination depth buffer.

[0130] On the other hand, when it is determined that the ray has collided with an object (step S136: YES), the processor 81 calculates the color of the first specular reflection from the color at the collision position of the ray (step S139). For example, the processor 81 may calculate the color of the pixel at the collision position stored in the frame buffer as the color of the first specular reflection, or may calculate the color obtained by applying a predetermined process to the color of the pixel at the collision position as the color of the first specular reflection. The color of the first specular reflection is different between the case where the collision is determined based on the first depth buffer and the case where the collision is determined based on the second depth buffer. For example, when the first depth buffer is set as the depth buffer for determination, depending on the trace distance, the color of the pixel at the collision position may be lightly changed (the alpha value is decreased), and the changed color may be calculated as the color of the first specular reflection. Also, when the second depth buffer is set as the depth buffer for determination, regardless of the trace distance, the color of the pixel at the collision position may be calculated as the color of the first specular reflection. Further, when the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position is lightly changed according to the trace distance, but is changed to be darker than when the first depth buffer is set as the depth buffer for determination, and the changed color may be calculated as the color of the first specular reflection. Then, the processor 81 stores the calculated color of the first specular reflection in the specular buffer. Here, the color of the first specular reflection stored in the specular buffer when it is determined that the ray has collided based on the first depth buffer is denoted as "the color of the first specular reflection (first depth buffer)". Also, the color of the first specular reflection stored in the specular buffer when it is determined that the ray has collided based on the second depth buffer is denoted as "the color of the first specular reflection (second depth buffer)".

[0131] Next, the processor 81 determines whether the depth buffer for determination is the first depth buffer and whether the color of the first reflection calculated in step S139 is translucent (step S140). For example, when the trace distance is less than or equal to La and the color of the first reflection is translucent, further ray tracing is performed deeper. Being translucent means that the alpha value of the reflection is less than 1.0, and the alpha value is uniquely calculated based on the distance La, the collision position, and other various parameters.

[0132] If the determination in step S140 is NO, the processor 81 proceeds to step S141. If the determination in step S140 is YES, the processor 81 proceeds to step S138.

[0133] In step S141, the processor 81 calculates the color of the reflection of the character object (the color of the second reflection). The details of the process of step S141 will be described below.

[0134] FIG. 27 is a flowchart showing an example of the process for calculating the color of the reflection of the character in step S141.

[0135] As shown in FIG. 27, the processor 81 calculates the intersection position of the ray and the character plane (step S151). Specifically, the processor 81 calculates the depth when the character plane is arranged according to the position and orientation of the character object in the virtual space, and determines whether the ray used in the above ray tracing intersects the character plane. When the ray intersects the character plane, the processor 81 calculates the intersection position.

[0136] Next, the processor 81 stores the color of the intersection position in the reflection buffer as the color of the second reflection (step S152). Specifically, the processor 81 stores, as the color of the second reflection, a color based on the color of the pixel of the texture image of the character object corresponding to the intersection position. Note that the color of the second reflection may be the color of the pixel of the texture image of the character object, or may be a color calculated by applying a predetermined process to the color of the pixel. When the processor 81 performs the process of step S152, it ends the process shown in FIG. 27 and returns the process to FIG. 26. When the ray intersects a plurality of character planes, a plurality of colors of the second reflection are stored in the reflection buffer. In this case, the plurality of colors of the second reflection may be stored in the reflection buffer in the order from the intersection position closer to the virtual camera.

[0137] By performing the processes of step S139 and step S152, a plurality of reflection colors from 0 are stored in the reflection buffer. For example, in the reflection buffer, the reflection colors corresponding to the collision position or the intersection position of the ray may be stored in the order from closer to the virtual camera. For example, when it is determined that the ray has collided at a first position where the trace distance is less than La, and it is determined that the ray has collided at a second position where the trace distance is equal to or greater than La and less than Lb, and when the ray intersects the character plane between the first position and the second position, the reflection buffer stores, in the order from closer to the virtual camera, the first reflection color (first depth buffer), the second reflection color, the first reflection color (second depth buffer) (see FIG. 23). Note that when the ray does not collide with any object and does not intersect the character plane, no reflection color is stored in the reflection buffer.

[0138] Returning to FIG. 26, after the process of step S141, the processor 81 calculates the color to be reflected onto the target pixel based on the reflected color stored in the reflection buffer (step S142). When a plurality of reflected colors are stored in the reflection buffer, the processor 81 may calculate the color to be reflected onto the target pixel by alpha-blending the plurality of reflected colors in the order from the farthest to the virtual camera. For example, as shown in FIG. 23, when the first reflected color (first depth buffer), the second reflected color, and the first reflected color (second depth buffer) are stored in the reflection buffer in the order from the nearest to the virtual camera, the processor 81 calculates a color obtained by alpha-blending the second reflected color with the first reflected color (second depth buffer), and further alpha-blends the calculated color with the first reflected color (first depth buffer) to calculate the color to be reflected onto the target pixel. Also, for example, when the second reflected color and the first reflected color are stored in the reflection buffer in the order from the nearest to the virtual camera (that is, when there is no object in front of the character as seen from the virtual camera and there is an object behind the character), the processor 81 may calculate the color to be reflected onto the target pixel by alpha-blending the first reflected color with the second reflected color. For example, when the second reflected color on the front side is opaque, the second reflected color is set as the color to be reflected onto the target pixel. Also, when the second reflected color on the front side is translucent, a color obtained by blending the first reflected color and the second reflected color is calculated as the color to be reflected onto the target pixel. The processor 81 stores the calculated reflected color in the memory.

[0139] When the process of step S142 is performed, the processor 81 ends the process of FIG. 26 and returns the process to FIG. 25.

[0140] Returning to FIG. 25, after step S122, the processor 81 performs rendering to the frame buffer based on the result of the ray tracing process (step S123). For example, the processor 81 adds the reflected color calculated in step S142 to the color of the pixel of interest stored in the frame buffer. Thereby, the reflected color calculated in step S142 is reflected in the color of the pixel of interest.

[0141] Next, the processor 81 determines whether the processes of steps S122 and S123 have been performed for all pixels (step S124). If the processor 81 determines NO in step S124, it re-executes the process of step S122. By performing the processes of steps S122 and S123 for all pixels, the reflections (52, 54) of objects other than the character objects arranged in the virtual space and the reflections (61, 63) of the character objects are drawn in the frame buffer. If the processor 81 determines YES in step S124, it ends the process of FIG. 25 and returns the process to FIG. 24.

[0142] Returning to FIG. 24, after step S105, the processor 81 further renders the character objects 41 and 43 to the frame buffer (step S106).

[0143] Next, the processor 81 outputs the image stored in the frame buffer to the display device (step S107). Thereby, the game image is displayed.

[0144] Next, the processor 81 determines whether to end the game (step S108). For example, when the player instructs the end of the game, the processor 81 ends the game process shown in FIG. 24. If the processor 81 determines not to end the game (step S108: NO), it re-executes the process of step S101. Thus, the description of the main process shown in FIG. 24 ends.

[0145] Note that the order, content, values used for determination, etc. of the processes in the above flowchart are merely examples, and these may be changed as appropriate.

[0146] As described above, in this embodiment, when the trace distance is less than La, ray collision determination is performed based on the first depth buffer, and when the trace distance is La or more, ray collision determination is performed based on the second depth buffer (steps S135 to S140). Thereby, even when the trace distance is long for a specific object, the reflection can be displayed. Also, for objects other than the specific object, the reflection can be prevented from being displayed, and the desired object can be made to reflect or not reflect. Further, the reflection is generated based on the first depth buffer and the second depth buffer, and when based on the first depth buffer, the color of the reflection is made lighter according to the trace distance, so that the specific object can be reflected clearly and largely, and the other objects can be reflected small.

[0147] Also, in this embodiment, when the pixel of interest (reflection position) is close to the edge of the screen, the direction obtained by correcting the reflection direction based on the incident direction and the normal direction at the reflection position is set as the ray tracing direction (step S132). Thereby, the color of the reflection can be calculated based on the pixels drawn in the frame buffer, and the reflection can be generated even at the edge of the screen.

[0148] Also, in this embodiment, objects other than the character object are drawn in the frame buffer first, the reflection of the object is drawn, then the reflection of the character object is drawn, and finally the character object is drawn. Thereby, the reflection of the object behind the character object as seen from the virtual camera can be displayed, and the reflection of the character object can also be displayed.

[0149] Also, in this embodiment, the color of the collision position calculated based on the first depth buffer, the color of the collision position calculated based on the second depth buffer, and the color of the intersection position with the character plane are calculated, and these colors are alpha-blended in order from the back. Thereby, even when objects overlap when viewed from the virtual camera, the reflection of each object can be displayed.

[0150] (Modification example) As described above, the image processing of this embodiment has been described. However, the above embodiment is merely an example, and for example, the following modifications may be added.

[0151] For example, in the above embodiment, deferred rendering is used as the rendering method. However, in other embodiments, forward rendering may be used. When forward rendering is used, in the above SSR process, for example, the normal direction of the reflection position corresponding to the pixel of interest is estimated based on the depths of a plurality of pixels around the pixel of interest, and the reflection direction may be calculated based on the incident direction and the normal direction. Alternatively, the posture of the mesh in the virtual space may be calculated, the normal direction of the reflection position corresponding to the pixel of interest may be calculated, and the reflection direction may be calculated based on the normal direction.

[0152] Also, in the above embodiment, the reflection direction is corrected for the pixels in the region within a predetermined range from the left end or the right end of the screen. However, in other embodiments, the reflection direction may also be corrected for the pixels in the region within a predetermined range from the upper end or the lower end of the screen. In the above embodiment, in the correction of the reflection direction, the position in the Sx-axis direction of the ray tracing end position in the screen coordinate system is made closer to the position in the Sx-axis direction of the reflection position. In other embodiments, the reflection direction may be corrected by moving the position in the Sy-axis direction of the ray tracing end position in the positive or negative Sy-axis direction. The reflection direction may be corrected so that the collision position of the ray falls within the screen by moving the ray tracing end position in the horizontal and / or vertical directions.

[0153] Also, in the above embodiment, ray tracing was performed in the screen coordinate system. However, in other embodiments, ray tracing may be performed in the xyz coordinate system of the virtual space. That is, in the xyz coordinate system of the virtual space, rays may be extended, the collision position of the rays may be calculated, and the color of the pixel at the position corresponding to the collision position may be calculated as the reflected color.

[0154] Also, in the above embodiment, a character plane was created by setting an image of a character object as a texture on a planar object, and the reflected color of the character object was calculated using the character plane. In other embodiments, an image of a character object may be set as a texture on a three-dimensional object, and the reflected color of the character object may be calculated using the three-dimensional object. Also, in the above embodiment, the character object was assumed to be a flat object, but in other embodiments, the character object may be a three-dimensional object having a thickness.

[0155] Also, in the above embodiment, the case where an object is reflected on the ground with the ground as the reflecting surface was described. However, other surfaces (for example, wall surfaces or ceiling surfaces) may be used as the reflecting surface, and the object may be reflected by the above-described processing.

[0156] Also, in the above embodiment, it was assumed that a game image was generated. However, the above-described image processing may be used not only for game images but also when generating any image.

[0157] Also, the above hardware configuration is merely an example, and the above image processing may be performed on any other hardware. For example, the above processing may be executed on any information processing device such as a personal computer, a tablet terminal, a smartphone, a server on the Internet, etc. Further, the above image processing may be executed in an information processing system including a plurality of devices. The plurality of devices may execute the above image processing in a distributed manner. Also, the device that performs the above image processing and the device that displays the image may be different. For example, a first device (e.g., a server) may generate an image by executing part or all of the above image processing, and transmit the generated image to a second device via a network (e.g., the Internet or a LAN), and the second device may display the image.

[0158] Also, the configurations according to the above embodiments and their modifications can be arbitrarily combined as long as they do not contradict each other. Also, the above is merely an exemplification of the present invention, and various improvements and modifications other than the above may be added.

Explanation of Reference Numerals

[0159] 1 Game system 2 Main body device 81 Processor 32 Tree object 34 Cylindrical object 36 Mountain object 41, 43 Character object

Claims

1. The computer of the information processing device About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; causing a frame buffer to perform drawing based on the results of the first depth test; performing a second depth test using a second depth buffer and updating the second depth buffer for a first type of object among the objects; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on a depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than a first distance, and based on a depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; an image processing program that, when the collision position is determined within a range in which the trace distance is within the first distance, determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as a reflection color to be added to the color of the pixel of interest.

2. The computer further comprises:

2. The image processing program according to claim 1, further comprising: determining a reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

3. The computer further comprises: for a second type of object, placing a planar object having a texture set to an image that is rendered without drawing to a frame buffer at a position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color is determined based on the color of the texture at the intersection position to be further added to the color of the pixel of interest; Reflecting the color of the reflection in the frame buffer; The image processing program according to claim 1 , further comprising: drawing the second type of object in a frame buffer after reflecting the color of the reflected light.

4. The computer further comprises: 2 . The image processing program according to claim 1 , further comprising: calculating, as the ray tracing direction, a direction obtained by further correcting a reflection direction based on the incident direction and a normal direction of the reflection position, toward an inner side of the screen.

5. The image processing program according to claim 3 , wherein the second type of object is a flat-shaped object.

6. 1. An image processing system comprising a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on the results of the first depth test; performing a second depth test using a second depth buffer and updating the second depth buffer for the first type of objects among the objects; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, based on the depth of the first depth buffer, calculating a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on a depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than a first distance, and based on a depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the trace distance is within the first distance, an image processing system determines a color based on the color of the pixel in the frame buffer corresponding to the collision position as the reflection color to be added to the color of the pixel of interest.

7. The processor further comprises: The image processing system of claim 6, wherein the reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position is determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

8. The processor further comprises: for a second type of object, a plane object having a texture set to an image that is rendered without drawing to a frame buffer is placed at a position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color is determined based on the color of the texture at the intersection position to be added to the color of the pixel of interest; Reflecting the color of the reflection in the frame buffer; The image processing system according to claim 6 , wherein the second type of object is rendered in a frame buffer after the reflection color is reflected.

9. The processor further comprises: The image processing system according to claim 6 , wherein a reflection direction based on the incident direction and the normal direction of the reflection position is further corrected toward the inside of the screen to calculate the ray tracing direction.

10. The image processing system of claim 8 , wherein the second type of object is a flat shaped object.

11. 1. An image processing method, comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on a result of the first depth test; performing a second depth test using a second depth buffer and updating the second depth buffer for a first type of object among the objects; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on a depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than a first distance, and based on a depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the trace distance is within the first distance, a color based on the color of a pixel in the frame buffer corresponding to the collision position is determined as a reflection color to be added to the color of the pixel of interest.

12. The image processing method according to claim 11, further comprising: determining a reflection color to be added to the color of the pixel of interest based on a color of a pixel in the frame buffer corresponding to the collision position, the reflection color being determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

13. For a second type of object, placing a planar object having a texture set to an image rendered without drawing to a frame buffer at a position of the second type of object in the virtual space; determining, when the ray intersects with the planar object, a reflection color to be added to the color of the pixel of interest based on a color of the texture at the intersection position; reflecting the color of the reflection in the frame buffer; The image processing method according to claim 11 , further comprising: drawing the second type object in a frame buffer after reflecting the color of the reflection.

14. The image processing method according to claim 11 , further comprising: calculating, as the ray tracing direction, a direction obtained by further correcting a reflection direction based on the incident direction and a normal direction of the reflection position toward an inner side of the screen.

15. The image processing method according to claim 13 , wherein the second type of object is a flat-shaped object.

16. An image processing device comprising a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; performing drawing to a frame buffer based on the results of the first depth test; performing a second depth test using a second depth buffer and updating the second depth buffer for the first type of objects among the objects; For each pixel of the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, based on the depth of the first depth buffer, calculating a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on a depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than a first distance, and based on a depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the trace distance is within the first distance, an image processing device determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as a reflection color to be added to the color of the pixel of interest.

17. The processor further comprises: The image processing device according to claim 16, wherein a reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position is determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

18. The processor further comprises: for a second type of object, a plane object having a texture set to an image that is rendered without drawing to a frame buffer is placed at a position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color is determined based on the color of the texture at the intersection position to be added to the color of the pixel of interest; Reflecting the color of the reflection in the frame buffer; The image processing device according to claim 16 , wherein the second type of object is rendered in a frame buffer after the reflection color is reflected.

19. The processor further comprises: The image processing device according to claim 16 , further comprising: a reflection direction based on the incident direction and the normal direction of the reflection position, and a correction made toward the inside of the screen to calculate the ray tracing direction as the ray tracing direction.

20. The image processing device according to claim 18 , wherein the second type of object is a flat-shaped object.

Images