Image processing program, image processing system, image processing method, and image processing apparatus.

Abstract

This provides an image processing program capable of generating reflections from objects located behind other objects. [Solution] An image processing system according to one embodiment renders objects other than characters in a virtual space into a frame buffer, and creates a character plane by setting the image of the character, rendered in a separate buffer, as a texture for a planar object. The image processing system performs ray tracing based on the image drawn into the frame buffer, calculates the color at the collision position between the ray and the object other than the character as the first reflection color, calculates the color at the intersection position between the ray and the character plane as the second reflection color, and adds the colors based on these results to each pixel of the frame buffer.

Description

Technical Field

[0001] The present invention relates to an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of expressing reflection based on a drawn image.

Background Art

[0002] Conventionally, there is a method of expressing reflection by performing processing on a drawn image (for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, it may not be possible to generate a reflection from an object existing in the depth direction of the object on the screen.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of generating a reflection from an object existing in the depth direction of an object.

Means for Solving the Problems

[0006] In order to solve the above problems, the present invention adopts the following configuration.

[0007] (First composition) The first configuration image processing program causes the computer of the information processing device to perform a first depth test and update the first depth buffer using the first depth buffer for objects in the virtual space, excluding objects of the first type, to draw to the frame buffer based on the results of the first depth test, and for objects of the first type, to place a planar object with the rendered image set as a texture without drawing to the frame buffer at the position of the first object in the virtual space. The image processing program causes the computer to, for each pixel in the frame buffer that has been drawn, calculate the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest based on the depth of the first depth buffer, and to determine the direction of the ray that reflects to that position as the reflection position as the ray tracing direction, trace the ray along the ray tracing direction, determine the collision position where the ray collides with an object in the virtual space based on the first depth buffer, and if the collision position is determined within a range where the tracing distance of the ray is within a first distance, determine a reflection color based on the color of the pixel in the frame buffer corresponding to the collision position to be added to the color of the pixel of interest, and if the ray intersects with the planar object, determine a reflection color based on the color of the texture at the intersection position to be added to the color of the pixel of interest. The image processing program causes the computer to reflect the reflection color in the frame buffer, and after reflecting the reflection color, draw the first type of object in the frame buffer.

[0008] According to the above, objects other than the first type are first drawn to the frame buffer, ray tracing is performed, and the color of the pixel at the ray collision position is determined as the reflection color to be added to the color of the pixel of interest. On the other hand, when a planar object with the image of the first type object set as its texture intersects with the ray, the color of the texture at the intersection position is determined as the reflection color to be added to the color of the pixel of interest. This makes it possible to generate reflections of both the first type object and the other object even when another object exists behind the first type object.

[0009] (Second structure) In the second configuration, in the first configuration described above, the first type of object may be a flat-shaped object.

[0010] As described above, even when generating reflections using a planar object with an image of the first type of object set as its texture, the image can be made to look natural because the first type of object itself has a flat shape.

[0011] (The third structure) In the third configuration, in the first or second configuration described above, the first type of object may be a player character object controlled based on an operation input.

[0012] According to the above, even when the first type of object is controlled based on an operation input, it is possible to generate reflections of the first type of object and objects located behind it.

[0013] Furthermore, other embodiments may include an image processing system, an image processing apparatus, or an image processing method that performs the image processing described above. [Effects of the Invention]

[0014] According to the present invention, even when there is another object in the depth direction of the first type of object, it is possible to generate a reflection of the first type of object and the other object.

Brief Description of the Drawings

[0015] [Figure 1] A diagram showing an example of a state in which a left controller 3 and a right controller 4 are attached to the main body device 2 [Figure 2] A block diagram showing an example of the internal configuration of the main body device 2 [Figure 3] A diagram showing an example of a plurality of objects arranged in a virtual space during the execution of the game of this embodiment [Figure 4] A diagram showing an example of a game image displayed on a display device [Figure 5] A diagram showing an outline of image processing related to the reflection of objects other than the character object [Figure 6] A diagram showing an example of the first depth buffer [Figure 7] A diagram showing an example of an image drawn on a frame buffer after rendering by step S2 [Figure 8] A diagram showing an example of the second depth buffer [Figure 9] A diagram showing the state in which ray tracing is performed, and showing the state in which the ray reflected from the reflection position RPa collides with the cylindrical object 34 [Figure 10] A diagram showing the state in which collision determination of the ray reflected from the reflection position RPa is performed using the first depth buffer [Figure 11] A diagram showing the state in which ray tracing is performed, and showing the state in which the ray reflected from the reflection position RPb collides with the tree object 32 [Figure 12] A diagram showing the state in which ray tracing is performed, and showing the state in which the ray reflected from the reflection position RPc does not collide with an object [Figure 13] [[ID=4"]] [Figure 14] ​A diagram showing an example of an image drawn on a frame buffer when colors reflected on each pixel are added [Figure 15] A diagram showing an example of a ray tracing direction with corrected reflection direction [Figure 16] A diagram for explaining details of correction with respect to the reflection direction [Figure 17] A diagram showing an example of a game image when the character object 41 does not exist in front of the cylindrical object 34 [Figure 18] A diagram showing an example of a game image when the character object 41 moves and the character object 41 exists in front of the cylindrical object 34 [Figure 19] A diagram showing an outline of image processing for generating reflections of a character object and objects behind it [Figure 20] A diagram showing an example of a character plane created in step S11 [Figure 21] A diagram for explaining the calculation of the reflected color of a character in step S13 [Figure 22] A diagram showing an example of a game image displayed when the image processing of FIG. 19 is performed [Figure 23] A diagram showing an example of data stored in the memory of the main body device 2 [Figure 24] A flowchart showing an example of the main processing executed by the processor 81 of the main body device 2 [Figure 25] A flowchart showing an example of the SSR processing in step S105 [Figure 26] A flowchart showing an example of the ray tracing processing in step S122 [Figure 27] A flowchart showing an example of the calculation process of the reflected color of a character in step S141

Embodiments for Carrying Out the Invention

[0016] (System Configuration) The following describes a game system according to an example of this embodiment. An example of the game system 1 in this embodiment includes a main unit (information processing device; functioning as the game device main unit in this embodiment) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are detachable from the main unit 2. In other words, the game system 1 can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main unit 2. Alternatively, the game system 1 can be used with the main unit 2 and the left controller 3 and right controller 4 as separate components. The hardware configuration of the game system 1 in this embodiment will be described below, followed by a description of the control of the game system 1 in this embodiment.

[0017] Figure 1 shows an example of the main unit 2 with the left controller 3 and right controller 4 attached. As shown in Figure 1, the left controller 3 and right controller 4 are attached to the main unit 2 and integrated together. The main unit 2 is a device that performs various processes (e.g., game processing) in the game system 1. The main unit 2 is equipped with a display 12. The left controller 3 and right controller 4 are devices equipped with operation parts for user input.

[0018] The left controller 3 and the right controller 4 are detachable from the main unit 2. In the following, the left controller 3 and the right controller 4 will be collectively referred to as "controllers."

[0019] The main unit 2 alone, or the integrated unit in which the left controller 3 and right controller 4 are attached to the main unit 2, may be a portable device. Furthermore, the main unit 2 or the integrated unit may be a handheld device. Also, the main unit 2 or the integrated unit may be a portable device.

[0020] Furthermore, the main unit 2 is equipped with a touch panel 13 on the screen of the display 12. In this embodiment, the touch panel 13 is of a type that allows multi-touch input (for example, a capacitive touch panel). However, the touch panel 13 may be of any type, for example, a type that allows single-touch input (for example, a resistive touch panel).

[0021] Figure 2 is a block diagram showing an example of the internal configuration of the main unit 2.

[0022] The main unit 2 includes a processor 81. The processor 81 is an information processing unit that performs various information processing operations in the main unit 2, and includes one or more CPUs (Central Processing Units) and one or more GPUs (Graphics Processing Units). The processor 81 may be composed of a System-on-a-chip (SoC) that includes multiple functions such as CPU functions and GPU functions. The CPU and GPU may be configured as separate processors. The processor 81 is also provided with one or more memories for temporarily storing data. The processor 81 performs various information processing operations by executing information processing programs (for example, game programs) stored in a storage unit (specifically, an internal storage medium such as flash memory 84, or an external storage medium installed in slot 23).

[0023] The main unit 2 includes, as an example of an internal storage medium built into itself, a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85. The flash memory 84 and DRAM 85 are connected to the processor 81. The flash memory 84 is a memory mainly used to store various types of data (which may be programs) stored in the main unit 2. The DRAM 85 is a memory used to temporarily store various types of data used in information processing.

[0024] The main unit 2 is equipped with a slot interface (hereinafter abbreviated as "I / F") 91. The slot I / F 91 is connected to the processor 81. The slot I / F 91 is connected to slot 23 and reads and writes data to a predetermined type of storage medium (for example, a dedicated memory card) installed in slot 23, according to instructions from the processor 81.

[0025] The processor 81 performs the above-mentioned information processing by appropriately reading and writing data to and from the flash memory 84 and DRAM 85, as well as to each of the above-mentioned storage media.

[0026] The main unit 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates with external devices via a network (specifically, wirelessly). In this embodiment, the network communication unit 82 communicates with external devices by connecting to a wireless LAN using a method compliant with the Wi-Fi standard as a first communication mode. The network communication unit 82 also communicates wirelessly with other main units 2 of the same type using a predetermined communication method (for example, communication using a proprietary protocol or infrared communication) as a second communication mode.

[0027] The main unit 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 communicates wirelessly with the left controller 3 and / or the right controller 4. The communication method between the main unit 2 and the left controller 3 and the right controller 4 is arbitrary, but in this embodiment, the controller communication unit 83 communicates with the left controller 3 and with the right controller 4 in accordance with the Bluetooth® standard.

[0028] The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 communicates with the left controller 3 via a wired connection, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. When the processor 81 communicates with the right controller 4 via a wired connection, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. When the processor 81 communicates with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in this embodiment, the main unit 2 can perform both wired and wireless communication with the left controller 3 and the right controller 4, respectively. Furthermore, when the left controller 3 and the right controller 4 are mounted on the main unit 2 as an integrated unit, or when the main unit 2 alone is mounted on the cradle, the main unit 2 can output data (e.g., image data and audio data) to a stationary monitor or the like via the cradle.

[0029] The main unit 2 includes a touch panel controller 86, which is a circuit that controls the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. Based on signals from the touch panel 13, the touch panel controller 86 generates data indicating, for example, the position where a touch input occurred, and outputs it to the processor 81.

[0030] The main unit 2 comprises a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Although not shown in the figures, the power control unit 97 is also connected to various parts of the main unit 2 (specifically, the parts that receive power from the battery 98, the left terminal 17, and the right terminal 21). Based on commands from the processor 81, the power control unit 97 controls the power supply from the battery 98 to the aforementioned parts.

[0031] The battery 98 is also connected to the lower terminal 27. When an external charging device (for example, a cradle) is connected to the lower terminal 27 and power is supplied to the main unit 2 via the lower terminal 27, the supplied power charges the battery 98.

[0032] (Overview of image processing) Next, the image processing of this embodiment will be described. In the game system 1 of this embodiment (an example of an image processing system), multiple objects are placed in a three-dimensional virtual space, and the game is played.

[0033] Figure 3 shows an example of multiple objects placed in the virtual space during the execution of the game of this embodiment. When the game of this embodiment starts, a three-dimensional virtual space defined by the xyz Cartesian coordinate system is set up. For example, the y-axis is the upward axis of the virtual space, the x-axis is the rightward axis, and the z-axis is the backward axis.

[0034] As shown in Figure 3, multiple objects, including a ground object 30, are placed in the virtual space. The ground object 30 is an object that mimics the ground, for example, a plane parallel to the xz plane. The ground object 30 may have irregularities or slopes. A tree object 32 and a cylinder object 34 are placed on the ground object 30. In addition, for example, a mountain object 36 is placed at a predetermined position in the z-axis direction of the virtual space. These ground objects 30, tree objects 32, and cylinder objects 34 are fixed in the virtual space.

[0035] Furthermore, character objects 41 and 43 are placed in the virtual space. Character objects 41 and 43 are flat objects, plate-like objects in which the length in the thickness direction is smaller than the length in the up, down, left, and right directions. For example, character objects 41 and 43 are 3D objects that include a planar mesh forming the front and a planar mesh forming the back, respectively. Character object 41 is a player character controlled by the player. In the virtual space, character object 41 performs actions in response to input to the controller. For example, as actions, character object 41 may move on the ground object 30, move its limbs in the virtual space, jump, or change the direction of its face. When character object 41 moves left or right, the screen is scrolled left or right. Character object 43 is a non-player character that moves in response to the movement of character object 41 and is automatically controlled by processor 81. Character object 43 may also be controlled by the player.

[0036] A virtual camera is placed in the virtual space, and a game image is generated from the virtual camera's perspective of the virtual space. This generated game image is then displayed on the display 12 or a stationary monitor (hereinafter referred to as the "display device").

[0037] Figure 4 shows an example of a game image displayed on a display device. The display device shows images of each object (30, 32, 34, 36, 41, 43). For example, the ground object 30 is a reflective surface, and each object is reflected on the ground object 30. In this embodiment, the reflections of each object are generated using a method called SSR (Screen Space Reflection).

[0038] For example, as shown in Figure 4, the reflection 52 of the tree object 32 and the reflection 54 of the cylinder object 34 are displayed on the ground object 30. Also, the reflection 61 of the character object 41 and the reflection 63 of the character object 43 are displayed on the ground object 30. On the other hand, the reflection of the mountain object 36 is not displayed on the ground object 30.

[0039] In this embodiment, the tree object 32 and the cylinder object 34 are pre-configured as specific objects and are set to appear on the ground object even when they are a certain distance away from the virtual camera. On the other hand, the mountain object 36 is not configured as a specific object.

[0040] A specific object is defined on a mesh (a collection of polygons) basis. For example, each of the multiple meshes that make up the tree object 32 may be defined as a specific object. Alternatively, any of the multiple meshes that make up the tree object 32 may be defined as a specific object. For example, if the tree object 32 is composed of a first mesh that makes up the trunk and a second mesh that makes up the leafy part, only the first mesh may be defined as a specific object. In other words, a "specific object" does not necessarily have to be an object that appears as a single object visually, but may be a mesh that makes up part of an object that appears as a single object visually. For example, a terrain object in which a ground surface parallel to the xz plane and a wall surface perpendicular to the xz plane are continuously formed may appear to form a single terrain with steps, but the ground surface and the wall surface may be composed of different meshes. In such a terrain object, the wall surface may be defined as a specific object, while the ground surface does not have to be defined as a specific object.

[0041] In this embodiment, the tree object 32 and the cylinder object 34, which are set as specific objects, are reflected on the ground object. The image processing related to the reflection of these objects will be described below.

[0042] Figure 5 shows an overview of image processing related to the reflection of objects other than character objects.

[0043] As shown in Figure 5, the processor 81 first performs a first depth test (step S1). Here, a depth test is performed on a pixel-by-pixel basis for all objects other than character objects (30, 32, 34, 36), and the results of the depth test are stored in the first depth buffer.

[0044] Figure 6 shows an example of the first depth buffer. As shown in Figure 6, the first depth buffer stores the pixel-by-pixel depth values ​​for all objects other than character objects (30, 32, 34, 36). In Figure 6, images of each object are displayed, which conceptually represent the depth values ​​of each pixel stored in the first depth buffer. The position of each pixel is expressed by coordinate values ​​in the screen coordinate system. For example, the Sx axis of the screen coordinate system is the axis to the right in the image (screen), and the Sy axis is the axis upward in the image. The origin of the screen coordinate system may be set, for example, at the center of the image, or at the bottom left of the image.

[0045] Returning to Figure 5, after step S1, the processor 81 renders to the frame buffer based on the results of the first depth test (step S2). In step S2, the image is drawn to the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering (also called deferred shading) is used as the rendering method. In step S2, the normal buffer, which stores the normal information for each pixel, is updated, and the image is drawn to the frame buffer. Note that forward rendering may be used instead of deferred rendering as the rendering method.

[0046] Figure 7 shows an example of an image drawn to the frame buffer after rendering in step S2. As shown in Figure 7, in step S2, images of all objects other than character objects (30, 32, 34, 36) are drawn to the frame buffer. Color information is stored for each pixel. The color information includes, for example, RGB values ​​representing three colors and an alpha value representing transparency (or opacity). Although shading is omitted in Figure 7, an image with shading is generated in step S2.

[0047] Returning to Figure 5, after step S2, the processor 81 performs a second depth test on specific objects among the multiple objects placed in the virtual space (step S3). Specifically, the second depth test is performed on the tree object 32 and the cylinder object 34, which are set as specific objects among the multiple objects (30, 32, 34, 36), and the results of the second depth test are stored in the second depth buffer. As a result, the second depth buffer stores the pixel-by-pixel depth values ​​only for the tree object 32 and the cylinder object 34 among the multiple objects (30, 32, 34, 36) included in the imaging range of the virtual camera.

[0048] Figure 8 shows an example of a second depth buffer. As shown in Figure 8, the second depth buffer stores the depth values ​​for each pixel of the tree object 32 and the cylinder object 34. On the other hand, the depth value of the mountain object 36 is not stored in the second depth buffer. Figure 8 conceptually represents the depth values ​​of each pixel stored in the second depth buffer.

[0049] Returning to Figure 5, after step S3, the processor 81 performs ray tracing using the first or second depth buffer (step S4). Here, processing is performed for each pixel (hereinafter referred to as the "focus pixel"). Specifically, the processor 81 sets the focus pixel, calculates the reflection position of a ray (a virtual ray of light) cast from the virtual camera's position toward the focus pixel, and performs a collision determination to determine whether the ray reflected from that reflection position collides with an object.

[0050] Next, the processor 81 calculates the color of the pixel in the frame buffer corresponding to the ray collision position as the color reflected on the pixel of interest (step S5). The process in step S5 is also performed for each pixel.

[0051] The following describes in detail the processing of steps S4 and S5 when each pixel PIXa to PIXc shown in Figure 7 is set as the pixel of interest.

[0052] For example, the processor 81 sets the pixel PIXa shown in Figure 7 as the pixel of interest. Pixel PIXa is a pixel that is closer to the virtual camera than the cylindrical object 34. Based on the position of the pixel of interest PIXa in the screen coordinate system and the depth value of that position stored in the first depth buffer, the processor 81 calculates the position of the pixel of interest PIXa in virtual space as the ray reflection position RPa.

[0053] Figure 9 shows how ray tracing is performed, illustrating how a ray reflected from reflection point RPa collides with the cylindrical object 34. The rightward direction in Figure 9 represents the depth of the screen when the game image generated based on the virtual camera VC is displayed on the screen.

[0054] As shown in Figure 9, the processor 81 defines the direction from the virtual camera VC towards the reflection position RPa as the incident direction, and calculates the direction of the ray reflected from the reflection position RPa as the ray tracing direction RTDa. Specifically, the processor 81 calculates the reflection direction of the ray based on the incident direction and the normal to the reflection position RPa. The angle of incidence in the incident direction is equal to the angle of reflection in the reflection direction. The processor 81 then calculates the reflection direction, or a direction corrected from the reflection direction, as the ray tracing direction RTDa. The correction of the reflection direction will be explained later.

[0055] Processor 81 propagates a ray from the reflection position RPa in the ray tracing direction RTDa and performs a collision determination to determine whether the ray collides with an object. If the trace distance of the ray is less than La, processor 81 performs the collision determination using the first depth buffer. Here, the trace distance is the distance from the reflection position along the ray tracing direction.

[0056] Figure 10 shows how collision detection of a ray reflected from the reflection position RPa is performed using the first depth buffer. First, the processor 81 calculates the position in the virtual space by traveling the maximum distance Lb in the ray tracing direction RTDa from the reflection position RPa in the virtual space, and then converts the calculated position to the screen coordinate system to calculate the ray tracing end position in the screen coordinate system. In the screen coordinate system, the processor 81 uses the reflection position RPa (the pixel of interest RPa) as the ray tracing start position, extends the ray from the ray tracing start position to the ray tracing end position, and determines whether the tip of the ray collides with an object. Specifically, the processor 81 advances the ray a predetermined distance and determines whether the ray collides with an object based on the depth of the tip of the ray and the depth of each position stored in the first depth buffer. In the example shown in Figures 9 and 10, it is determined that the ray reflected at the reflection position RPa collides at the collision position CPa.

[0057] If it is determined that the ray collides with an object at collision position CPa, in step S5, the color of the pixel in the frame buffer corresponding to the collision position CPa is calculated as the color reflected in the pixel of interest RPa.

[0058] Next, we will explain the cases where pixel PIXb, shown in Figure 7, is set as the pixel of interest, and where pixel PIXc is set as the pixel of interest. Pixel PIXb is a pixel that is closer to the virtual camera than the tree object 32. Pixel PIXc is a pixel that is closer to the virtual camera than the mountain object 36.

[0059] Figure 11 shows how ray tracing is performed, illustrating how a ray reflected from reflection position RPb collides with tree object 32. Figure 12 shows how ray tracing is performed, illustrating how a ray reflected from reflection position RPc does not collide with the object. Figure 13 shows how ray collision detection is performed using the second depth buffer.

[0060] As shown in Figure 11, when pixel PIXb is set as the pixel of interest, the processor 81 calculates the reflection position RPb corresponding to the pixel of interest PIXb, as described above, and calculates the direction of the ray reflected from the reflection position RPb as the ray tracing direction RTDb. The processor 81 propagates the ray in the ray tracing direction RTDb and performs a collision determination to determine whether the ray collides with an object. When the trace distance is less than La, the processor 81 performs a collision determination of the ray using the first depth buffer, but when the trace distance is less than La, the ray does not collide with an object. Therefore, the processor 81 propagates the ray further and performs a collision determination of the ray using the second depth buffer (Figure 13). As shown in Figures 11 and 13, when the trace distance is greater than or equal to La and less than or equal to the maximum distance Lb, it is determined that the ray collides with the tree object 32 at the collision position CPb.

[0061] On the other hand, as shown in Figure 12, when the processor 81 sets pixel PIXc as the pixel of interest, it calculates the reflection position RPc corresponding to the pixel of interest PIXc, calculates the direction of the ray reflected from the reflection position RPc as the ray tracing direction RTDc, and propagates the ray in the ray tracing direction RTDb. When the trace distance is less than La, the processor 81 uses the first depth buffer to determine the ray collision, and when the trace distance is La or greater and less than or equal to the maximum distance Lb, it uses the second depth buffer to determine the ray collision. Here, as shown in Figure 13, the depth of the mountain object 36 is not stored in the second depth buffer. Therefore, as shown in Figures 12 and 13, the processor 81 determines that even if the ray is propagated to the maximum distance Lb, the ray will not collide with the object.

[0062] Furthermore, the parameters used to add the color of the pixel at the collision point to the color of the pixel of interest may differ depending on whether the ray collision point is calculated based on the first depth buffer or the second depth buffer. For example, when the collision point is calculated based on the second depth buffer, the parameters may be changed so that the reflected color becomes darker than when the collision point is calculated based on the first depth buffer.

[0063] The processing described in steps S4 and S5 above is performed for each pixel of interest, thereby determining the color reflected in each pixel of interest. Then, by adding the determined reflected color to the color of each pixel of interest, the reflection of each object is drawn to the frame buffer.

[0064] Figure 14 shows an example of an image drawn to the frame buffer when the color reflected in each pixel is added.

[0065] As shown in Figure 14, the color of the pixel at the collision position CPa, calculated as described above, is added to the pixel RPa. The same process is performed on each pixel in the area in front of the cylinder object 34 (the area on the virtual camera side), so that the reflection 54 of the cylinder object 34 is drawn to the frame buffer. In addition, the color of the pixel at the collision position CPb is added to the pixel RPb. The same process is performed on each pixel in the area in front of the tree object 32, so that the reflection 52 of the tree object 32 is drawn to the frame buffer. On the other hand, the collision position is not calculated for the pixel RPc. The same applies when processing is performed on each pixel in the area in front of the mountain object 36. For this reason, the reflection of the mountain object 36 is not drawn to the frame buffer.

[0066] In this embodiment, the results of a first depth test performed on multiple objects are stored in a first depth buffer, and the results of a second depth test performed on a specific object among the multiple objects are stored in a second depth buffer. When the trace distance is less than La, ray collision detection is performed based on the first depth buffer, and when the trace distance is La or greater and less than or equal to the maximum distance Lb, ray collision detection is performed based on the second depth buffer. This makes it possible to use SSR to display reflections of a specific object among multiple objects, even when it is far from the virtual camera, while preventing reflections from being displayed for objects other than the specific object. By designating the object to be reflected as a specific object, the desired object can be reflected, for example, on the ground. Conversely, by not designating the object that you do not want to be reflected as a specific object, the desired object can be prevented from being reflected.

[0067] Furthermore, the parameters used to reflect the color of the pixel at the collision point onto the pixel of interest are different depending on whether the ray collision point is calculated based on the first depth buffer or the second depth buffer. This allows, for example, a specific object to be reflected more intensely and clearly.

[0068] (Correction of reflection direction) Next, the correction of the reflection direction will be explained. As described above, the direction from the virtual camera VC toward the reflection position is calculated as the incident direction, and the reflection direction is calculated based on this incident direction and the normal direction of the reflection position. In this embodiment, if the correction conditions are met, the direction in which the reflection direction has been corrected is set as the ray tracing direction. If the correction conditions are not met, the reflection direction is set as the ray tracing direction. The correction conditions for correcting the reflection direction and the method of correction will be explained in detail below.

[0069] Figure 15 shows an example of a ray tracing direction after correction of the reflection direction. As shown in Figure 15, for example, when ray tracing is performed on a pixel of interest near the left edge of the screen (reflection position RP1), the reflection direction RD1, which is calculated based on the incident direction and the normal direction, is corrected, and the corrected direction is set as the ray tracing direction RTD1. Similarly, when ray tracing is performed on a pixel of interest to the right of the screen than reflection position RP1 (reflection position RP2), the reflection direction RD2 is corrected, and the corrected direction is set as the ray tracing direction RTD2.

[0070] The correction conditions are met when both the conditions relating to the reflection position and the conditions relating to the reflection direction are satisfied. The condition relating to the reflection position is that the reflection position is within a predetermined range from the edge of the screen. The condition relating to the reflection direction is that the reflection direction is directed outwards from the screen. If either the condition relating to the reflection position or the condition relating to the reflection direction is not satisfied, the correction conditions are not satisfied, and the reflection direction is not corrected. For example, the correction conditions are satisfied if the reflection position is within a range from the left edge of the screen to 1 / 4 of the screen's horizontal length, and the reflection direction is directed to the left of the screen. Also, the correction conditions are satisfied if the reflection position is within a range from the right edge of the screen to 1 / 4 of the screen's length, and the reflection direction is directed to the right of the screen.

[0071] The degree of correction is greater at reflection position RP1, which is closer to the left edge of the screen, than at reflection position RP2. Specifically, the degree of correction CR is determined based on the position of the reflection position in the Sx axis direction of the screen coordinate system. The closer the position of the reflection position in the Sx axis direction is to the left or right edge of the screen, the greater the degree of correction CR. Also, the larger the angle between the reflection direction in the screen coordinate system and the Sy axis direction, the smaller the degree of correction CR.

[0072] Figure 16 is a diagram illustrating the details of the correction for the reflection direction. As shown in Figure 16, for example, if the angle between the reflection direction RD and the upward direction of the screen (Sy axis direction) is 45 degrees or less, the degree of correction CR is set to a value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP. CR(Sx) is set in the range of 0 to 1, for example, and becomes larger the closer the reflection position is to the left or right edge of the screen. If the angle between the reflection direction RD and the upward direction of the screen exceeds 45 degrees, the value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP decreases according to the angle exceeding 45 degrees. If the angle between the reflection direction RD and the upward direction of the screen exceeds a threshold (for example, 60 to 65 degrees), the degree of correction CR is set to "0".

[0073] Depending on the degree of correction (CR), a correction is applied in the direction of reflection, directed inward from the screen. Specifically, the position obtained by converting the maximum distance Lb from the reflection position RP in the reflection direction RD in the virtual space to the screen coordinate system is calculated as the ray tracing end position EP. The Sx coordinate value of this ray tracing end position EP is corrected to move inward from the screen, depending on the degree of correction (CR). For example, linear interpolation is used to correct the Sx coordinate value of the ray tracing end position EP to approach the Sx coordinate value of the reflection position RP. For example, if the reflection position RP is located within a predetermined range from the left edge of the screen, the ray tracing end position EP is moved to the right (positive Sx axis direction) depending on the degree of correction (CR). The Sx coordinate value of the ray tracing end position EP is not moved beyond the Sx coordinate value of the reflection position RP. Also, the Sy coordinate value of the ray tracing end position EP does not change before and after the correction. The direction from the reflection position EP to the moved ray tracing end position EP' is calculated as the ray tracing direction (RTD).

[0074] Thus, in areas near the edges of the screen, the direction corrected for the reflection direction is calculated as the ray tracing direction. This makes it possible to generate reflections using SSR at the edges of the screen. For example, as shown in Figure 15, if ray tracing is performed in the reflection direction RD1 for a pixel at position RP1 on the left edge of the screen, the ray collision point may be outside the screen, and the color of the pixel at the collision point cannot be added to the pixel at position RP1. Therefore, if no correction is performed for the reflection direction, the edges of the screen may have a different color than other parts, resulting in an unnatural-looking image.

[0075] In this embodiment, the reflection direction is corrected based on the incident direction and the normal direction, and the corrected direction is set as the ray tracing direction. This makes it easier to keep the ray collision position within the screen and allows for the generation of reflections based on the rendered image. The Sx coordinate value of the ray tracing end position EP is moved to approach the Sx coordinate value of the reflection position RP. Therefore, even with maximum correction, the reflection direction that points outward from the screen will not point inward from the screen, but rather upward from the screen. Consequently, for example, the color of pixels to the right of the left edge of the screen will not be reflected in the pixels at that position. This prevents objects from being unnaturally distorted and reflected on the ground.

[0076] Furthermore, in this embodiment, the degree of correction increases as the reflection position gets closer to the edge of the screen. Therefore, even near the edge of the screen, reflections can be generated based on pixels within the screen.

[0077] Furthermore, for example, if the reflection direction is directed inward from the screen, correcting the reflection direction may result in a reflection that is far removed from the actual reflection. In this embodiment, if the reflection direction is not directed outward from the screen, no correction of the reflection direction is performed, thus preventing the generation of a reflection that is far removed from the actual reflection.

[0078] Furthermore, when the angle between the reflection direction and the top of the screen is relatively large (for example, exceeding 45 degrees), the degree of correction (CR) decreases. When the angle between the reflection direction and the top of the screen exceeds a threshold, the degree of correction becomes 0. This prevents unnatural reflections from being generated due to excessive correction when, for example, the ground is sloped. It also suppresses correction for reflections on walls where the reflection direction is horizontal.

[0079] Furthermore, in this embodiment, the correction of the reflection direction is performed only for the left-right component in the screen coordinate system. For this reason, it is suitable for games in which the virtual space expands horizontally and is scrolled horizontally, for example.

[0080] (Generating reflections of character objects) Next, we will explain how to generate reflections of character objects 41 and 43. Figure 17 shows an example of a game image when character object 41 is not in front of cylindrical object 34. Figure 18 shows an example of a game image when character object 41 has moved and is in front of cylindrical object 34.

[0081] As shown in Figure 17, if there is no character object 41 in front of the cylindrical object 34, for example, ray tracing is performed from the reflection position RPd on the front side to calculate the collision position CPd, and the color of the collision position CPd is added to the color of the reflection position RPd. By performing the same process for each pixel, the reflection 54 of the cylindrical object 34 is displayed.

[0082] On the other hand, in Figure 18, the character object 41 is located in front of the cylindrical object 34, and the head of the character object 41 is located at the same position CPd as in Figure 17. In this case, the pixel at the reflection position RPd cannot have the color of the pixel at the collision position CPd of the cylindrical object 34 shown in Figure 17, and part of the cylindrical object 34 is not reflected at the reflection position RPd. Thus, for example, when the character object 41 is located in front of the cylindrical object 34, it may not be possible to represent a natural reflection that corresponds to the actual positional relationship of each object.

[0083] In this embodiment, image processing is performed in the manner shown in Figure 19 to display the reflections of the character object and the object behind it. Figure 19 is a diagram illustrating the overview of the image processing for generating the reflections of the character object and the object behind it.

[0084] As shown in Figure 19, the processor 81 first renders objects other than character objects (step S10). Here, the process of drawing multiple objects 30, 32, 34, and 36 placed in the virtual space to the frame buffer is performed. The process in step S10 is the same as the process in steps S1 to S2 in Figure 5.

[0085] Next, the processor 81 renders the character object into a separate buffer and creates a character plane with the rendered image set as a texture (step S11).

[0086] Figure 20 shows an example of a character plane created in step S11. As shown in Figure 20, the character object 41 is rendered to a separate buffer, and the rendered image of the character object 41 is set as a texture on a two-dimensional plane object to create the character plane 65. Similarly, a character plane 66 is created by setting the rendered image of the character object 43 as a texture. In this embodiment, an image of each character object is generated every frame. Note that the rendered images of each character object may be prepared in advance. For example, images corresponding to each action of the character object 41 may be stored in advance on a storage medium, and a character plane 65 may be created by setting these pre-stored images as textures.

[0087] Next, the processor 81 calculates the reflection colors of objects other than character objects (step S12). Here, the reflection colors are calculated for multiple objects 30, 32, 34, and 36 placed in the virtual space. Specifically, the processor 81 performs the processing of steps S3 to S5 in Figure 5 described above. At this point, since the character objects have not yet been drawn to the frame buffer, the reflection color of the character object can be calculated even if the character object is located in front of the other objects as seen from the virtual camera VC in the virtual space. The calculated reflection colors of objects other than character objects (hereinafter referred to as "first reflection colors") are stored in the reflection buffer. For example, the reflection buffer stores RGB values ​​and alpha values ​​as color information indicating the first reflection color.

[0088] Next, the processor 81 calculates the color of the character's reflection (step S13). Here, the processor 81 calculates the color of the character object's reflection when the character plane created in step S11 is placed in the virtual space.

[0089] Figure 21 is a diagram illustrating the calculation of the color of the character reflection in step S13. As shown in Figure 21, the processor 81 calculates the intersection position of the character plane 65 and the ray when the character plane 65 is placed in the virtual space according to the position and orientation of the character object 41 in the virtual space. Specifically, the processor 81 calculates the depth when the character plane 65 is placed in the virtual space and calculates the intersection position CRPe of the ray from the reflection position RPe toward the ray tracing direction RTDe and the character plane 65. The color of the pixel at the intersection position CRPe is calculated as the color of the character reflection (hereinafter referred to as the "second reflection color"). Here, the part of the character plane 65 other than the image of the character object 41 is set to transparent and does not intersect with the ray. For example, a ray from the reflection position RPf toward the ray tracing direction RTDf does not intersect with the image of the character object 41 in the character plane 65. In this case, the intersection position is not calculated. The calculated second reflection color is stored in the reflection buffer. For example, the reflection buffer stores RGB values ​​and alpha values ​​as color information indicating the color of the second reflection.

[0090] Returning to Figure 19, the processor 81 then calculates the color reflected in the pixel of interest based on the results of step S12 and step S13 (step S14). For example, the color reflected in the pixel of interest is calculated by adding the second reflected color to the first reflected color. Here, "adding the second color to the first color" may also mean mixing the two colors. For example, "adding the second color to the first color" may mean performing an alpha blend with the first color as the background color and the second color as the foreground color. Alternatively, "adding the second color to the first color" may mean performing an alpha blend with the second color as the background color and the first color as the foreground color. Furthermore, "adding the second color to the first color" may mean mixing the first and second colors based on their respective alpha values. Also, "adding the second color to the first color" may mean that the second color takes precedence over the first color and the second color is overpainted onto the first color.

[0091] Next, processor 81 renders the result calculated in step S14 into the frame buffer (step S15). Here, processor 81 adds the reflected color calculated in step S14 to the color of the pixel of interest stored in the frame buffer. For example, processor 81 performs alpha blending using the color of the pixel of interest as the background color and the reflected color calculated in step S14 as the foreground color. As a result, the reflections of the character object and other objects are drawn into the frame buffer.

[0092] After the processing in step S15 is completed, the processor 81 renders the character objects 41 and 43 into the frame buffer (step S16). This generates a game image as shown in Figure 22, which is then displayed on the display device.

[0093] Figure 22 shows an example of a game image displayed after the image processing shown in Figure 19 has been performed. As shown in Figure 22, a tree object 32 and its reflection 52 are displayed. A character object 43 and its reflection 63 are also displayed. A cylinder object 34 and its reflection 54 are also displayed. A character object 41 is located in front of the cylinder object 34, and the character object 41 and its reflection 61 are displayed. The reflection 54 of the cylinder object 34 and the reflection 61 of the character object 41 overlap in part, but both have shapes that correspond to their actual shapes and relative positions, allowing for a natural representation of reflections.

[0094] In this embodiment, the character object is a flat object. As described above, when generating a reflection of the character object based on a character plane with the character object's image set as a texture, the reflection of the character object will appear flat. If the character object itself is a three-dimensional object with a certain thickness, its reflection appearing flat may result in an unnatural image. However, in this embodiment, since the character object itself is a flat object, an image without unnaturalness can be achieved.

[0095] (Image processing details) Next, we will explain the details of the image processing described above. First, we will explain the data stored in the memory of the main unit 2 (memory in the processor 81, DRAM 85, flash memory 84, or external storage medium, etc.). Figure 23 is a diagram showing an example of the data stored in the memory of the main unit 2.

[0096] As shown in Figure 23, the memory of the main unit 2 stores the program, operation data, object data, character data, and character plane data. The memory of the main unit 2 also stores the first depth buffer, the second depth buffer, the normal buffer, the reflection buffer, and the frame buffer.

[0097] The program is for executing the main processing described later and includes an image processing program for performing the image processing related to reflections mentioned above. The program is pre-stored in an external storage medium or flash memory 84 installed in slot 23 and is loaded into DRAM 85 when the game is run. The program may also be obtained from other devices via a network (e.g., the Internet).

[0098] Operation data is data related to operations acquired from the left controller 3 and the right controller 4. For example, operation data is transmitted from the left controller 3 and the right controller 4 to the main unit 2 at predetermined time intervals (e.g., 1 / 200 second intervals) and stored in memory.

[0099] Object data pertains to each object (30, 32, 34, 36) placed in the virtual space, excluding character objects. Object data includes data on each object's position and orientation in the virtual space, as well as data on its shape and texture. Furthermore, object data includes information on whether each object is designated as a specific object. For example, tree object 32 and cylinder object 34 are designated as specific objects, while mountain object 36 is not.

[0100] The character data pertains to character objects 41 and 43. The character data includes data relating to the position and orientation of each character in virtual space, data relating to its shape, and data relating to its texture. In this embodiment, each character object is formed as a 3D model, but is formed as a planar object that is thinner in the thickness direction than in the up, down, left, and right directions.

[0101] Character plane data is a two-dimensional object, and it is data relating to a plane object for which the rendered image of a character object has been set as a texture. The character plane data includes data relating to character plane 65 for which the image of character object 41 has been set as a texture, and data relating to character plane 66 for which the image of character object 43 has been set as a texture.

[0102] The first depth buffer is a buffer that stores the depth values ​​for each pixel, which are generated and updated based on the results of the first depth test. The first depth buffer stores the depth values ​​of all objects other than character objects that are included in the imaging range of the virtual camera VC.

[0103] The second depth buffer is a buffer that stores the depth values ​​for each pixel, which are generated and updated based on the results of the second depth test. The second depth buffer stores the depth values ​​of specific objects included in the imaging range of the virtual camera VC.

[0104] The normal buffer is a buffer that stores the normal information of each pixel.

[0105] The reflection buffer is a buffer for storing the first reflection color (the reflection color of objects other than character objects) and the second reflection color (the reflection color of character objects). In the ray tracing process described later, the reflection buffer stores the first reflection color when a ray collides with an object other than a character object, and stores the second reflection color when a ray intersects with the character plane. As will be explained in more detail later, the reflection buffer may store multiple reflection colors.

[0106] A frame buffer is a buffer that stores the game images to be displayed. The images stored in the frame buffer are output to the display device at predetermined timings and displayed on the display device.

[0107] (Details of game processing in main unit 2) Next, with reference to Figures 24 to 27, the details of the processing performed in the main unit 2 will be explained. Figure 24 is a flowchart showing an example of the main processing performed by the processor 81 of the main unit 2. The processing shown in Figures 24 to 27 is performed by the CPU or GPU of the processor 81.

[0108] As shown in Figure 24, the processor 81 first performs initial processing (step S100). Specifically, the processor 81 sets up a three-dimensional virtual space and places each object (30, 32, 34, 36), each character object (41, 43), a virtual camera VC, a light source, and various other objects used in the game into the virtual space. After performing the initial processing, the processor 81 repeatedly executes the processing in the next steps S101 to S108 at predetermined frame time intervals (for example, 1 / 60 second intervals).

[0109] In step S101, the processor 81 acquires operation data from the controller.

[0110] Next, the processor 81 performs game processing based on the acquired operation data (step S102). For example, the processor 81 moves the character object 41 in the virtual space or causes the character object 41 to perform a predetermined action (e.g., a jump action, an attack action, etc.) based on the operation data. The processor 81 also moves the character object 43 in the virtual space or causes the character object 43 to perform a predetermined action according to a predetermined algorithm. Furthermore, the processor 81 controls enemy objects other than character objects in the virtual space or moves obstacle objects that hinder character objects in the virtual space.

[0111] Next, the processor 81 performs rendering processing on objects other than character objects (step S103). Specifically, the processor 81 performs a first depth test on each object other than character objects 41 and 43 that are included in the imaging range of the virtual camera VC, and updates the first depth buffer. The processor 81 also draws each object to the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering is used as the rendering method. In step S103, the normal buffer is also updated. Forward rendering may also be used as the rendering method.

[0112] Next, the processor 81 renders the character object into a separate buffer to create a character plane (step S104). Specifically, the processor 81 renders the character object 41 into a separate buffer and creates a character plane 65 by setting the image of the character object 41 rendered into the separate buffer as a texture to a plane object. Similarly, the processor 81 also creates a character plane 66 by setting the image of the character object 43 as a texture to a plane object.

[0113] After step S104, processor 81 performs SSR processing (step S105). SSR processing is the process of drawing reflections of all objects, including character objects, to the frame buffer. The details of SSR processing are described below.

[0114] (SSR processing) Figure 25 is a flowchart showing an example of the SSR process in step S105.

[0115] As shown in Figure 25, the processor 81 performs a second depth test on a specific object and updates the second depth buffer (step S121). For example, the processor 81 performs a depth test on the tree object 32 and the cylinder object 34, which have been set as specific objects in advance, and updates the second depth buffer. Note that the setting of specific objects is done on a mesh basis.

[0116] Next, processor 81 performs ray tracing (step S122). Ray tracing is performed for each pixel of interest. Here, the color of the reflection on each pixel of interest is calculated. Note that ray tracing may be performed for all pixels, or only for a specific range of pixels. For example, the load may be reduced by excluding parts where the model is not drawn, parts where SSR is disabled on a model or mesh basis, or parts where the normal direction is outside a specific range. The details of ray tracing are described below.

[0117] (Ray tracing) Figure 26 is a flowchart showing an example of the ray tracing process in step S122.

[0118] As shown in Figure 26, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position and calculates the reflection direction (step S131). Specifically, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position based on the position of the pixel of interest in the screen coordinate system and the depth of the pixel of interest stored in the first depth buffer. The processor 81 also defines the direction from the position of the virtual camera VC toward the reflection position as the incident direction and calculates the reflection direction based on the incident direction and the normal direction of the reflection position.

[0119] Next, the processor 81 sets the calculated reflection direction, or the direction obtained by correcting the reflection direction, as the ray tracing direction (step S132). Here, if the correction conditions are met, the direction obtained by correcting the reflection direction is set as the ray tracing direction; otherwise, the reflection direction is set as the ray tracing direction. The ray tracing end position is also set. The correction conditions and correction method are as described above.

[0120] After step S132, the processor 81 sets a first depth buffer as a depth buffer for determination and starts ray tracing from the pixel of interest (step S133). Ray tracing is performed in screen coordinates. Next, the processor 81 advances the ray a predetermined distance (step S134).

[0121] Next, the processor 81 determines whether the trace distance has exceeded the maximum distance Lb (step S135). Specifically, the processor 81 determines whether the leading edge of the ray in the screen coordinate system has reached the ray trace end position.

[0122] If it is determined that the trace distance exceeds Lb (step S135: YES), the processor 81 then performs the process in step S141.

[0123] On the other hand, if the trace distance is less than or equal to Lb (step S135: NO), the processor 81 determines whether or not the ray has collided with an object (step S136). Specifically, the processor 81 uses a depth buffer for determination to determine whether or not the leading edge of the ray has collided with an object.

[0124] If it is determined that the ray has not collided with an object (step S136: NO), the processor 81 determines whether the trace distance is greater than or equal to La (step S137).

[0125] If the trace distance is greater than or equal to La (step S137: YES), the processor 81 changes the determination depth buffer to the second depth buffer (step S138) and executes the process in step S134 again. On the other hand, if the trace distance is less than La (step S137: NO), the processor 81 maintains the determination depth buffer and executes the process in step S134 again.

[0126] On the other hand, if it is determined that the ray has collided with an object (step S136: YES), the processor 81 calculates the color of the first reflection from the color of the ray's collision location (step S139). For example, the processor 81 may calculate the color of the pixel at the collision location stored in the frame buffer as the color of the first reflection, or it may calculate the color of the pixel at the collision location after applying a predetermined process as the color of the first reflection. The color of the first reflection differs depending on whether the collision is determined based on the first depth buffer or the second depth buffer. For example, if the first depth buffer is set as the depth buffer for determination, the color of the pixel at the collision location may be lightened (alpha value reduced) according to the trace distance, and the changed color may be calculated as the color of the first reflection. Also, if the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision location may be calculated as the color of the first reflection regardless of the trace distance. Furthermore, if a second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position is changed to a lighter shade according to the trace distance, but it is changed to a darker shade than when the first depth buffer is set as the depth buffer for determination, and this changed color may be calculated as the color of the first reflection. The processor 81 then stores the calculated color of the first reflection in the reflection buffer. Here, the color of the first reflection stored in the reflection buffer when it is determined that a ray has collided based on the first depth buffer is denoted as "color of the first reflection (first depth buffer)". Also, the color of the first reflection stored in the reflection buffer when it is determined that a ray has collided based on the second depth buffer is denoted as "color of the first reflection (second depth buffer)".

[0127] Next, the processor 81 determines whether the depth buffer for determination is the first depth buffer and whether the color of the first reflection calculated in step S139 is semi-transparent (step S140). For example, if the trace distance is less than or equal to La and the color of the first reflection is semi-transparent, ray tracing is performed further into the background. Semi-transparent means that the alpha value of the reflection is less than 1.0, but the alpha value is calculated independently based on the distance La, the collision position, and various other parameters.

[0128] If the result in step S140 is NO, the processor 81 proceeds to step S141. If the result in step S140 is YES, the processor 81 proceeds to step S138.

[0129] In step S141, the processor 81 calculates the color of the reflection of the character object (the color of the second reflection). The details of the process in step S141 are described below.

[0130] Figure 27 is a flowchart showing an example of the process for calculating the color of the character's reflection in step S141.

[0131] As shown in Figure 27, the processor 81 calculates the intersection position of the ray and the character plane (step S151). Specifically, the processor 81 calculates the depth when the character plane is positioned according to the position and orientation of the character object in virtual space, and determines whether the ray used in the ray tracing intersects with the character plane. If the ray intersects with the character plane, the processor 81 calculates the intersection position.

[0132] Next, the processor 81 stores the color of the intersection position as the color of the second reflection in the reflection buffer (step S152). Specifically, the processor 81 stores a color based on the color of a pixel in the texture image of the character object corresponding to the intersection position as the color of the second reflection. The color of the second reflection may be the color of a pixel in the texture image of the character object, or it may be a color calculated by applying a predetermined process to the color of that pixel. When the processor 81 has performed the process in step S152, it terminates the process shown in Figure 27 and returns to the process in Figure 26. If the ray intersects with multiple character planes, multiple colors of the second reflection are stored in the reflection buffer. In this case, the multiple colors of the second reflection may be stored in the reflection buffer in order of proximity of the intersection position to the virtual camera.

[0133] As a result of the processing in steps S139 and S152, the reflection buffer stores zero to multiple reflection colors. For example, the reflection buffer may store reflection colors corresponding to the ray's collision or intersection positions, in order of proximity to the virtual camera. For example, if it is determined that a ray has collided at a first position where the trace distance is less than La, and also determined that a ray has collided at a second position where the trace distance is greater than or equal to La and less than Lb, and the ray intersects the character plane between the first and second positions, the reflection buffer stores the first reflection color (first depth buffer), the second reflection color, and the first reflection color (second depth buffer) in order of proximity to the virtual camera (see Figure 23). If the ray does not collide with any object and does not intersect the character plane, no reflection color is stored in the reflection buffer.

[0134] Returning to Figure 26, after processing in step S141, the processor 81 calculates the color reflected in the pixel of interest based on the reflected colors stored in the reflection buffer (step S142). If multiple reflected colors are stored in the reflection buffer, the processor 81 may calculate the color reflected in the pixel of interest by alpha blending the multiple reflected colors in order of distance from the virtual camera. For example, as shown in Figure 23, if the reflection buffer stores the first reflected color (first depth buffer), the second reflected color, and the first reflected color (second depth buffer) in order of proximity to the virtual camera, the processor 81 may calculate a color by alpha blending the second reflected color with the first reflected color (second depth buffer), and then alpha blend the first reflected color (first depth buffer) with the calculated color to calculate the color reflected in the pixel of interest. Furthermore, for example, if the reflection buffer stores the colors of the second reflection and the first reflection in order of proximity to the virtual camera (i.e., when there are no objects in front of the character from the perspective of the virtual camera, and there are objects behind the character), the processor 81 may calculate the color reflected in the pixel of interest by alpha blending the color of the second reflection with the color of the first reflection. For example, if the color of the second reflection on the near side is opaque, the color of the second reflection is set as the color reflected in the pixel of interest. Also, if the color of the second reflection on the near side is semi-transparent, a color is calculated by blending the color of the first reflection and the color of the second reflection as the color reflected in the pixel of interest. The processor 81 stores the calculated reflection color in memory.

[0135] If the process in step S142 is completed, the processor 81 terminates the process in Figure 26 and returns to the process in Figure 25.

[0136] Returning to Figure 25, after step S122, the processor 81 renders to the frame buffer based on the results of the ray tracing process (step S123). For example, the processor 81 adds the reflected color calculated in step S142 to the color of the pixel of interest stored in the frame buffer. As a result, the reflected color calculated in step S142 is reflected in the color of the pixel of interest.

[0137] Next, the processor 81 determines whether or not the processes in steps S122 and S123 have been performed for all pixels (step S124). If the processor 81 determines NO in step S124, it repeats the process in step S122. By performing the processes in steps S122 and S123 for all pixels, reflections of objects other than character objects placed in the virtual space (52, 54) and reflections of character objects (61, 63) are drawn to the frame buffer. If the processor 81 determines YES in step S124, it terminates the process in Figure 25 and returns the process to Figure 24.

[0138] Returning to Figure 24, after step S105, the processor 81 further renders the character objects 41 and 43 into the frame buffer (step S106).

[0139] Next, the processor 81 outputs the image stored in the frame buffer to the display device (step S107). This displays the game image.

[0140] Next, the processor 81 determines whether or not to terminate the game (step S108). For example, if the player instructs the game to end, the processor 81 terminates the game processing shown in Figure 24. If the processor 81 determines that the game should not be terminated (step S108: NO), it executes the process in step S101 again. This concludes the explanation of the main processing shown in Figure 24.

[0141] The order of processes, content, and values ​​used for determination in the flowchart above are merely examples and may be changed as appropriate.

[0142] As described above, in this embodiment, if the trace distance is less than La, ray collision detection is performed based on the first depth buffer, and if the trace distance is La or greater, ray collision detection is performed based on the second depth buffer (steps S135 to S140). This makes it possible to display reflections even when the trace distance is long for a particular object. Furthermore, reflections can be hidden from objects other than the specific object, allowing for the display or exclusion of desired objects. Additionally, by generating reflections based on the first and second depth buffers, and by lightening the color of the reflection according to the trace distance when based on the first depth buffer, it is possible to make specific objects appear large and clear, while other objects appear small.

[0143] Furthermore, in this embodiment, when the pixel of interest (reflection position) is close to the edge of the screen, the direction obtained by correcting the reflection direction based on the incident direction and the normal direction at the reflection position is set as the ray tracing direction (step S132). This makes it possible to calculate the color of the reflection based on the pixels drawn to the frame buffer, and to generate reflections even at the edge of the screen.

[0144] Furthermore, in this embodiment, objects other than character objects are drawn to the frame buffer first, their reflections are drawn, then the reflections of character objects are drawn, and finally the character objects are drawn. This makes it possible to display the reflections of objects behind the character object as seen from the virtual camera, as well as the reflection of the character object.

[0145] Furthermore, in this embodiment, the color of the collision position calculated based on the first depth buffer, the color of the collision position calculated based on the second depth buffer, and the color of the intersection position with the character plane are calculated, and these colors are alpha blended in order from the back. As a result, even when each object overlaps as seen from the virtual camera, the reflection of each object can be displayed.

[0146] (modified version) The image processing of this embodiment has been described above, but the above embodiment is merely an example, and modifications such as the following may be made.

[0147] For example, in the above embodiment, deferred rendering was used as the rendering method, but in other embodiments, forward rendering may be used. When forward rendering is used, in the above SSR processing, for example, the normal direction of the reflection position corresponding to the pixel of interest may be estimated based on the depth of multiple pixels surrounding the pixel of interest, and the reflection direction may be calculated based on the incident direction and said normal direction. Alternatively, the pose of the mesh in virtual space may be calculated, the normal direction of the reflection position corresponding to the pixel of interest may be calculated, and the reflection direction may be calculated based on said normal direction.

[0148] Furthermore, in the above embodiment, the reflection direction was corrected for pixels in a predetermined range from the left or right edge of the screen, but in other embodiments, the reflection direction may also be corrected for pixels in a predetermined range from the top or bottom edge of the screen. In the above embodiment, in correcting the reflection direction, the position of the ray tracing end position in the Sx axis direction in the screen coordinate system was brought closer to the position of the reflection position in the Sx axis direction. In other embodiments, the reflection direction may also be corrected by moving the position of the ray tracing end position in the Sy axis direction in the positive or negative direction of the Sy axis. The reflection direction may also be corrected by moving the ray tracing end position horizontally and / or vertically so that the ray collision position fits within the screen.

[0149] Furthermore, while ray tracing was performed in the screen coordinate system in the above embodiment, in other embodiments, ray tracing may be performed in the xyz coordinate system of a virtual space. That is, rays may be extended in the xyz coordinate system of a virtual space, the collision positions of the rays may be calculated, and the color of the pixel at the position corresponding to the collision position may be calculated as the color of the reflection.

[0150] In the above embodiment, a character plane was created by setting the image of the character object as a texture on a planar object, and the color of the reflection of the character object was calculated using this character plane. In other embodiments, the image of the character object may be set as a texture on a three-dimensional object, and the color of the reflection of the character object may be calculated using this three-dimensional object. Also, in the above embodiment, the character object was assumed to be a flat object, but in other embodiments, the character object may be a three-dimensional object with thickness.

[0151] Furthermore, although the above embodiment described the case where the ground is used as a reflective surface and an object is reflected on the ground, an object may also be reflected using the above-described process on a surface other than the ground (for example, a wall or ceiling).

[0152] Furthermore, although the above embodiment was used to generate game images, the image processing described above may be used to generate any image, not just game images.

[0153] Furthermore, the above hardware configuration is merely an example, and the image processing may be performed on any other hardware. For example, the processing may be performed on any information processing device such as a personal computer, tablet terminal, smartphone, or server on the internet. Also, the image processing may be performed in an information processing system including multiple devices. Multiple devices may perform the image processing in a distributed manner. In addition, the device that performs the image processing and the device that displays the image may be different. For example, a first device (e.g., a server) may generate an image by performing part or all of the image processing, transmit the generated image to a second device via a network (e.g., the internet or a LAN), and the second device may display the image.

[0154] Furthermore, the configurations of the above embodiments and their modified forms can be combined in any way, as long as they do not contradict each other. Moreover, the above is merely an example of the present invention, and various other improvements and modifications may be made. [Explanation of Symbols]

[0155] 1. Game System 2. Main unit 81 processors 32 tree objects 34 Cylinder Objects 36 Mountain Objects 41, 43 Character Objects

Claims

1. In the computer of the information processing device, Of the objects in the virtual space, all objects except for the first type, The first depth buffer is used to perform a first depth test and update the first depth buffer. Based on the results of the first depth test, the frame buffer is used to draw. For the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first object in the virtual space. For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest. Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction. The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer. When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest. When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflected color to be added to the color of the pixel of interest. The frame buffer is then configured to reflect the color of the reflection. An image processing program that, after reflecting the colors of the reflections, draws the first type of object to the frame buffer.

2. The image processing program according to claim 1, wherein the first type of object is a flat-shaped object.

3. The image processing program according to claim 2, wherein the first type of object is a player character object controlled based on an operation input.

4. An image processing system comprising a processor, wherein the processor is Of the objects in the virtual space, all objects except for the first type, A first depth test and an update of the first depth buffer are performed using the first depth buffer. Based on the results of the first depth test, the data is drawn to the frame buffer. For the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first object in the virtual space. For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest. Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction. The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer. When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest. When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest. The frame buffer reflects the color of the reflection, An image processing system that, after reflecting the colors of the reflections, draws the first type of object to a frame buffer.

5. The image processing system according to claim 4, wherein the first type of object is a flat-shaped object.

6. The image processing system according to claim 5, wherein the first type of object is a player character object controlled based on an operation input.

7. An image processing method, Of the objects in the virtual space, all objects except for the first type, Perform a first depth test and update the first depth buffer using the first depth buffer. Based on the results of the first depth test, draw to the frame buffer. With respect to the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first object in the virtual space. For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest. Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction. The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer. When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest. When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest. The frame buffer reflects the color of the reflection. An image processing method comprising: reflecting the color of the reflection, and then drawing the first type of object to the frame buffer.

8. The image processing method according to claim 7, wherein the first type of object is a flat-shaped object.

9. The image processing method according to claim 8, wherein the first type of object is a player character object controlled based on an operation input.

10. An image processing apparatus comprising a processor, wherein the processor is Of the objects in the virtual space, all objects except for the first type, A first depth test and an update of the first depth buffer are performed using the first depth buffer. Based on the results of the first depth test, the data is drawn to the frame buffer. For the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first object in the virtual space. For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest. Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction. The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer. When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest. When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest. The frame buffer reflects the color of the reflection, An image processing device that, after reflecting the color of the reflection, draws the first type of object to the frame buffer.

11. The image processing apparatus according to claim 10, wherein the first type of object is a flat-shaped object.

12. The image processing apparatus according to claim 11, wherein the first type of object is a player character object controlled based on an operation input.

Images