Image processing program, image processing system, image processing method, and image processing apparatus

Abstract

To provide an image processing program capable of changing the way each object is reflected.SOLUTION: An image processing system according to one embodiment updates a first depth buffer by performing a first depth test on an object in a virtual space, and performs rendering to a frame buffer based on the first depth buffer. The image processing system updates a second depth buffer by performing a second depth test on a specific object among the objects. The image processing system performs ray tracing based on the first depth buffer if a tracing distance is less than a second distance that is smaller than a first distance, and performs ray tracing based on the second depth buffer if the tracing distance is equal to or greater than the second distance, and calculates the color of a pixel in the frame buffer corresponding to the impact position of the ray as a reflected color.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing program, an image processing system, an image processing method, and an image processing device that are capable of expressing reflection based on a drawn image. [Background technology]

[0002] Conventionally, there is a method for expressing reflection by processing a drawn image (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Screen Space Reflection”, [online], 2023, Unity Technologies, [Retrieved December 20, 2023], Internet <URL:https: / / docs.unity3d.com / ja / 2019.4 / Manual / PostProcessing-ScreenSpaceReflection.html> Summary of the Invention [Problem to be solved by the invention]

[0004] Since the above-mentioned conventional techniques are applied to a rendered image, it is difficult to adjust the way each object is reflected.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing system, an image processing method, and an image processing device that are capable of changing the way each object is reflected. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention employs the following configuration.

[0007] (First Configuration) The image processing program of the first configuration causes a computer of an information processing device to perform a first depth test and update a first depth buffer for objects in a virtual space, to draw in a frame buffer based on the results of the first depth test, and to further perform a second depth test and update a second depth buffer for a first type of object among the objects, using a second depth buffer. The image processing program also causes the computer to, for each pixel in the frame buffer where drawing has been performed, calculate, based on the depth of the first depth buffer, the pixel as a pixel of interest, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest as the incident direction, and the direction of the ray reflected from that position as the reflection position as the ray tracing direction, trace the ray along the ray tracing direction, and determine a collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer if the tracing distance of the ray is less than a second distance that is smaller than the first distance, or based on the depth of the second depth buffer if the tracing distance is equal to or greater than the second distance, and when the collision position is determined to be within a range where the tracing distance is within the first distance, determine a color based on the color of the pixel in the frame buffer corresponding to the collision position as the reflection color to be added to the color of the pixel of interest.

[0008] According to the above, by switching the depth buffer to be referenced depending on the tracing distance, for example, it is possible to make some objects appear farther away, and to make each object appear differently.

[0009] (Second Configuration) In a second configuration, in the above-mentioned first configuration, the computer may further determine the reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position, based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

[0010] According to the above, the color of the pixel corresponding to the collision position can be added to the color of the pixel of interest based on parameters that differ depending on the depth buffer to be referenced.

[0011] (Third Configuration) In a third configuration, in the first or second configuration, the computer may further place a planar object at the position of the second type of object in the virtual space, the planar object having a texture set to an image rendered without drawing to a frame buffer, and when the ray intersects with the planar object, determine a reflection color to be added to the color of the pixel of interest based on the color of the texture at the intersection position, reflect the reflection color in the frame buffer, and draw the second type of object in the frame buffer after reflecting the reflection color.

[0012] According to the above, it is further possible to generate a reflection of a second type of object, and for example, even if the second type of object is present in front of the object, it is possible to generate a reflection of the object and a reflection of the second type of object.

[0013] (Fourth Configuration) In a fourth configuration, in any of the first to third configurations, the computer may further calculate, as the ray tracing direction, a reflection direction based on the incident direction and the normal direction of the reflection position, with a correction made toward the inside of the screen.

[0014] Based on the above, ray tracing can be performed based on a direction obtained by correcting the reflection direction based on the normal direction.

[0015] (Fifth Configuration) In a fifth configuration, in the third configuration, the second type of object may be a flat-shaped object.

[0016] According to the above, the second type of object is an object with a flat shape, and the reflection also appears planar, so that an image that does not look unnatural can be obtained.

[0017] Further, other embodiments may be an image processing system that performs the image processing, an image processing device, or an image processing method. [Effects of the Invention]

[0018] According to the present invention, it is possible to make the way objects are reflected differently for each object. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 shows an example of a state in which the left controller 3 and the right controller 4 are attached to the main unit 2. [Figure 2] A block diagram showing an example of the internal configuration of the main unit 2. [Figure 3] FIG. 10 is a diagram showing an example of a plurality of objects placed in a virtual space during execution of a game according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a game image displayed on a display device. [Figure 5] A diagram showing an overview of image processing for reflections of objects other than character objects. [Figure 6] FIG. 10 illustrates an example of a first depth buffer. [Figure 7] FIG. 10 is a diagram showing an example of an image drawn in a frame buffer after rendering in step S2. [Figure 8] FIG. 10 illustrates an example of a second depth buffer. [Figure 9] FIG. 10 is a diagram showing how ray tracing is performed, and how a ray reflected from a reflection position RPa collides with a cylindrical object 34. [Figure 10] A diagram showing how collision detection for a ray reflected from reflection position RPa is performed using the first depth buffer. [Figure 11]FIG. 10 is a diagram showing how ray tracing is performed, and how a ray reflected from a reflection position RPb collides with a tree object 32. [Figure 12] This is a diagram showing how ray tracing is performed, and how the ray reflected from the reflection position RPc does not collide with an object. [Figure 13] A diagram showing how ray collision detection is performed using the second depth buffer. [Figure 14] A diagram showing an example of an image drawn in the frame buffer when a reflection color is added to each pixel. [Figure 15] A diagram showing an example of ray tracing direction with reflection direction corrected. [Figure 16] FIG. 10 is a diagram for explaining details of correction for reflection direction. [Figure 17] FIG. 10 shows an example of a game image when a character object 41 does not exist in front of a cylindrical object 34. [Figure 18] FIG. 10 shows an example of a game image when a character object 41 moves and is present in front of a cylindrical object 34. [Figure 19] A diagram showing an overview of image processing for generating a character object and a reflection of an object behind it. [Figure 20] FIG. 10 is a diagram showing an example of a character plane created in step S11. [Figure 21] FIG. 10 is a diagram illustrating calculation of the color of the character reflection in step S13. [Figure 22] FIG. 20 is a diagram showing an example of a game image displayed when the image processing of FIG. 19 is performed; [Figure 23] FIG. 10 is a diagram showing an example of data stored in the memory of the main device 2. [Figure 24] 10 is a flowchart showing an example of main processing executed by the processor 81 of the main device 2. [Figure 25] Flowchart showing an example of the SSR process in step S105 [Figure 26]A flowchart showing an example of the ray tracing process in step S122. [Figure 27] A flowchart showing an example of the process of calculating the color of the character reflection in step S141. DETAILED DESCRIPTION OF THE INVENTION

[0020] (System Configuration) A game system according to an example of this embodiment will be described below. An example of the game system 1 according to this embodiment includes a main unit (information processing device; in this embodiment, it functions as a game device main unit) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are each detachable from the main unit 2. In other words, the game system 1 can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main unit 2. The game system 1 can also be used as a separate unit from the main unit 2, with the left controller 3 and the right controller 4. The hardware configuration of the game system 1 according to this embodiment will be described below, followed by a description of the control of the game system 1 according to this embodiment.

[0021] FIG. 1 is a diagram showing an example of a state in which a left controller 3 and a right controller 4 are attached to a main unit 2. As shown in FIG. 1, the left controller 3 and the right controller 4 are each attached to and integrated with the main unit 2. The main unit 2 is a device that executes various processes (e.g., game processes) in the game system 1. The main unit 2 is equipped with a display 12. The left controller 3 and the right controller 4 are devices that have operation units that allow the user to perform inputs.

[0022] The left controller 3 and the right controller 4 are detachable from the main unit 2. In the following, the left controller 3 and the right controller 4 may be collectively referred to as the "controller."

[0023] The main unit 2 alone or an integrated device in which the left controller 3 and right controller 4 are attached to the main unit 2 may be a portable device. The main unit 2 or the integrated device may be a handheld device. The main unit 2 or the integrated device may be a portable device.

[0024] The main device 2 also includes a touch panel 13 on the screen of the display 12. In this embodiment, the touch panel 13 is of a type that allows multi-touch input (for example, a capacitance type). However, the touch panel 13 may be of any type, and may be of a type that allows single-touch input (for example, a resistive type).

[0025] FIG. 2 is a block diagram showing an example of the internal configuration of the main unit 2. As shown in FIG.

[0026] The main unit 2 includes a processor 81. The processor 81 is an information processing unit that executes various types of information processing executed in the main unit 2, and includes one or more central processing units (CPUs) and one or more graphics processing units (GPUs). The processor 81 may be configured as a system-on-a-chip (SoC) that includes multiple functions, such as a CPU function and a GPU function. The CPU and the GPU may each be configured as separate processors. The processor 81 also includes one or more memories that temporarily store data. The processor 81 executes various types of information processing by executing an information processing program (e.g., a game program) stored in a storage unit (specifically, an internal storage medium such as a flash memory 84, or an external storage medium inserted into the slot 23).

[0027] The main device 2 includes a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85 as examples of internal storage media built into the main device 2. The flash memory 84 and the DRAM 85 are connected to the processor 81. The flash memory 84 is a memory used primarily to store various types of data (which may be programs) saved in the main device 2. The DRAM 85 is a memory used to temporarily store various types of data used in information processing.

[0028] The main device 2 includes a slot interface (hereinafter abbreviated as "I / F") 91. The slot I / F 91 is connected to the processor 81. The slot I / F 91 is connected to the slot 23, and reads and writes data from and to a predetermined type of storage medium (e.g., a dedicated memory card) inserted into the slot 23 in accordance with instructions from the processor 81.

[0029] The processor 81 reads and writes data from and to the flash memory 84, DRAM 85, and the above-mentioned storage media as appropriate, to execute the above-mentioned information processing.

[0030] The main unit 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates with an external device via a network (specifically, wireless communication). In this embodiment, the network communication unit 82 connects to a wireless LAN and communicates with the external device using a method conforming to the Wi-Fi standard as a first communication mode. The network communication unit 82 also communicates wirelessly with other main units 2 of the same type using a predetermined communication method (for example, communication using a proprietary protocol or infrared communication) as a second communication mode.

[0031] The main unit 2 is equipped with a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 performs wireless communication with the left controller 3 and / or right controller 4. Any communication method may be used between the main unit 2 and the left controller 3 and right controller 4, but in this embodiment, the controller communication unit 83 performs communication with the left controller 3 and right controller 4 in accordance with the Bluetooth (registered trademark) standard.

[0032] The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When performing wired communication with the left controller 3, the processor 81 transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. When performing wired communication with the right controller 4, the processor 81 transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. When performing communication with the cradle, the processor 81 transmits data to the cradle via the lower terminal 27. As described above, in this embodiment, the main unit 2 can perform both wired and wireless communication with the left controller 3 and the right controller 4. When an integrated device in which the left controller 3 and the right controller 4 are attached to the main unit 2 or when the main unit 2 alone is attached to the cradle, the main unit 2 can output data (e.g., image data and audio data) to a stationary monitor or the like via the cradle.

[0033] The main device 2 includes a touch panel controller 86, which is a circuit that controls the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. Based on a signal from the touch panel 13, the touch panel controller 86 generates data indicating, for example, the position where a touch input was made, and outputs the data to the processor 81.

[0034] The main device 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Although not shown, the power control unit 97 is also connected to each part of the main device 2 (specifically, each part that receives power from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls the power supply from the battery 98 to each of the above parts based on instructions from the processor 81.

[0035] Furthermore, battery 98 is connected to lower terminal 27. When an external charging device (e.g., a cradle) is connected to lower terminal 27 and power is supplied to main device 2 via lower terminal 27, battery 98 is charged with the supplied power.

[0036] (Image Processing Overview) Next, the image processing of this embodiment will be described. In the game system 1 of this embodiment (an example of an image processing system), a plurality of objects are placed in a three-dimensional virtual space and a game is played.

[0037] 3 is a diagram showing an example of a plurality of objects placed in a virtual space during execution of the game of this embodiment. When the game of this embodiment starts, a three-dimensional virtual space defined by an xyz Cartesian coordinate system is set. For example, the y-axis is the upward axis of the virtual space, the x-axis is the rightward axis, and the z-axis is the back axis.

[0038] As shown in FIG. 3, a ground object 30 is placed as a plurality of objects in the virtual space. The ground object 30 is an object that resembles the ground, and is, for example, a plane parallel to the xz plane. The ground object 30 may have an uneven surface or a slope. A tree object 32 and a cylinder object 34 are placed on the ground object 30. Furthermore, for example, a mountain object 36 is placed at a predetermined position in the z-axis direction of the virtual space. The ground object 30, the tree object 32, and the cylinder object 34 are fixed in the virtual space.

[0039] Character objects 41 and 43 are also placed in the virtual space. The character objects 41 and 43 are flat, plate-like objects whose thickness is smaller than their vertical and horizontal dimensions. For example, the character objects 41 and 43 are each 3D objects including a planar mesh constituting the front surface and a planar mesh constituting the back surface. The character object 41 is a player character controlled by the player. The character object 41 performs actions in the virtual space in response to inputs to the controller. For example, the character object 41 performs actions such as moving on the ground object 30, moving its limbs in the virtual space, jumping, and changing the direction of its face. When the character object 41 moves left or right, the screen scrolls left or right. The character object 43 is a non-player character that moves in response to the movement of the character object 41 and is automatically controlled by the processor 81. The character object 43 may also be controlled by the player.

[0040] A virtual camera is placed in the virtual space, and a game image of the virtual space viewed from the virtual camera is generated, and the generated game image is displayed on a display 12 or a stationary monitor (hereinafter referred to as a "display device").

[0041] 4 is a diagram showing an example of a game image displayed on a display device. Images of each object (30, 32, 34, 36, 41, 43) are displayed on the display device. For example, the ground object 30 is a reflective surface, and each object is reflected on the ground object 30. In this embodiment, the reflection of each object is generated by a method called SSR (Screen Space Reflection).

[0042] 4, a reflection 52 of a tree object 32 and a reflection 54 of a cylinder object 34 are displayed on the ground object 30. Also, a reflection 61 of a character object 41 and a reflection 63 of a character object 43 are displayed on the ground object 30. On the other hand, a reflection of a mountain object 36 is not displayed on the ground object 30.

[0043] In this embodiment, the tree object 32 and the cylinder object 34 are set in advance as specific objects, and are set so that they are reflected on the ground object even when they are at a certain distance from the virtual camera. On the other hand, the mountain object 36 is not set as a specific object.

[0044] A specific object is set in units of meshes (a collection of polygons). For example, each of the multiple meshes constituting the tree object 32 may be set as a specific object. Alternatively, any one of the multiple meshes constituting the tree object 32 may be set as a specific object. For example, if the tree object 32 is composed of a first mesh constituting the trunk and a second mesh constituting the leafy portion, only the first mesh of the first and second meshes may be set as a specific object. In other words, a "specific object" does not necessarily have to be an object that appears to be a single object from the outside, but may be a mesh that constitutes a part of an object that appears to be a single object from the outside. For example, a terrain object in which a ground surface parallel to the xz plane and a wall surface perpendicular to the xz plane are continuously formed may appear to form a single terrain with steps from the outside, but the ground surface and the wall surface may be composed of different meshes. The wall surface of such a terrain object may be set as a specific object, but the ground surface may not be set as a specific object.

[0045] In this embodiment, a tree object 32 and a cylinder object 34, which are set as specific objects, are reflected on the ground object. Image processing relating to the reflection of these objects will be described below.

[0046] FIG. 5 is a diagram showing an outline of image processing relating to the reflection of objects other than character objects.

[0047] 5, the processor 81 first performs a first depth test (step S1). Here, a depth test is performed on a pixel-by-pixel basis for all objects (30, 32, 34, 36) other than the character object, and the results of the depth test are stored in a first depth buffer.

[0048] FIG. 6 is a diagram showing an example of the first depth buffer. As shown in FIG. 6, the first depth buffer stores depth values for each pixel for all objects (30, 32, 34, 36) other than character objects. Note that while an image of each object is displayed in FIG. 6, this conceptually represents the depth value of each pixel stored in the first depth buffer. The position of each pixel is represented by coordinate values in a screen coordinate system. For example, the Sx axis of the screen coordinate system is the axis pointing to the right in the image (screen), and the Sy axis is the axis pointing to the top in the image. The origin of the screen coordinate system may be set, for example, to the center of the image or to the bottom left of the image.

[0049] Returning to FIG. 5, after step S1, the processor 81 performs rendering to the frame buffer based on the result of the first depth test (step S2). In step S2, an image is drawn to the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering (also called deferred shading) is used as the rendering method. In step S2, the normal buffer that stores normal information for each pixel is updated, and the image is drawn to the frame buffer. Note that forward rendering may be used as the rendering method instead of deferred rendering.

[0050] FIG. 7 is a diagram showing an example of an image drawn in the frame buffer after rendering in step S2. As shown in FIG. 7, in step S2, images of all objects (30, 32, 34, 36) other than character objects are drawn in the frame buffer. Color information is stored for each pixel. The color information includes, for example, RGB values representing three colors and an alpha value representing transparency (or opacity). Note that shading is omitted in FIG. 7, but a shaded image is generated in step S2.

[0051] 5, after step S2, the processor 81 further performs a second depth test on a specific object among the plurality of objects arranged in the virtual space (step S3). Specifically, a second depth test is further performed on the tree object 32 and the cylinder object 34 that are set as specific objects among the plurality of objects (30, 32, 34, 36), and the results of the second depth test are stored in the second depth buffer. As a result, the second depth buffer stores the depth values for each pixel of only the tree object 32 and the cylinder object 34 among the plurality of objects (30, 32, 34, 36) included in the imaging range of the virtual camera.

[0052] Fig. 8 is a diagram showing an example of the second depth buffer. As shown in Fig. 8, the second depth buffer stores depth values for each pixel of a tree object 32 and a cylinder object 34. On the other hand, the depth value of a mountain object 36 is not stored in the second depth buffer. Note that Fig. 8 conceptually shows the depth values of each pixel stored in the second depth buffer.

[0053] Returning to Fig. 5, after step S3, the processor 81 performs ray tracing using the first depth buffer or the second depth buffer (step S4). Here, processing is performed for each pixel (hereinafter referred to as a "pixel of interest"). Specifically, the processor 81 sets the pixel of interest, calculates the reflection position of a ray (virtual light ray) when it is emitted in a direction from the position of the virtual camera toward the pixel of interest, and performs collision determination to determine whether the ray reflected from the reflection position will collide with an object.

[0054] Next, the processor 81 calculates the color of the pixel in the frame buffer corresponding to the collision position of the ray as the color reflected in the pixel of interest (step S5). The processing of step S5 is also performed for each pixel.

[0055] The processing of steps S4 and S5 when each of the pixels PIXa to PIXc shown in FIG. 7 is set as a pixel of interest will be specifically described below.

[0056] 7 as the pixel of interest. The pixel PIXa is located closer to the virtual camera than the cylindrical object 34. The processor 81 calculates the position of the pixel of interest PIXa in the virtual space as the reflection position RPa of the ray based on the position of the pixel of interest PIXa in the screen coordinate system and the depth value of that position stored in the first depth buffer.

[0057] 9 is a diagram showing how ray tracing is performed, and how a ray reflected from reflection position RPa collides with a cylindrical object 34. The right direction in FIG. 9 is the back direction of the screen when a game image generated based on virtual camera VC is displayed on the screen.

[0058] 9, the processor 81 defines the direction from the position of the virtual camera VC toward the reflection position RPa as the incident direction, and calculates the direction of the ray reflected from the reflection position RPa as the ray tracing direction RTDa. Specifically, the processor 81 calculates the reflection direction of the ray based on the incident direction and the normal to the reflection position RPa. The angle of incidence of the incident direction and the angle of reflection of the reflection direction are equal. The processor 81 then calculates the reflection direction, or a direction obtained by correcting the reflection direction, as the ray tracing direction RTDa. Note that correction of the reflection direction will be described later.

[0059] The processor 81 advances a ray from the reflection position RPa in the ray tracing direction RTDa and performs collision determination to determine whether the ray will collide with an object. If the tracing distance of the ray is less than La, the processor 81 performs collision determination using the first depth buffer. Here, the tracing distance is the distance from the reflection position along the ray tracing direction.

[0060] FIG. 10 is a diagram illustrating how a collision determination is performed for a ray reflected from a reflection position RPa using the first depth buffer. The processor 81 first calculates a position in virtual space that is a maximum distance Lb from the reflection position RPa in the ray tracing direction RTDa in the virtual space, and then converts the calculated position into a screen coordinate system to calculate a ray tracing end position in the screen coordinate system. The processor 81 sets the reflection position RPa (the pixel of interest RPa) as the ray tracing start position in the screen coordinate system, extends a ray from the ray tracing start position to the ray tracing end position, and determines whether the tip of the ray will collide with an object. Specifically, the processor 81 advances the ray a predetermined distance and determines whether the ray will collide with an object based on the depth of the ray tip and the depths of each position stored in the first depth buffer. In the example shown in FIGS. 9 and 10, it is determined that the ray reflected at the reflection position RPa will collide at the collision position CPa.

[0061] If it is determined that the ray will collide with the object at the collision position CPa, then in step S5, the color of the pixel in the frame buffer corresponding to the collision position CPa is calculated as the color reflected in the pixel of interest RPa.

[0062] 7 is set as the pixel of interest, and pixel PIXc is set as the pixel of interest. Pixel PIXb is a pixel located closer to the virtual camera than tree object 32. Pixel PIXc is a pixel located closer to the virtual camera than mountain object 36.

[0063] Fig. 11 is a diagram showing how ray tracing is performed, and how a ray reflected from reflection position RPb collides with a tree object 32. Fig. 12 is a diagram showing how ray tracing is performed, and how a ray reflected from reflection position RPc does not collide with an object. Fig. 13 is a diagram showing how ray collision determination is performed using the second depth buffer.

[0064] As shown in FIG. 11, when the processor 81 sets pixel PIXb as the pixel of interest, it calculates a reflection position RPb corresponding to the pixel of interest PIXb in the same manner as described above, and calculates the direction of the ray reflected from the reflection position RPb as a ray tracing direction RTDb. The processor 81 advances the ray in the ray tracing direction RTDb and performs a collision determination to determine whether the ray will collide with an object. When the tracing distance is less than La, the processor 81 performs a collision determination for the ray using the first depth buffer. However, when the tracing distance is less than La, the ray does not collide with an object. Therefore, the processor 81 advances the ray further and performs a collision determination for the ray using the second depth buffer (FIG. 13). As shown in FIGS. 11 and 13, when the tracing distance is equal to or greater than La and equal to or less than the maximum distance Lb, it is determined that the ray will collide with the tree object 32 at the collision position CPb.

[0065] On the other hand, as shown in FIG. 12, when the processor 81 sets pixel PIXc as the pixel of interest, it calculates the reflection position RPc corresponding to the pixel of interest PIXc in the same manner as described above, calculates the direction of the ray reflected from the reflection position RPc as the ray tracing direction RTDc, and advances the ray in the ray tracing direction RTDb. When the tracing distance is less than La, the processor 81 performs collision determination for the ray using the first depth buffer. When the tracing distance is equal to or greater than La and equal to or less than the maximum distance Lb, the processor 81 performs collision determination for the ray using the second depth buffer. Here, as shown in FIG. 13, the depth of the mountain object 36 is not stored in the second depth buffer. Therefore, as shown in FIGS. 12 and 13, the processor 81 determines that the ray will not collide with an object even if it advances the ray up to the maximum distance Lb.

[0066] Note that the parameters used to add the color of the pixel at the collision position to the color of the pixel of interest may be different when the collision position of the ray is calculated based on the first depth buffer and when the collision position of the ray is calculated based on the second depth buffer. For example, when the collision position is calculated based on the second depth buffer, the parameters may be changed so that the color of the reflection is darker than when the collision position is calculated based on the first depth buffer.

[0067] The above-described steps S4 and S5 are performed for each pixel of interest to determine the color reflected in that pixel. Then, the reflection of each object is drawn in the frame buffer by adding the determined reflected color to the color of the pixel of interest.

[0068] FIG. 14 is a diagram showing an example of an image drawn in the frame buffer when a reflected color is added to each pixel.

[0069] As shown in FIG. 14, the color of the pixel at the collision position CPa calculated as described above is added to pixel RPa. A similar process is performed on each pixel included in the region on the front side of the cylinder object 34 (the region on the virtual camera side), thereby drawing a reflection 54 of the cylinder object 34 in the frame buffer. Furthermore, the color of the pixel at the collision position CPb is added to pixel RPb. A similar process is performed on each pixel included in the region on the front side of the tree object 32, thereby drawing a reflection 52 of the tree object 32 in the frame buffer. On the other hand, no collision position is calculated for pixel RPc. The same applies when processing is performed on each pixel included in the region on the front side of the mountain object 36. Therefore, the reflection of the mountain object 36 is not drawn in the frame buffer.

[0070] In this embodiment, the results of a first depth test performed on a plurality of objects are stored in a first depth buffer, and the results of a second depth test performed on a specific object among the plurality of objects are stored in a second depth buffer. When the tracing distance is less than La, ray collision determination is performed based on the first depth buffer. When the tracing distance is equal to or greater than La and equal to or less than the maximum distance Lb, ray collision determination is performed based on the second depth buffer. This allows the SSR to display reflections of specific objects among the plurality of objects even when they are far from the virtual camera, while preventing reflections from being displayed for objects other than the specific object. By designating an object to be reflected as a specific object, the desired object can be reflected, for example, on the ground. Conversely, by not designating an object not to be reflected as a specific object, the desired object can be prevented from being reflected.

[0071] Furthermore, the parameters used to reflect the color of the pixel at the collision position onto the pixel of interest are different when the collision position of the ray is calculated based on the first depth buffer and when the collision position of the ray is calculated based on the second depth buffer, which allows, for example, a specific object to be reflected more intensely and clearly.

[0072] (Reflection direction correction) Next, correction of the reflection direction will be described. As described above, the direction from the virtual camera VC toward the reflection position is calculated as the incident direction, and the reflection direction is calculated based on this incident direction and the normal direction of the reflection position. In this embodiment, if the correction conditions are satisfied, the direction obtained by correcting the reflection direction is set as the ray tracing direction. If the correction conditions are not satisfied, the reflection direction is set as the ray tracing direction. Below, the correction conditions for correcting the reflection direction and the correction method will be specifically described.

[0073] Fig. 15 is a diagram showing an example of a ray tracing direction in which the reflection direction has been corrected. As shown in Fig. 15, for example, when ray tracing is performed on a pixel of interest (reflection position RP1) near the left edge of the screen, the reflection direction RD1 calculated based on the incident direction and the normal direction is corrected, and the corrected direction is set as the ray tracing direction RTD1. Also, when ray tracing is performed on a pixel of interest (reflection position RP2) that is located to the right of the reflection position RP1 on the screen, the reflection direction RD2 is corrected, and the corrected direction is set as the ray tracing direction RTD2.

[0074] The correction condition is satisfied when both the condition regarding the reflection position and the condition regarding the reflection direction are satisfied. The condition regarding the reflection position is that the reflection position is within a predetermined range from the edge of the screen. Furthermore, the condition regarding the reflection direction is that the reflection direction faces outward from the screen. If either the condition regarding the reflection position or the condition regarding the reflection direction is not satisfied, the correction condition is not satisfied and the reflection direction is not corrected. For example, if the reflection position is within a range from the left edge of the screen to 1 / 4 of the horizontal length of the screen and the reflection direction faces leftward on the screen, the correction condition is satisfied. Furthermore, if the reflection position is within a range from the right edge of the screen to 1 / 4 of the horizontal length of the screen and the reflection direction faces rightward on the screen, the correction condition is satisfied.

[0075] The degree of correction is greater at reflection position RP1, which is closer to the left edge of the screen, than at reflection position RP2. Specifically, the degree of correction CR is determined based on the position of the reflection position in the Sx axis direction of the screen coordinate system. The closer the position of the reflection position in the Sx axis direction is to the left or right edge of the screen, the greater the degree of correction CR. Furthermore, the greater the angle between the reflection direction and the Sy axis direction in the screen coordinate system, the smaller the degree of correction CR.

[0076] FIG. 16 is a diagram for explaining the details of the correction to the reflection direction. As shown in FIG. 16, for example, when the angle between the reflection direction RD and the upward direction of the screen (Sy-axis direction) is 45 degrees or less, the degree of correction CR is set to a value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP. CR(Sx) is set, for example, in the range of 0 to 1, and becomes larger the closer the reflection position is to the left or right edge of the screen. When the angle between the reflection direction RD and the upward direction of the screen exceeds 45 degrees, the value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP becomes smaller depending on the angle exceeded. When the angle between the reflection direction RD and the upward direction of the screen exceeds a threshold value (for example, 60 to 65 degrees), the degree of correction CR is set to "0."

[0077] The reflection direction is corrected toward the inside of the screen according to the correction degree CR. Specifically, a position in the virtual space that is a maximum distance Lb from the reflection position RP in the reflection direction RD is converted into the screen coordinate system, and the resulting position is calculated as the ray trace end position EP. The Sx coordinate value of this ray trace end position EP is corrected so as to move toward the inside of the screen according to the correction degree CR. For example, by linear interpolation, the Sx coordinate value of the ray trace end position EP is corrected so as to approach the Sx coordinate value of the reflection position RP. For example, if the reflection position RP is located within a predetermined range from the left edge of the screen, the ray trace end position EP is moved to the right (the positive direction of the Sx axis) according to the correction degree CR. The Sx coordinate value of the ray trace end position EP does not move beyond the Sx coordinate value of the reflection position RP. Furthermore, the Sy coordinate value of the ray trace end position EP does not change before and after the correction. The direction from the reflection position EP toward the moved ray trace end position EP' is then calculated as the ray trace direction RTD.

[0078] In this way, in areas near the edges of the screen, the direction in which the reflection direction is corrected is calculated as the ray tracing direction. This makes it possible to generate SSR reflections at the edges of the screen. For example, as shown in Figure 15, when ray tracing is performed in the reflection direction RD1 for a pixel at position RP1 on the left edge of the screen, the collision position of the ray may be outside the screen, and the color of the pixel at the collision position cannot be added to the pixel at position RP1. For this reason, if correction is not made to the reflection direction, the edges of the screen may end up in a different color from the rest of the screen, resulting in an unnatural image.

[0079] In this embodiment, the reflection direction is corrected based on the incident direction and normal direction, and the corrected direction is set as the ray tracing direction. This makes it easier to fit the ray collision position within the screen, and allows reflections to be generated based on the rendered image. The Sx coordinate value of the ray tracing end position EP is moved so that it approaches the Sx coordinate value of the reflection position RP. Therefore, even with maximum correction, the reflection direction that faces outside the screen will not face inside the screen, but will instead face upward on the screen. Therefore, for example, the color of a pixel to the right of the pixel at the left edge of the screen will not be reflected. This prevents objects from being unnaturally distorted and reflected on the ground.

[0080] In this embodiment, the closer the reflection position is to the edge of the screen, the greater the degree of correction, so that even near the edge of the screen, reflections can be generated based on pixels within the screen.

[0081] Furthermore, for example, if the reflection direction is directed inward from the screen, correcting the reflection direction may result in a reflection that is far removed from the actual reflection. In this embodiment, if the reflection direction is not directed outward from the screen, the reflection direction is not corrected, thereby preventing the generation of a reflection that is far removed from the actual reflection.

[0082] Furthermore, if the angle between the reflection direction and the top of the screen is relatively large (for example, if it exceeds 45 degrees), the degree of correction CR is reduced. If the angle between the reflection direction and the top of the screen exceeds a threshold, the degree of correction becomes 0. This prevents the generation of unnatural reflections due to the degree of correction being too large, for example, when the ground is sloped. Furthermore, correction for reflections on walls where the reflection direction is horizontal can be suppressed.

[0083] In this embodiment, the correction of the reflection direction is performed only on the left-right component in the screen coordinate system, which is suitable for games in which the virtual space extends left and right on the screen and is scrolled horizontally, for example.

[0084] (Generating reflections of character objects) Next, a method for generating reflections of the character objects 41 and 43 will be described. Fig. 17 is a diagram showing an example of a game image when the character object 41 is not present in front of the cylinder object 34. Fig. 18 is a diagram showing an example of a game image when the character object 41 has moved and is present in front of the cylinder object 34.

[0085] 17, when there is no character object 41 on the front side of the cylindrical object 34, for example, ray tracing is performed from the reflection position RPd on the front side, the collision position CPd is calculated, and the color of the collision position CPd is added to the color of the reflection position RPd. By performing the same process on each pixel, a reflection 54 of the cylindrical object 34 is displayed.

[0086] 18, on the other hand, the character object 41 is located in front of the cylindrical object 34, and the head of the character object 41 is located at the same position CPd as in FIG. 17. In this case, the color of the pixel at the collision position CPd of the cylindrical object 34 shown in FIG. 17 cannot be added to the pixel at the reflection position RPd, and no part of the cylindrical object 34 is reflected at the reflection position RPd. In this way, for example, when the character object 41 is located in front of the cylindrical object 34, it may not be possible to express a natural reflection that corresponds to the original positional relationship of the objects.

[0087] In this embodiment, in order to display the reflection of the character object and the object behind it, image processing is performed by the method shown in Fig. 19. Fig. 19 is a diagram showing an overview of image processing for generating the reflection of the character object and the object behind it.

[0088] 19, processor 81 first performs rendering of objects other than character objects (step S10). Here, a process of drawing a plurality of objects 30, 32, 34, and 36 arranged in the virtual space in the frame buffer is performed. The process of step S10 corresponds to the processes of steps S1 and S2 in FIG.

[0089] Next, the processor 81 renders the character object in a separate buffer and creates a character plane in which the rendered image is set as a texture (step S11).

[0090] FIG. 20 is a diagram showing an example of a character plane created in step S11. As shown in FIG. 20, character object 41 is rendered in a separate buffer, and the rendered image of character object 41 is set as a texture on a two-dimensional plane object, thereby creating character plane 65. Similarly, character plane 66 is created in which the rendered image of character object 43 is set as a texture. In this embodiment, a rendered image of each character object is generated for each frame. Note that the rendered image of each character object may be prepared in advance. For example, an image corresponding to each action of character object 41 may be stored in a storage medium in advance, and character plane 65 may be created in which the pre-stored image is set as a texture.

[0091] Next, processor 81 calculates the reflection color of objects other than the character object (step S12). Here, the reflection color is calculated for multiple objects 30, 32, 34, and 36 arranged in the virtual space. Specifically, processor 81 performs the processes of steps S3 to S5 in FIG. 5 described above. At this point, the character object has not been rendered in the frame buffer, so even if the character object is located in front of the object as viewed from virtual camera VC in the virtual space, the reflection color of the object can be calculated. The calculated reflection color of the object other than the character object (hereinafter referred to as the "first reflection color") is stored in the reflection buffer. For example, the reflection buffer stores RGB values and alpha values as color information indicating the first reflection color.

[0092] Next, processor 81 calculates the color of the character's reflection (step S13). Here, processor 81 calculates the color of the character object's reflection when the character plane created in step S11 is placed in the virtual space.

[0093] FIG. 21 is a diagram for explaining the calculation of the character reflection color in step S13. As shown in FIG. 21, the processor 81 calculates the intersection position of the character plane 65 and a ray when the character plane 65 is placed in the virtual space according to the position and posture of the character object 41 in the virtual space. Specifically, the processor 81 calculates the depth when the character plane 65 is placed in the virtual space, and calculates the intersection position CRPe between the ray traveling from the reflection position RPe in the ray tracing direction RTDe and the character plane 65. The color of the pixel at the intersection position CRPe is calculated as the character reflection color (hereinafter referred to as the "second reflection color"). Here, parts of the character plane 65 other than the image of the character object 41 are set transparent and do not intersect with the ray. For example, a ray traveling from the reflection position RPf in the ray tracing direction RTDf does not intersect with the image of the character object 41 in the character plane 65. In this case, the intersection position is not calculated. The calculated second reflection color is stored in the reflection buffer. For example, the reflection buffer stores RGB values and alpha values as color information indicating the color of the second reflection.

[0094] Returning to FIG. 19, the processor 81 next calculates the color reflected in the pixel of interest based on the results of steps S12 and S13 (step S14). For example, the color reflected in the pixel of interest is calculated by adding the second reflected color to the first reflected color. Here, "adding the second color to the first color" may mean mixing two colors. For example, "adding the second color to the first color" may mean performing alpha blending with the first color as the background color and the second color as the foreground color. Furthermore, "adding the second color to the first color" may mean performing alpha blending with the second color as the background color and the first color as the foreground color. Furthermore, "adding the second color to the first color" may mean mixing the first color and the second color based on their respective alpha values. Furthermore, "adding the second color to the first color" may mean overpainting the first color with the second color taking precedence over the first color.

[0095] Next, processor 81 renders the result calculated in step S14 in the frame buffer (step S15). Here, processor 81 adds the reflected color calculated in step S14 to the color of the pixel of interest stored in the frame buffer. For example, processor 81 performs alpha blending using the color of the pixel of interest as the background color and the reflected color calculated in step S14 as the foreground color. As a result, the character object and the reflection of objects other than the character object are drawn in the frame buffer.

[0096] After the process of step S15 is performed, processor 81 renders character objects 41 and 43 in the frame buffer (step S16), thereby generating a game image as shown in Fig. 22 and displaying it on the display device.

[0097] FIG. 22 is a diagram showing an example of a game image that is displayed when the image processing of FIG. 19 is performed. As shown in FIG. 22, a tree object 32 and its reflection 52 are displayed. Also, a character object 43 and its reflection 63 are displayed. Also, a cylinder object 34 and its reflection 54 are displayed. A character object 41 exists in front of the cylinder object 34, and the character object 41 and its reflection 61 are displayed. Although the reflection 54 of the cylinder object 34 and the reflection 61 of the character object 41 partially overlap, both of these have shapes that correspond to their actual shapes and positional relationships, making it possible to express a natural reflection.

[0098] In this embodiment, the character object is a flat-shaped object. As described above, when a reflection of the character object is generated based on a character plane in which an image of the character object is set as a texture, the reflection of the character object appears flat. If the character object itself is a three-dimensional object with a certain thickness, the image may appear unnatural if the reflection appears flat. However, in this embodiment, the character object itself is a flat-shaped object, so the image can appear natural.

[0099] (Image processing details) Next, the above-mentioned image processing will be described in detail. First, the data stored in the memory of the main device 2 (memory in the processor 81, DRAM 85, flash memory 84, external storage medium, etc.) will be described. Fig. 23 is a diagram showing an example of data stored in the memory of the main device 2.

[0100] 23, a program, operation data, object data, character data, and character plane data are stored in the memory of main unit 2. In addition, a first depth buffer, a second depth buffer, a normal buffer, a reflection buffer, and a frame buffer are stored in the memory of main unit 2.

[0101] The program is a program for executing the main processing described below, and includes an image processing program for performing the image processing related to the above-mentioned reflections. The program is stored in advance in an external storage medium inserted into the slot 23 or in the flash memory 84, and is read into the DRAM 85 when the game is executed. The program may also be obtained from another device via a network (for example, the Internet).

[0102] The operation data is data related to operations acquired from the left controller 3 and the right controller 4. For example, the operation data is transmitted from the left controller 3 and the right controller 4 to the main unit 2 at predetermined time intervals (for example, 1 / 200 second intervals) and stored in memory.

[0103] The object data is data relating to each object (30, 32, 34, 36) other than the character object placed in the virtual space. The object data includes data relating to the position and orientation of each object in the virtual space, data relating to the shape, and data relating to the texture. The object data also includes information relating to whether or not each object is set as a specific object. For example, the tree object 32 and the cylinder object 34 are set as specific objects. On the other hand, the mountain object 36 is not set as a specific object.

[0104] The character data is data related to the character objects 41 and 43. The character data includes data related to the position and posture of each character in the virtual space, data related to the shape, and data related to the texture. In this embodiment, each character object is formed as a 3D model, but is formed as a planar object that is thinner in the thickness direction than in the vertical and horizontal directions.

[0105] The character plane data is a two-dimensional object, and is data relating to a plane object having a texture set to an image obtained by rendering a character object. The character plane data includes data relating to character plane 65 having a texture set to an image of character object 41, and data relating to character plane 66 having a texture set to an image of character object 43.

[0106] The first depth buffer is a buffer that stores depth values for each pixel that are generated and updated based on the results of the first depth test. The first depth buffer stores depth values for all objects other than character objects that are included in the imaging range of the virtual camera VC.

[0107] The second depth buffer is a buffer that stores depth values for each pixel that are generated and updated based on the results of the second depth test. The second depth buffer stores depth values for specific objects included in the imaging range of the virtual camera VC.

[0108] The normal buffer is a buffer that stores normal information for each pixel.

[0109] The reflection buffer is a buffer for storing the first reflection color (the reflection color of an object other than a character object) and the second reflection color (the reflection color of a character object). In the ray tracing process described below, the reflection buffer stores the first reflection color when a ray collides with an object other than a character object, and stores the second reflection color when the ray intersects with a character plane. As will be described in detail later, the reflection buffer may store multiple reflection colors.

[0110] The frame buffer is a buffer that stores game images to be displayed. The images stored in the frame buffer are output to the display device at a predetermined timing and displayed on the display device.

[0111] (Details of Game Processing on Main Unit 2) Next, details of the processing performed in the main unit 2 will be described with reference to Fig. 24 to Fig. 27. Fig. 24 is a flowchart showing an example of main processing executed by the processor 81 of the main unit 2. The processing shown in Fig. 24 to Fig. 27 is performed by the CPU or GPU of the processor 81.

[0112] 24, processor 81 first executes initial processing (step S100). Specifically, processor 81 sets up a three-dimensional virtual space and arranges in the virtual space the objects (30, 32, 34, 36), the character objects (41, 43), the virtual camera VC, the light source, and various other objects used in the game. After executing the initial processing, processor 81 repeatedly executes the processes of the following steps S101 to S108 at predetermined frame time intervals (for example, 1 / 60 second intervals).

[0113] In step S101, the processor 81 acquires operation data from the controller.

[0114] Next, processor 81 performs game processing based on the acquired operation data (step S102). For example, processor 81 moves character object 41 within the virtual space and causes character object 41 to perform a predetermined action (e.g., a jump action, an attack action, etc.) based on the operation data. Processor 81 also moves character object 43 within the virtual space and causes character object 43 to perform a predetermined action according to a predetermined algorithm. Processor 81 also controls enemy objects other than the character object within the virtual space and moves obstacle objects that obstruct the character object within the virtual space.

[0115] Next, the processor 81 performs rendering processing for objects other than the character object (step S103). Specifically, the processor 81 performs a first depth test for each object other than the character objects 41 and 43 included in the imaging range of the virtual camera VC, and updates the first depth buffer. The processor 81 also draws each object in the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering is used as the rendering method. In step S103, the normal buffer is also updated. Note that forward rendering may also be used as the rendering method.

[0116] Next, processor 81 renders the character object in a separate buffer to create a character plane (step S104). Specifically, processor 81 renders character object 41 in a separate buffer and sets the image of character object 41 rendered in the separate buffer as a texture to a plane object, thereby creating character plane 65. Similarly, processor 81 also creates character plane 66 by setting the image of character object 43 as a texture to a plane object.

[0117] After step S104, the processor 81 performs SSR processing (step S105). The SSR processing is processing for drawing the reflections of all objects, including character objects, in the frame buffer. The SSR processing will be described in detail below.

[0118] (SSR processing) FIG. 25 is a flowchart showing an example of the SSR process in step S105.

[0119] 25, the processor 81 performs a second depth test on a specific object and updates the second depth buffer (step S121). For example, the processor 81 performs a depth test on a tree object 32 and a cylinder object 34 that have been set as specific objects in advance, and updates the second depth buffer. Note that the setting of the specific object is performed on a mesh-by-mesh basis.

[0120] Next, the processor 81 performs ray tracing (step S122). The ray tracing is performed on each pixel of interest. Here, the color of the reflection on each pixel of interest is calculated. The ray tracing may be performed on all pixels, or only on pixels in a specific range. For example, the load may be reduced by excluding areas where no model is rendered, areas where SSR is disabled on a model or mesh basis, areas where the normal direction is outside a specific range, etc. The ray tracing is described in detail below.

[0121] (Ray tracing processing) FIG. 26 is a flowchart showing an example of the ray tracing process in step S122.

[0122] 26, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position and calculates the reflection direction (step S131). Specifically, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position based on the position of the pixel of interest in the screen coordinate system and the depth of the pixel of interest stored in the first depth buffer. The processor 81 also calculates the reflection direction based on the incident direction and the normal direction of the reflection position, assuming that the direction from the position of the virtual camera VC toward the reflection position is the incident direction.

[0123] Next, the processor 81 sets the calculated reflection direction or the direction obtained by correcting the reflection direction as the ray tracing direction (step S132). Here, if the correction conditions are satisfied, the direction obtained by correcting the reflection direction is set as the ray tracing direction, and if the correction conditions are not satisfied, the reflection direction is set as the ray tracing direction. Also, the ray tracing end position is set. The correction conditions and correction method are as described above.

[0124] After step S132, the processor 81 sets the first depth buffer as a depth buffer for determination and starts ray tracing from the pixel of interest (step S133). The ray tracing is performed in the screen coordinate system. Next, the processor 81 advances the ray a predetermined distance (step S134).

[0125] Next, the processor 81 determines whether or not the tracing distance exceeds the maximum distance Lb (step S135). Specifically, the processor 81 determines whether or not the tip of the ray in the screen coordinate system has reached the ray tracing end position.

[0126] If it is determined that the tracing distance exceeds Lb (step S135: YES), the processor 81 then performs the process of step S141.

[0127] On the other hand, if the tracing distance is equal to or less than Lb (step S135: NO), the processor 81 determines whether or not the ray has collided with an object (step S136). Specifically, the processor 81 uses the determination depth buffer to determine whether or not the tip of the ray has collided with an object.

[0128] If it is determined that the ray has not collided with an object (step S136: NO), the processor 81 determines whether the trace distance is equal to or greater than La (step S137).

[0129] If the tracing distance is equal to or greater than La (step S137: YES), the processor 81 changes the determination depth buffer to the second depth buffer (step S138) and executes the process of step S134 again. On the other hand, if the tracing distance is less than La (step S137: NO), the processor 81 maintains the determination depth buffer and executes the process of step S134 again.

[0130] On the other hand, if it is determined that the ray has collided with the object (step S136: YES), the processor 81 calculates a first reflection color from the color of the collision position of the ray (step S139). For example, the processor 81 may calculate the color of the pixel at the collision position stored in the frame buffer as the first reflection color, or may calculate the color of the pixel at the collision position by applying a predetermined process to the color of the pixel at the collision position as the first reflection color. The color of the first reflection differs when the collision is determined based on the first depth buffer and when the collision is determined based on the second depth buffer. For example, when the first depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position may be lightened (the alpha value may be reduced) depending on the tracing distance, and the changed color may be calculated as the first reflection color. Furthermore, when the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position may be calculated as the first reflection color regardless of the tracing distance. Furthermore, when the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position is changed to a lighter color according to the tracing distance, but may be changed to a darker color than when the first depth buffer is set as the depth buffer for determination, and the changed color may be calculated as the first reflection color. Then, processor 81 stores the calculated first reflection color in the reflection buffer. Here, the first reflection color stored in the reflection buffer when it is determined that a ray has collided based on the first depth buffer is referred to as the "first reflection color (first depth buffer)." Furthermore, the first reflection color stored in the reflection buffer when it is determined that a ray has collided based on the second depth buffer is referred to as the "first reflection color (second depth buffer)."

[0131] Next, processor 81 determines whether the determination depth buffer is the first depth buffer and whether the color of the first reflection calculated in step S139 is translucent (step S140). For example, if the tracing distance is less than or equal to La and the color of the first reflection is translucent, ray tracing is performed further to the back. Translucency means that the alpha value of the reflection is less than 1.0, and the alpha value is calculated independently based on the distance La, the collision position, and various other parameters.

[0132] If the determination in step S140 is NO, the processor 81 proceeds to step S141. If the determination in step S140 is YES, the processor 81 proceeds to step S138.

[0133] In step S141, processor 81 calculates the reflection color of the character object (second reflection color). Details of the process of step S141 will be described below.

[0134] FIG. 27 is a flowchart showing an example of the process of calculating the color of the reflected light of the character in step S141.

[0135] 27, processor 81 calculates the intersection position between the ray and the character plane (step S151). Specifically, processor 81 calculates the depth when the character plane is placed according to the position and orientation of the character object in the virtual space, and determines whether the ray used in the ray tracing intersects with the character plane. If the ray intersects with the character plane, processor 81 calculates the intersection position.

[0136] Next, processor 81 stores the color of the intersection position in the reflection buffer as the second reflection color (step S152). Specifically, processor 81 stores a color based on the color of a pixel of the texture image of the character object corresponding to the intersection position as the second reflection color. Note that the second reflection color may be the color of a pixel of the texture image of the character object, or may be a color calculated by applying predetermined processing to the color of the pixel. When processor 81 has performed the processing of step S152, it ends the processing shown in FIG. 27 and returns the processing to FIG. 26. Note that when the ray intersects with multiple character planes, multiple second reflection colors are stored in the reflection buffer. In this case, the multiple second reflection colors may be stored in the reflection buffer in order of proximity of the intersection position to the virtual camera.

[0137] By performing the processes of steps S139 and S152, the reflection buffer stores from 0 to a plurality of reflection colors. For example, the reflection buffer may store reflection colors corresponding to the collision position or intersection position of a ray in order of proximity to the virtual camera. For example, if it is determined that a ray has collided at a first position where the tracing distance is less than La and that the ray has collided at a second position where the tracing distance is equal to or greater than La and less than Lb, and the ray intersects with the character plane between the first and second positions, the reflection buffer stores the first reflection color (first depth buffer), the second reflection color, and the first reflection color (second depth buffer) in order of proximity to the virtual camera (see FIG. 23). Note that if the ray does not collide with any object or intersect with the character plane, no reflection color is stored in the reflection buffer.

[0138] Returning to FIG. 26, after processing step S141, processor 81 calculates the color reflected in the pixel of interest based on the reflection colors stored in the reflection buffer (step S142). If multiple reflection colors are stored in the reflection buffer, processor 81 may calculate the color reflected in the pixel of interest by alpha blending the multiple reflection colors in order of distance from the virtual camera. For example, as shown in FIG. 23, if the reflection buffer stores a first reflection color (first depth buffer), a second reflection color, and a first reflection color (second depth buffer) in order of distance from the virtual camera, processor 81 may calculate a color by alpha blending the first reflection color (second depth buffer) with the second reflection color, and then alpha blending the calculated color with the first reflection color (first depth buffer), to calculate the color reflected in the pixel of interest. Furthermore, for example, when the reflection buffer stores the second reflection color and the first reflection color in order of proximity to the virtual camera (i.e., when there is no object in front of the character as viewed from the virtual camera, and there is an object behind the character), processor 81 may calculate the color reflected in the pixel of interest by alpha blending the second reflection color with the first reflection color. For example, when the second reflection color in the front is opaque, the second reflection color is set as the color reflected in the pixel of interest. When the second reflection color in the front is semi-transparent, a color obtained by blending the first reflection color and the second reflection color is calculated as the color reflected in the pixel of interest. Processor 81 stores the calculated reflection color in memory.

[0139] If the process of step S142 has been performed, processor 81 ends the process of FIG. 26 and returns the process to FIG.

[0140] 25, after step S122, the processor 81 performs rendering to the frame buffer based on the result of the ray tracing process (step S123). For example, the processor 81 adds the reflected color calculated in step S142 to the color of the pixel of interest stored in the frame buffer. As a result, the reflected color calculated in step S142 is reflected in the color of the pixel of interest.

[0141] Next, processor 81 determines whether the processes of steps S122 and S123 have been performed for all pixels (step S124). If processor 81 determines NO in step S124, it executes the process of step S122 again. By performing the processes of steps S122 and S123 for all pixels, reflections (52, 54) of objects other than the character object arranged in the virtual space and reflections (61, 63) of the character object are drawn in the frame buffer. If processor 81 determines YES in step S124, it ends the process of FIG. 25 and returns the process to FIG. 24.

[0142] Returning to FIG. 24, after step S105, the processor 81 further renders the character objects 41 and 43 in the frame buffer (step S106).

[0143] Next, processor 81 outputs the image stored in the frame buffer to the display device (step S107), thereby displaying the game image.

[0144] Next, processor 81 determines whether to end the game (step S108). For example, if the player instructs to end the game, processor 81 ends the game processing shown in FIG. 24. If processor 81 determines not to end the game (step S108: NO), it executes the processing of step S101 again. This concludes the description of the main processing shown in FIG. 24.

[0145] The order of processing, content, values used for determination, etc. in the above flowchart are merely examples, and may be changed as appropriate.

[0146] As described above, in this embodiment, when the tracing distance is less than La, ray collision determination is performed based on the first depth buffer, and when the tracing distance is equal to or greater than La, ray collision determination is performed based on the second depth buffer (steps S135 to S140). This makes it possible to display a reflection even when the tracing distance is long for a specific object. Furthermore, it is possible to prevent reflections from being displayed for objects other than the specific object, making it possible to reflect or not reflect a desired object. Furthermore, by generating a reflection based on the first depth buffer and the second depth buffer, and by fading the color of the reflection according to the tracing distance when based on the first depth buffer, it is possible to make a specific object appear large and clear, and other objects appear small.

[0147] In this embodiment, if the pixel of interest (reflection position) is close to the edge of the screen, the direction obtained by correcting the reflection direction based on the incident direction and the normal direction at the reflection position is set as the ray tracing direction (step S132). This makes it possible to calculate the color of the reflection based on the pixels drawn in the frame buffer, and to generate reflection even at the edge of the screen.

[0148] In this embodiment, objects other than the character object are first drawn in the frame buffer, then the reflection of the object is drawn, then the reflection of the character object is drawn, and finally the character object is drawn. This makes it possible to display the reflection of the character object as well as the reflection of objects behind the character object as seen from the virtual camera.

[0149] In this embodiment, the color of the collision position calculated based on the first depth buffer, the color of the collision position calculated based on the second depth buffer, and the color of the intersection with the character plane are calculated, and these colors are alpha blended in order from the back. This allows the reflection of each object to be displayed even if the objects overlap when viewed from the virtual camera.

[0150] (Variation) The image processing of this embodiment has been described above, but the above embodiment is merely an example, and the following modifications may be made, for example.

[0151] For example, in the above embodiment, deferred rendering was used as the rendering method, but in other embodiments, forward rendering may be used. When forward rendering is used, in the above SSR processing, for example, the normal direction of the reflection position corresponding to the pixel of interest may be estimated based on the depths of multiple pixels surrounding the pixel of interest, and the reflection direction may be calculated based on the incident direction and the normal direction. Alternatively, the orientation of the mesh in virtual space may be calculated, the normal direction of the reflection position corresponding to the pixel of interest may be calculated, and the reflection direction may be calculated based on the normal direction.

[0152] Furthermore, in the above embodiment, the reflection direction is corrected for pixels in a region within a predetermined range from the left or right edge of the screen. However, in other embodiments, the reflection direction may also be corrected for pixels in a region within a predetermined range from the top or bottom edge of the screen. In the above embodiment, the reflection direction is corrected by moving the position of the ray tracing end position in the Sx-axis direction in the screen coordinate system closer to the position of the reflection position in the Sx-axis direction. In other embodiments, the reflection direction may also be corrected by moving the position of the ray tracing end position in the Sy-axis direction in the positive or negative direction of the Sy-axis. The reflection direction may also be corrected by moving the ray tracing end position in the horizontal and / or vertical directions so that the collision position of the ray falls within the screen.

[0153] In addition, while in the above embodiment ray tracing is performed in the screen coordinate system, in other embodiments ray tracing may be performed in the xyz coordinate system of virtual space. That is, rays may be extended in the xyz coordinate system of virtual space, the collision position of the ray may be calculated, and the color of the pixel at the position corresponding to the collision position may be calculated as the color of the reflection.

[0154] In the above embodiment, a character plane is created in which an image of the character object is set as a texture on a planar object, and the character plane is used to calculate the color of the character object's reflection. In other embodiments, an image of the character object may be set as a texture on a three-dimensional object, and the three-dimensional object may be used to calculate the color of the character object's reflection. In the above embodiment, the character object is a flat-shaped object, but in other embodiments, the character object may be a three-dimensional object with thickness.

[0155] In addition, in the above embodiment, the ground is used as a reflective surface and an object is reflected on the ground, but a surface other than the ground (for example, a wall surface or a ceiling surface) may be used as a reflective surface and the object may be reflected using the above-mentioned processing.

[0156] Furthermore, in the above embodiment, a game image is generated, but the image processing described above may be used when generating any image other than a game image.

[0157] Furthermore, the above hardware configuration is merely an example, and the image processing may be performed by any other hardware. For example, the processing may be performed by any information processing device, such as a personal computer, a tablet terminal, a smartphone, or a server on the Internet. The image processing may also be performed in an information processing system including multiple devices. The image processing may be performed in a distributed manner by multiple devices. The device performing the image processing may be different from the device displaying the image. For example, a first device (e.g., a server) may generate an image by performing part or all of the image processing, and transmit the generated image to a second device via a network (e.g., the Internet or a LAN), and the second device may display the image.

[0158] Furthermore, the configurations according to the above-described embodiments and their modifications can be combined in any manner as long as they are not inconsistent with each other. Furthermore, the above is merely an example of the present invention, and various other improvements and modifications may be made thereto. [Explanation of symbols]

[0159] 1. Game System 2 Main unit 81 processors 32 Tree Objects 34 Cylinder Object 36 Mountain Objects 41, 43 Character Object

Claims

1. The computer of the information processing device About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; causing a frame buffer to perform drawing based on the results of the first depth test; performing a second depth test and updating the second depth buffer for the first type of objects among the objects; For each pixel in the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than the first distance, or based on the depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; an image processing program that, when the collision position is determined within a range in which the tracing distance is within the first distance, determines a color based on the color of a pixel in the frame buffer that corresponds to the collision position as a reflection color to be added to the color of the pixel of interest.

2. The computer further comprises:

2. The image processing program according to claim 1, wherein the color of the reflection to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position is determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

3. The computer further comprises: For a second type of object, a plane object having a texture set to an image that is rendered without drawing to a frame buffer is placed at the position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color is determined to be added to the color of the pixel of interest based on the color of the texture at the intersection position; Reflecting the color of the reflection in the frame buffer; The image processing program according to claim 1 , wherein the second type of object is drawn in a frame buffer after the reflection color is reflected.

4. The computer further comprises:

2. The image processing program according to claim 1, further comprising: calculating, as the ray tracing direction, a direction obtained by further correcting a reflection direction based on the incident direction and the normal direction of the reflection position, the reflection direction being directed inward on the screen.

5. The image processing program according to claim 3 , wherein the second type of object is a flat-shaped object.

6. 1. An image processing system comprising a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; drawing to a frame buffer based on the results of the first depth test; performing a second depth test and updating a second depth buffer for a first type of object among the objects; For each pixel in the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculate, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than the first distance, or based on the depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the tracing distance is within the first distance, an image processing system determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as the reflection color to be added to the color of the pixel of interest.

7. The processor further comprises:

7. The image processing system of claim 6, wherein the reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position is determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

8. The processor further comprises: for a second type of object, placing a plane object having a texture set to an image that is rendered without drawing to a frame buffer at a position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color to be added to the color of the pixel of interest is determined based on the color of the texture at the intersection position; reflecting the color of the reflection in the frame buffer; The image processing system according to claim 6 , wherein the second type of object is drawn in a frame buffer after the reflection color is reflected.

9. The processor further comprises: The image processing system according to claim 6 , wherein the ray tracing direction is calculated by adding a correction directed inward on the screen to a reflection direction based on the incident direction and the normal direction of the reflection position.

10. The image processing system of claim 8 , wherein the second type of object is a flat-shaped object.

11. 1. An image processing method, comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; drawing to a frame buffer based on the results of the first depth test; performing a second depth test using a second depth buffer and updating the second depth buffer for the first type of objects among the objects; For each pixel in the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculating, based on the depth in the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than the first distance, and based on the depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the tracing distance is within the first distance, a color based on the color of a pixel in the frame buffer corresponding to the collision position is determined as a reflection color to be added to the color of the pixel of interest.

12. 12. The image processing method according to claim 11, further comprising: determining a reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position, the reflection color being determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

13. For a second type of object, placing a planar object having a texture set to an image rendered without drawing to a frame buffer at the position of the second type of object in the virtual space; determining a reflection color to be added to the color of the pixel of interest based on the color of the texture at the intersection position when the ray intersects with the planar object; Reflecting the color of the reflection in the frame buffer; The image processing method according to claim 11 , further comprising: drawing the second type object in a frame buffer after reflecting the color of the reflection.

14. The image processing method according to claim 11 , further comprising: calculating, as the ray tracing direction, a direction obtained by further correcting a reflection direction based on the incident direction and a normal direction of the reflection position, with the correction being directed inward on the screen.

15. The image processing method according to claim 13 , wherein the second type of object is a flat-shaped object.

16. An image processing device including a processor, the processor comprising: About objects in virtual space performing a first depth test using a first depth buffer and updating the first depth buffer; drawing to a frame buffer based on the results of the first depth test; performing a second depth test and updating a second depth buffer for a first type of object among the objects; For each pixel in the frame buffer on which drawing has been performed, the pixel is designated as a pixel of interest, calculate, based on the depth of the first depth buffer, a direction from a virtual camera toward a position in the virtual space corresponding to the pixel of interest as an incident direction, and a direction of a ray reflected from the position as a reflection position as a ray tracing direction; tracing the ray along the ray tracing direction, and determining a collision position where the ray collides with an object in the virtual space based on the depth of the first depth buffer when the tracing distance of the ray is less than a second distance that is smaller than the first distance, or based on the depth of the second depth buffer when the tracing distance of the ray is equal to or greater than the second distance; When the collision position is determined within a range where the tracing distance is within the first distance, an image processing device determines a color based on the color of a pixel in the frame buffer corresponding to the collision position as the reflection color to be added to the color of the pixel of interest.

17. The processor further comprises:

17. The image processing device according to claim 16, wherein the reflection color to be added to the color of the pixel of interest based on the color of the pixel in the frame buffer corresponding to the collision position is determined based on parameters that differ between the collision position based on the first depth buffer and the collision position based on the second depth buffer.

18. The processor further comprises: for a second type of object, placing a plane object having a texture set to an image that is rendered without drawing to a frame buffer at a position of the second type of object in the virtual space; When the ray intersects with the planar object, a reflection color to be added to the color of the pixel of interest is determined based on the color of the texture at the intersection position; reflecting the color of the reflection in the frame buffer; The image processing device according to claim 16 , wherein the second type object is drawn in a frame buffer after the reflection color is reflected.

19. The processor further comprises: The image processing device according to claim 16 , wherein a direction obtained by further correcting a reflection direction based on the incident direction and the normal direction of the reflection position, toward the inside of the screen, is calculated as the ray tracing direction.

20. The image processing device according to claim 18 , wherein the second type of object is a flat-shaped object.

Images