Image processing program, image processing apparatus, image processing system, and image processing method

Abstract

To provide an image processing program, an image processing system, an image processing device, and an image processing method capable of expressing an object having transparency with a low processing load.SOLUTION: An image processing device manages a first texture including first color information, a second texture including second color information, and a third texture including height information. The image processing device acquires the first color information of the first texture corresponding to a surface of an object, acquired on the basis of on a first correspondence. The image processing device corrects, on the basis of a relation between a visual line direction of a virtual camera and an orientation of a surface of an object, and height information identified on the basis of a third correspondence, a second correspondence which is a relation between the surface of the object and the second texture, and acquires the second color information from the second texture on the basis of the corrected second correspondence. The image processing device then uses the acquired color information to determine a drawing color and draws the surface of the object.SELECTED DRAWING: Figure 22

Description

Technical Field

[0001] The present invention relates to image processing for expressing a three-dimensional object having a sense of transparency.

Background Art

[0002] Conventionally, as one method of rendering a three-dimensional virtual object, there is a method called texture mapping in which a texture image is attached to the object and rendered (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the above texture mapping, it is possible to enhance the expressiveness of the texture of the surface of an object. However, when expressing a transparent object such as an object with a translucent surface, this method may not be sufficient in some cases. Therefore, when trying to draw a transparent object, it has been common to perform a transparency process using, for example, the α-blending method or the like. However, conventional transparency processes generally have a high processing load. For example, assume a case where the surface of the first object is transparent and the second object on the other side is visible. In this case, first, the second object, which is the object to be transparent, is drawn. Next, in front of the second object, the first object, which is a translucent object, is placed and drawn. Alternatively, the translucent first object is placed so as to cover the second object and drawn. At this time, furthermore, various effects are also applied to the surface portion of the first object and drawn. Also, for the second object, there are cases where an effect that distorts the image further is applied and drawn. In such a process, the drawing order of the objects is important, but generally, its management is complicated, or the drawing order interferes with each other, resulting in a high processing load. As a result, in some cases, a broken image may be displayed.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing apparatus, an image processing system, and an image processing method that can express a transparent object with a low processing load.

Means for Solving the Problems

[0006] In order to achieve the above object, for example, the following configuration examples can be cited.

[0007] (Configuration 1) Configuration 1 is an image processing program for rendering an object arranged in a virtual space, which causes a computer to function as a management means, a virtual camera control means, and a rendering means. The management means manages a first texture including first color information associated with the surface of the object based on a first correspondence relationship, a second texture including second color information associated with the surface of the object based on a second correspondence relationship, and a third texture including height information associated with the surface of the object based on a third correspondence relationship. The virtual camera control means controls the line-of-sight direction of a virtual camera arranged in the virtual space. The rendering means determines a rendering color using at least (1) the first color information of the first texture corresponding to the surface of the object obtained based on the first correspondence relationship, and (2) the second color information of the second texture corresponding to the surface of the object obtained based on a corrected correspondence relationship obtained by correcting the second correspondence relationship based on the relationship between the line-of-sight direction of the virtual camera and the orientation of the surface of the object and the correction based on the height information specified based on the third correspondence relationship. Then, the rendering means renders the surface of the object using the rendering color.

[0008] According to the above configuration example, in the process of determining the rendering color of each pixel, it is possible to represent an object with transparency without performing a transparency process or the like that actually makes the surface of the object transparent. Therefore, the representation of an object with transparency can be realized with a process having a low load.

[0009] (Configuration 2) In Configuration 2, in the above Configuration 1, the rendering means may obtain the second color information by shifting the position in the second texture referred to during rendering from the position specified based on the second correspondence relationship to a position further shifted based on the above correction.

[0010] According to the above configuration example, the second correspondence relationship is corrected for each pixel. That is, it is possible to perform rendering that accurately reflects the positional relationship between the virtual camera and the object for each pixel.

[0011] (Configuration 3) In Configuration 3, in the above Configuration 2, the drawing means may determine the direction and amount of displacement of the position within the second texture referred to during drawing based on the relative relationship between the line-of-sight direction of the virtual camera and the orientation of the surface of the object.

[0012] According to the above configuration example, the displacement direction and amount are determined in consideration of the relationship between the orientation (line-of-sight direction) of the virtual camera and the orientation of the object surface. Thereby, in the expression of a transparent object, an image with less visual discomfort can be provided. (Configuration 4) In Configuration 4, in the above Configuration 2, the drawing means may determine the amount of displacement such that the closer the line-of-sight direction of the virtual camera is to the opposite direction of the normal direction of the object surface, the smaller the amount of displacement.

[0013] According to the above configuration example, for example, the closer the line-of-sight direction of the virtual camera approaches horizontal with respect to the object (surface), the larger the amount of displacement. Thereby, the expression of transparency taking into account the unevenness and height difference of the object surface can be made richer.

[0014] (Configuration 5) In Configuration 5, in any of the above Configurations 1 to 4, the image processing program may further cause the computer to function as a light source arrangement means for arranging a virtual light source in the virtual space. And the drawing means may determine the drawing color by synthesizing at least the first color information and the second color information at a predetermined ratio. At this time, when the light source is located on the back side of the object surface, the drawing color may be determined such that the ratio of the second color information is higher than when the light source is located on the front side of the object surface.

[0015] According to the above-described configuration example, the color composition ratio of the colors related to the first color information and the second color information is determined according to the positional relationship between the light source and the virtual camera. Specifically, if the positional relationship with the light source is a backlit relationship, the ratio of the color related to the second color information is synthesized to be higher than in the case of frontlighting. Therefore, in the case of backlighting, the color related to the second color information can be expressed more strongly than in the case of frontlighting. By considering such a positional relationship with the light source, a more rich expression of transparency becomes possible.

Effect of the Invention

[0016] According to the present embodiment, the expression of a transparent object can be realized with a low-load process.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0018] Hereinafter, an embodiment will be described.

[0019] [Hardware Configuration of Information Processing Device] First, an information processing device for executing the information processing according to this embodiment will be described. The information processing device is, for example, a smartphone, a stationary or portable game device, a tablet terminal, a mobile phone, a personal computer, a wearable terminal, etc. Further, the information processing according to this embodiment is also applicable to a game system composed of the above-described game device or the like and a predetermined server. In this embodiment, a stationary game device (hereinafter simply referred to as a game device) will be described as an example of the information processing device.

[0020] FIG. 1 is a block diagram showing an example of the internal configuration of a game device 2 according to the present embodiment. The game device 2 includes a processor 81. The processor 81 is an information processing unit that executes various information processes executed in the game device 2. For example, it may be composed of only a CPU (Central Processing Unit), or may be composed of a SoC (System-on-a-chip) including a plurality of functions such as a CPU function and a GPU (Graphics Processing Unit) function. The processor 81 executes various information processes by executing an information processing program stored in the storage unit 84. Note that the storage unit 84 may be, for example, an internal storage medium such as a flash memory or a DRAM (Dynamic Random Access Memory), or may be configured to use an external storage medium or the like mounted in a slot (not shown).

[0021] The game device 2 also includes a controller communication unit 86 for the game device 2 to perform wired or wireless communication with the controller 4. Although not shown in the figure, the controller 4 is provided with various buttons such as a cross key and ABXY buttons, and an analog stick and the like.

[0022] The game device 2 is also connected to a display unit 5 (for example, a TV or the like) via an image and audio output unit 87. The processor 81 outputs the generated image and audio (for example, by executing the above information process) to the display unit 5 via the image and audio output unit 87.

[0023] Hereinafter, an example of the process according to the present embodiment will be described. The process according to the present embodiment relates to the drawing of (three-dimensional) objects. More specifically, it relates to a drawing process for expressing a transparent object. Further, the drawing process can be performed, for example, as part of a game process.

[0024] Generally, a technique of using a texture image to represent the texture or the like of the surface of an object is known (so-called texture mapping). In this embodiment, in order to represent the transparency of a certain object, at least two textures are used. Conceptually, the first texture (hereinafter referred to as the first texture) is an image for representing the surface of the object (so to speak, the color of the foreground). Conceptually, the second texture (hereinafter referred to as the second texture) is an image assuming that it is located inside the object, and is an image when it is assumed that the surface of the object is seen through. That is, it is an image for representing the inside of the object that can be seen through (so to speak, the color of the background on the other side of the transparency). When drawing an object (each pixel corresponding thereto), for the pixels of the portion where transparency is desired in the first texture, the color of the second texture is blended and drawn. By adjusting the blending ratio at this time, for the relevant portion of the first texture image, an expression can be achieved such that the image related to the second texture can be seen through. For example, if the blending ratio of the color of the first texture is increased, a semi-transparent feeling can be expressed such that the image of the second texture can be faintly seen through. Also, if the blending ratio of the second texture is increased, a feeling that the surface of the object is more transparent can be expressed.

[0025] Even with the above-described method, a certain degree of transparency can be represented for the surface of the object. However, in this embodiment, through the processing described below, it is possible to represent an object with a higher degree of transparency. Although the details will be described later, in this embodiment, only for the above-described second texture, a process of performing rendering using a so-called parallax mapping technique to make it look more three-dimensional is performed. Thereby, a higher degree of transparency is given to the surface of the object, and the sense of depth and three-dimensionality of the inside of the object that can be seen through (and is desired to be shown as such) can be more effectively represented. Also, such a representation of a transparent object can be realized with a lower processing load compared to the case of using a transparency process.

[0026] Using FIGS. 2 to 11, an example of an object image when the process according to this embodiment is not used and an example of an object image when the process according to this embodiment is used are shown. Here, as an example, an example of drawing a columnar object that is vertically long and rectangular as shown in FIG. 2 is shown. In the column object, a square opening is provided on the first surface, and a round opening is provided on the second surface. Also, as the movement of the virtual camera, assume a case where it moves from a state of imaging from a height that slightly looks down on the first surface so as to wrap around to the second surface (see FIG. 3). Also, in this example, it is assumed that the inner wall portion of these openings is expressed so as to be seen through.

[0027] Also, as the first texture, a texture as shown in FIG. 4 is prepared, and as the second texture, a texture as shown in FIG. 5 is prepared. Although they are schematically shown figures, the first texture is an image corresponding to the surface portion of the column object, and can be said to be a basic image. In FIG. 4, the first texture is an image in which a square opening and a round opening are represented. Also, for simplicity of explanation, assume that the color of the column object in the first texture is a predetermined single color. That is, assume that the first texture is a single-color image.

[0028] On the other hand, the second texture is an image assuming an inner wall that can be seen through from the opening. In this example, the second texture is an image in which a portion corresponding to the "edge" of the opening is expressed using a color different from the predetermined single color. For example, assume that the predetermined single color in the first texture is orange, and the "edge" portion of the second texture is white. Also, assume that the color of the second texture other than the "edge" is the same as the first texture.

[0029] An example is shown of the case where a pillar object is drawn without performing the process according to this embodiment using the texture as described above. FIGS. 6 to 8 are examples of the screen in such a case. In FIG. 6, the virtual camera is in a state of imaging from a height such that it slightly looks down diagonally upward from the right on a square opening. In FIG. 6, for the location (pixel) that forms the "edge" of the square opening, the color of the pixel on the first texture corresponding to that location (hereinafter referred to as the first color) and the color of the pixel on the second texture corresponding to that location (hereinafter referred to as the second color) are blended, and the blended color is used as the drawing color. Also, for the round opening in the upper right, it shows that the "edge" part is drawn with the drawing color in which the second color is blended with the first color.

[0030] When the virtual camera makes a movement such that it wraps around to the second surface side from the state of FIG. 6, images as shown in FIGS. 7 to 8 are displayed. In any of the figures, for the location that forms the "edge" of each opening, it is drawn with the drawing color in which the second color is blended with the first color. Therefore, these figures show that the "edge" part of each opening has an expression as if it is slightly transparent.

[0031] Next, FIGS. 9 to 11 show screen examples of the pillar object when the processing according to this embodiment is applied. In these figures, the portions indicated by the mesh pattern show the locations (pixels) where the second color related to the "edge" in the second texture is blended and drawn. When the processing according to this embodiment is applied, for the periphery of the "edge" of each opening, the locations where the second color of the "edge" portion is blended and drawn are in a larger (wider) range than in the cases of FIGS. 6 to 8. Also, in these figures, it is shown that the size and range of the locations where the second color of the "edge" portion is blended and drawn change according to the change in the position (imaging direction and height) of the virtual camera. By performing such drawing, the way of showing the inner wall portion (portion with a different drawing color) of the opening that appears transparent can be changed according to the change in the viewing point. As a result, for the portion that is supposed to be seen through (appears to be seen through), a more three-dimensional and depth-like expression can be achieved, and an image with a stronger sense of transparency can be provided. In this example, the expression is such that the transparency around the opening is felt stronger than in the above case. Also, the processing related to this expression is not actually making the object transparent, but is merely a process for making it look like it is. That is, since it does not perform so-called transparency processing or the like, there is no need to control the drawing order of the object, and the processing load is also light. Note that the processing according to this embodiment is processing in a so-called fragment shader (pixel shader).

[0032] Next, the principle and outline of the drawing processing of the object according to this embodiment will be described. In this embodiment, in addition to the two textures as described above, three textures are used, including a third texture (hereinafter, the third texture) used in the parallax mapping technique. Hereinafter, each texture will be described again.

[0033] First, the first texture is conceptually a texture corresponding to the "surface of the object". In this embodiment, the first texture is, for example, a so-called albedo image (an image not affected by the light source, also called an albedo map or the like).

[0034] The second texture is an image of an object that is desired to appear "see-through" as described above. In this embodiment, a normal RGB image will be described as an example, but in other embodiments, for example, the color brightness or darkness may be specified using RGB values. In this case, when viewed as an RGB image, it will be, for example, a grayscale image.

[0035] The third texture is a texture used when applying the above parallax mapping technique. Specifically, it is what is called a "height map" which is also used in bump mapping. The height map stores information (height data) indicating the height (undulation) of the object surface in the form of RGB in the image data. For example, the height map is image data in which data indicated by values within the range of 0 to 1 is stored, with the highest height being 1 (white) and the lowest height being 0 (black) (for example, it becomes a grayscale image).

[0036] Here, regarding the parallax mapping technique, since it is a technique that is already known per se, a detailed explanation will be omitted, and the points regarding the processing of this embodiment will be briefly explained. First, parallax mapping is a lighting technique that expresses the three-dimensional effect and height difference of the unevenness on the object surface by combining a texture without unevenness with height information associated with pixels. Further, the height map is information that specifies the height of the object surface. And parallax mapping is a technique of shifting (shifting) the texture coordinates referred to when drawing an object in consideration of this height. Thereby, shading considering the height can be performed. This parallax is determined based on the angle difference between the viewing direction of the virtual camera and the direction (normal) of the surface (polygon) of the object. Also, the direction in which the coordinates are shifted (hereinafter, the shift direction) and the amount of shift (hereinafter, the shift amount) vary depending on the positional relationship (parallax) between the virtual camera and the object surface. For example, the closer the viewing direction of the virtual camera is to horizontal with respect to the surface of the object (polygon), the larger the shift amount becomes. Conversely, the closer the viewing direction of the virtual camera is to the front direction of the surface of the object, the smaller the shift amount becomes (the shift amount becomes 0 when imaging from the true front). Also, based on the case where the object is imaged from the front, the shift direction is determined by whether the virtual camera has moved in any of the up, down, left, or right directions. For example, when the virtual camera moves to the right, the shift direction is determined to be to the left. That is, the parallax mapping technique is a technique of shifting the texture coordinates to be referred to in consideration of the orientation of the virtual camera and the height indicated by the height map. For example, as shown in FIG. 12, there are object coordinates that are the fixation points of the virtual camera when not considering the height, and based on the height information (height map) regarding this coordinate and the above parallax, the shift of the position of the fixation point is obtained. And it is a technique in which the texture coordinates corresponding to the coordinates reflecting this shift are referred to.

[0037] Applying the parallax mapping technique as described above, in this embodiment, an object is drawn through the following processing. First, prior to the drawing process of the object, association information indicating which part of the texture is used for each part of the surface (mesh) of the object, that is, texture coordinates corresponding to the object surface are specified in advance. That is, a first correspondence relationship indicating the first texture coordinates corresponding to each part of the object surface, a second correspondence relationship indicating the second texture coordinates corresponding to each part of the object surface, and a third correspondence relationship indicating the third texture coordinates corresponding to each part of the object surface are set. Hereinafter, these associations are collectively referred to as "basic correspondence definition".

[0038] Next, the color used for drawing each pixel is obtained from each texture. For the first color, according to the basic correspondence definition, the color of the texture coordinates (hereinafter, the first texture coordinates) of the first texture corresponding to the pixel to be drawn (drawing target pixel) is obtained.

[0039] On the other hand, for the second color, instead of directly using the texture coordinates of the second texture (hereinafter referred to as the second texture coordinates) determined according to the basic correspondence definition, the color obtained from the coordinates shifted using the above parallax mapping technique (hereinafter referred to as the shift coordinates) is used as the second color. In other words, in this embodiment, for the second color, after correcting the above second correspondence relationship using the parallax mapping technique, the second texture coordinates for obtaining (referring to) the second color are determined. For example, assume that for the second texture coordinates, texture coordinates A as shown in FIG. 13 are the second texture coordinates based on the above basic correspondence definition. In this case, by the parallax mapping technique, the coordinates are shifted to texture coordinates B, and the color of the texture coordinates B can be determined as the second color. As a result, as shown in FIGS. 9 to 11 above, the color of the "edge" part of the opening is blended with the first color, and the part drawn with the color after the blending (the shaded part in FIGS. 9 to 11) represents an image that changes according to the orientation and height of the virtual camera. For example, in the example of the above figures, the "edge" part (color) can be like a stretched picture. Also, according to the change in the position of the virtual camera, an expression is made such that the amount of stretching changes.

[0040] Furthermore, in this embodiment, when blending the first color and the second color, a process considering the positions of the virtual camera, the object, and the light source is also performed. Specifically, when the light source is on the viewpoint side of the object (front light) or on the depth side in the line-of-sight direction of the object (backlight) as seen from the virtual camera, the blending ratio of the first color and the second color is changed. In this embodiment, in the case of front light, blending is performed such that the ratio of the first color is higher than that of the second color. In the case of backlight, blending is performed such that the ratio of the second color is higher than that of the first color. Thereby, in the case of front light, an expression can be made as if light is reflected on the surface of the object. Also, in the case of backlight, an expression can be made as if light is transmitted and the inside of the object is more illuminated, and the transparency of the object can be more strongly expressed.

[0041] In the above description, a column object with an opening was used as an example for explanation. However, in addition to this, for example, when it is desired to express the transparency of an object such as "ice" that has a sense of transparency in the real world, the above processing is also useful. FIGS. 14 to 16 show an example of a case where an ice block object is drawn without using the above-described processing. FIGS. 17 to 19 show an example of a case where the above-described processing is applied for drawing. Also, as the movement of the virtual camera assumed in the changes of these figures, a movement is assumed in which the virtual camera moves from right to left while approaching the ice block object slightly. Regarding the texture used for the ice block object, for the first texture, for example, it is an image created centered on a bright blue color and is an image representing the surface of the ice block. Also, for the second texture, it is an image created centered on a dark blue or purple color and is an image representing the interior of the ice block that appears to be transparent. Also, in FIGS. 14 to 19, the portion related to the second texture is shown in a hatching pattern.

[0042] In FIGS. 14 to 16, for the portion related to the second texture, although it is an image that is enlarged and displayed as the virtual camera approaches, the shape itself is shown with little change. On the other hand, in FIGS. 17 to 19, for this portion, as the virtual camera moves, the change in the shape itself (especially the horizontal width) is larger than in the case of FIGS. 14 to 16. That is, for the image of the interior of the ice (second texture), it is shown that it moves out of sync with the change in the expression (movement) of the surface of the ice (first texture) accompanying the movement of the virtual camera. Therefore, for the portion related to the second texture where the surface of the ice appears to be transparent (appears to be), an expression can be made in which the shape (display range) and the like change according to the movement of the virtual camera. As a result, an expression can be made in which the three-dimensional sense of the ice (inside) can be felt more, and ultimately, a richer expression of the transparency can be achieved.

[0043] Incidentally, in the above-described pillar object, if we assume a case where there are no irregularities on the surface portion other than the opening, the same height value will be uniformly set for the above height map. As a result, the above shift amount will be uniform. On the other hand, when assuming an object with irregularities on the surface, the above shift amount can vary depending on the pixel. For example, in the case of the above ice block object example, assume that there are irregularities on the surface of the ice. In this case, a height map corresponding to the irregularities on the surface of the ice is prepared. That is, regarding the second texture coordinates (shift coordinates) obtained as the second color, the degree of irregularities on the surface of the ice is taken into consideration (reflected). Therefore, the above shift amount is not uniform and can vary for each pixel. As a result, in FIGS. 17 to 19 above, for the so-called outer periphery of the shaded portion, it has an expression as if there are steps according to the irregularities (height) on the surface of the ice block object.

[0044] In this way, in the present embodiment, the color of the second texture is blended with the color of the first texture to represent a transparent object. And at this time, only the second texture is applied with the technique of parallax mapping, and after shifting the texture coordinates referred to as the above second color, the drawing color is determined. Thereby, according to the movement of the virtual camera, it is possible to make the appearance of the image corresponding to the inside of the object change three-dimensionally. As a result, the three-dimensional sense inside the object that appears to be seen through (is made to appear to be seen through) can be expressed, and a more transparent object can be expressed with a small processing load.

[0045] [Details of the processing of the present embodiment] Next, with reference to FIGS. 20 to 22, the processing of the present embodiment will be described in more detail. Note that although this processing is assumed to be executed as part of, for example, game processing, in the following description, only the processing related to the drawing of the object as described above will be described, and the detailed description of other game processing will be omitted.

[0046] [Regarding the data used] First, various data used in the processing according to this embodiment will be described. FIG. 20 shows an example of programs and data stored in the storage unit 84 of the game device 2. The storage unit 84 stores a game program 301, object data 302, first texture data 303, second texture data 304, third texture data 305, virtual camera control data 306, light source data 307, and the like.

[0047] The game program 301 is a program for executing a game including object drawing processing according to this embodiment.

[0048] The object data 302 is data related to an object to be drawn. Specifically, the object data 302 includes polygon data of the object. The object data 302 also includes position information and posture information indicating the arrangement position of the object in the virtual space.

[0049] The first texture data 303, the second texture data 304, and the third texture data 305 are image data of the first texture, the second texture, and the third texture, respectively. These texture data are prepared for each object.

[0050] The virtual camera control data 306 is data for controlling the movement of the virtual camera in the virtual space. The virtual camera control data 306 includes information indicating the position of the virtual camera, the line-of-sight direction (imaging direction), the field of view angle, and the like.

[0051] The light source data 307 is data that defines the position of the light source and the intensity of the light in the virtual space.

[0052] In addition, various data necessary for object drawing processing are appropriately stored in the storage unit 84 as needed.

[0053] [Details of Drawing Processing] Next, with reference to the flowchart, the details of the processing according to this embodiment will be described. In this embodiment, one or more processors read and execute the above program stored in one or more memories, thereby realizing the following flowchart. Note that the flowchart is merely an example of the processing process. Therefore, if the same result can be obtained, the processing order of each step may be interchanged. Also, the values of variables and the threshold values used in the determination steps are merely examples, and other values may be adopted as necessary.

[0054] FIG. 21 is a flowchart showing the details of the processing according to this embodiment. Note that the processing loop in steps S4 to S7 in FIG. 21 is repeatedly executed for each frame.

[0055] In FIG. 21, first, in step S1, the processor 81 arranges the object to be drawn in the virtual space. Further, the processor 81 also arranges the light source in the virtual space. Furthermore, in step S2, the processor 81 arranges the virtual camera in the virtual space.

[0056] Next, in step S3, the processor 81 reads the polygon data from the object data 302. Further, the first texture data 303, the second texture data 304, and the third texture data 305 associated with the object are read. At this time, the above-described basic correspondence definition is set. That is, the correspondence between the surface of the object and the texture coordinates of each of the above textures is set.

[0057] Next, in step S4, the processor 81 controls the virtual camera. That is, the movement of the virtual camera and the setting of the line-of-sight direction (imaging direction) are performed. The control may be control to move the virtual camera based on the user's operation on the controller 4, or control to automatically move the virtual camera without the user's operation.

[0058] Next, in step S5, the processor 81 executes object drawing processing. FIG. 22 is a flowchart showing details of the object drawing processing. In FIG. 22, first, in step S21, the processor 81 determines whether or not all polygons constituting the object to be drawn have been drawn. If not all have been drawn yet (NO in step S21), then in step S22, the processor 81 selects a drawing target polygon to be the next drawing target from the polygons that have not been drawn yet.

[0059] Next, in step S23, the processor 81 determines whether or not all pixels corresponding to the current drawing target polygon have been drawn. As a result of this determination, if all pixels related to the drawing target polygon have been drawn (YES in step S23), the process returns to step S21 above and the process is repeated. On the other hand, if not all pixels have been drawn yet (NO in step S23), then in step S24, the processor 81 selects a drawing target pixel to be the next drawing target from the undrawn pixels.

[0060] Next, in step S25, the processor 81 obtains first color information indicating the color of the first texture coordinates corresponding to the drawing target pixel based on the correspondence between the object surface and the first texture based on the above basic correspondence definition.

[0061] Next, in step S26, the processor 81 obtains height information of the third texture coordinates corresponding to the drawing target pixel based on the correspondence between the object surface and the third texture (height map) based on the above basic correspondence definition.

[0062] Next, in step S27, the processor 81 acquires second color information, which is the information of the second color, from the second texture using the above-described parallax mapping technique. Specifically, first, the processor 81 specifies second texture coordinates corresponding to a pixel to be drawn based on the correspondence between the object surface based on the above basic correspondence definition and the second texture. Further, based on the height information and the position and orientation of the virtual camera, the shift amount is calculated. The shift amount is calculated, for example, by the following formula. Second texture coordinates - (XY components of the line-of-sight direction vector × height × predetermined coefficient) ··· Formula 1 Then, the processor 81 acquires, as the second color information, the color information corresponding to the second texture coordinates shifted by the calculated shift amount.

[0063] Next, in step S28, the processor 81 blends the first color with the second color to determine the drawing color of the current pixel to be drawn. At this time, the processor 81 determines the blending ratio of the first color and the second color in consideration of the positional relationship between the object to be drawn and the light source based on the light source data 307. In the present embodiment, this ratio is calculated as the blending rate. As described above, if the positional relationship is front lighting, the blending rate is calculated such that the ratio of the first color is higher than that of the second color. Also, in the case of a backlighting positional relationship, the blending rate is calculated such that the ratio of the second color is higher than that of the first color. Then, the processor 81 blends the first color and the second color based on the calculated blending rate to determine the drawing color. Then, the current pixel to be drawn is drawn with the determined drawing color (for example, writing to the frame buffer is performed). After that, the process returns to step S23 and the process is repeated.

[0064] On the other hand, if the drawing of all the polygons constituting the object to be drawn has been completed as a result of the determination in step S21 (YES in step S21), the object drawing process ends.

[0065] Returning to FIG. 21, next to the object drawing process, in step S6, the processor 81 outputs the image in which the above processing is reflected to the display unit 5.

[0066] Next, in step S7, the processor 81 determines whether the condition for ending the process according to the present embodiment is satisfied. If not (NO in step S7), the process returns to step S4 and the process is repeated. If it is satisfied (YES in step S7), the processor 81 ends the process according to the present embodiment.

[0067] With the above, the detailed description of the process according to the present embodiment ends.

[0068] As described above, in the present embodiment, when drawing an object by blending the colors of the first texture and the second texture, the parallax mapping technique is applied only to the second texture, and the texture for obtaining the second color is shifted in coordinates. Therefore, for the image portion related to the second texture, an image with a stronger sense of three-dimensionality and depth that takes into account the unevenness of the object surface and the positional relationship with the virtual camera can be expressed. As a result, a transparent object that seems to show the inside of the object can be expressed, and a representation in which the three-dimensionality and depth of the inside are more emphasized can be achieved. Furthermore, since such processing is performed at the so-called fragment / pixel shader stage, it is a process with a relatively light processing load. Therefore, a transparent object can be represented with a lower processing load compared to the case of representing it by a conventional transparency process using, for example, the α-blending method.

[0069] [Modification Example] In the above-described embodiment, as an example of the second texture, an image (an image that is desired to appear see-through) assumed to be inside an object is used as a normal RGB image. In addition, as also mentioned above, for example, information indicating the brightness or darkness of a color in the form of an RGB image may be used as the second texture. Conceptually, the second texture in this case can be said to define the amount of light passing through the object surface corresponding to each pixel (transparency) (however, as in the above-described process, the actual process of transmitting light is not performed). When using such a second texture, for example, the "brightness information" of the second color is obtained from the second texture coordinates (shift coordinates) determined using the above-described parallax mapping technique. Then, the second color may be determined by multiplying a predetermined color to be expressed by the "brightness information". Further, two or more colors may be multiplied as the colors to be multiplied.

[0070] Regarding the first texture, in addition to the above-described albedo map, a normal map, a roughness map, etc. may be used in combination for processing. Thereby, while performing more diverse expressions for the surface of the object, an object having a sense of transparency can be expressed.

[0071] Regarding the blending of the first color and the second color, in the above example, an example of calculating the blend ratio was given. However, as long as it is a parameter that can specify the ratio between the two, not limited to the blend ratio, other parameters may be used for specification.

[0072] In the above-described embodiment, the height information is used as a height map (third texture). In other embodiments, the height information may be embedded, for example, in the value of α in RGBα format image data, and may be configured to be summarized in, for example, one first texture. In this case, the number of textures to be used can be two.

[0073] In addition, in the above embodiment, the case where the above series of processes are executed in a single device has been described. However, in other embodiments, the above series of processes may be executed in an information processing system including a plurality of information processing devices. For example, in an information processing system including a terminal-side device and a server-side device that can communicate with the terminal-side device via a network, some of the above series of processes may be executed by the server-side device. Furthermore, in an information processing system including a terminal-side device and a server-side device that can communicate with the terminal-side device via a network, the main processes of the above series of processes may be executed by the server-side device, and some processes may be executed by the terminal-side device. Also, in the above information processing system, the server-side system may be composed of a plurality of information processing devices, and the plurality of information processing devices may share and execute the processes to be executed on the server side.

[0074] In addition, a so-called cloud gaming configuration may be adopted. For example, the game device 2 may be configured to send operation data indicating a user's operation to a predetermined server, various processes are executed on the server, and the execution results are streamed and distributed to the game device 2 as video and audio.

Explanation of Reference Numerals

[0075] 2 Game device 4 Controller 5 Display unit 81 Processor 84 Storage unit 87 Image and audio output unit

Claims

1. An image processing program for rendering an object arranged in a virtual space, wherein the computer is configured to: manage a first texture including first color information associated with the surface of the object based on a first correspondence relationship, a second texture including second color information associated with the surface of the object based on a second correspondence relationship, and a third texture including height information associated with the surface of the object based on a third correspondence relationship; virtual camera control means for controlling the line-of-sight direction of a virtual camera arranged in the virtual space; drawing means for determining a drawing color using at least the first color information of the first texture obtained based on the first correspondence relationship and the second color information of the second texture obtained based on a corrected correspondence relationship obtained by correcting the second correspondence relationship based on the first color information of the first texture obtained based on the first correspondence relationship, the height information specified based on the line-of-sight direction of the virtual camera, the orientation of the surface of the object, and the third correspondence relationship in response to a change in the line-of-sight direction of the virtual camera, and for rendering the surface of the object; light source arrangement means for arranging a virtual light source in the virtual space; functioning as; wherein the drawing means: determines the drawing color by synthesizing at least the first color information and the second color information at a predetermined ratio; and determines the drawing color such that the ratio of the second color information is higher when the light source is located on the back side of the surface of the object than when the light source is located on the front side of the surface of the object. An image processing program.

2. The image processing program according to claim 1, wherein the drawing means obtains the second color information by shifting the position in the second texture referred to during drawing from the position specified based on the second correspondence relationship to a position further shifted based on the correction.

3. The image processing program according to claim 2, wherein the drawing means determines the direction and amount of the shift of the position in the second texture referred to during drawing based on the relative relationship between the line-of-sight direction of the virtual camera and the orientation of the surface of the object.

4. The image processing program according to claim 3, wherein the drawing means determines the amount of the shift such that the amount of the shift becomes smaller as the line-of-sight direction of the virtual camera approaches the reverse direction of the normal direction of the surface of the object.

5. An image processing apparatus for rendering an object arranged in a virtual space, management means for managing a first texture including first color information associated with the surface of the object based on a first correspondence relationship, a second texture including second color information associated with the surface of the object based on a second correspondence relationship, and a third texture including height information associated with the surface of the object based on a third correspondence relationship; virtual camera control means for controlling the line-of-sight direction of a virtual camera arranged in the virtual space; rendering means for determining a rendering color using at least the first color information of the first texture obtained based on the first correspondence relationship and the second color information of the second texture obtained based on a corrected correspondence relationship obtained by correcting the second correspondence relationship based on the line-of-sight direction of the virtual camera and the height information specified based on the third correspondence relationship between the line-of-sight direction of the virtual camera and the orientation of the surface of the object in response to a change in the line-of-sight direction of the virtual camera, and for rendering the surface of the object; light source arrangement means for arranging a virtual light source in the virtual space, wherein the rendering means determines the rendering color by synthesizing at least the first color information and the second color information at a predetermined ratio, and determines the rendering color such that the ratio of the second color information is higher when the light source is located on the back side of the surface of the object than when the light source is located on the front side of the surface of the object. An image processing apparatus.

6. An image processing system for rendering an object arranged in a virtual space, management means for managing a first texture including first color information associated with the surface of the object based on a first correspondence relationship, a second texture including second color information associated with the surface of the object based on a second correspondence relationship, and a third texture including height information associated with the surface of the object based on a third correspondence relationship; virtual camera control means for controlling the line-of-sight direction of a virtual camera arranged in the virtual space; Based on the first color information of the first texture obtained based on the first correspondence relationship, and in response to the change in the viewing direction of the virtual camera, based on the correction based on the viewing direction of the virtual camera, the orientation of the surface of the object, and the height information specified based on the third correspondence relationship, the rendering color is determined using at least the second color information of the second texture obtained based on the corrected correspondence relationship obtained by correcting the second correspondence relationship, and a rendering means for rendering the surface of the object, a light source arranging means for arranging a virtual light source in the virtual space, wherein the rendering means, determines the rendering color by synthesizing at least the first color information and the second color information at a predetermined ratio, and when the light source is located on the back side of the surface of the object, determines the rendering color such that the ratio of the second color information is higher than when the light source is located on the front side of the surface of the object. An image processing system.

7. An image processing method for causing a computer of an image processing apparatus that renders an object arranged in a virtual space to execute, wherein the computer, manages a first texture including first color information associated with the surface of the object based on a first correspondence relationship, a second texture including second color information associated with the surface of the object based on a second correspondence relationship, and a third texture including height information associated with the surface of the object based on a third correspondence relationship, controls the viewing direction of a virtual camera arranged in the virtual space, Based on the first color information of the first texture obtained based on the first correspondence relationship, and in response to the change in the viewing direction of the virtual camera, based on the correction based on the viewing direction of the virtual camera, the orientation of the surface of the object, and the height information specified based on the third correspondence relationship, the rendering color is determined using at least the second color information of the second texture obtained based on the corrected correspondence relationship obtained by correcting the second correspondence relationship, and the surface of the object is rendered, arranges a virtual light source in the virtual space, in the rendering, determines the rendering color by synthesizing at least the first color information and the second color information at a predetermined ratio, An image processing method for determining the drawing color such that when the light source is located on the back side of the surface of the object, the ratio of the second color information is higher than when the light source is located on the surface side of the object.

Images