Image processing program, image processing device, image processing system, and image processing method

Abstract

To provide an image processing program, an image processing system, an image processing device, and an image processing method capable of expressing an object having transparency with a low processing load.SOLUTION: An image processing device manages a first texture including first color information, a second texture including second color information, and a third texture including height information. The image processing device acquires the first color information of the first texture corresponding to a surface of an object, acquired on the basis of on a first correspondence. The image processing device corrects, on the basis of a relation between a visual line direction of a virtual camera and an orientation of a surface of an object, and height information identified on the basis of a third correspondence, a second correspondence which is a relation between the surface of the object and the second texture, and acquires the second color information from the second texture on the basis of the corrected second correspondence. The image processing device then uses the acquired color information to determine a drawing color and draws the surface of the object.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present invention relates to image processing for expressing a three-dimensional object with a sense of transparency. [Background technology]

[0002] Conventionally, one of the techniques for rendering a three-dimensional virtual object is a technique called texture mapping, which renders the object by pasting a texture image onto the object (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-141822 A Summary of the Invention [Problem to be solved by the invention]

[0004] The texture mapping described above can improve the expressiveness of the texture of the object's surface. However, when expressing a transparent object, such as an object with a semi-transparent surface, this method may not be sufficient. Therefore, when trying to draw a transparent object, it was common to perform transparency processing using, for example, the alpha blending method. However, conventional transparency processing generally had a high processing load. For example, assume that the surface of a first object is transparent and the second object on the other side is visible. In this case, first, the second object, which is the object to be transparent, is drawn. Next, the first object, which is a semi-transparent object, is placed in front of the second object and drawn. Alternatively, the first semi-transparent object is placed so as to be overlaid on the second object and drawn. At this time, various effects may be applied to the surface of the first object and drawn. In addition, the second object may also be drawn with an effect that distorts the image. In such processing, the drawing order of the objects is important, but generally, the management of the order is complicated, or the drawing order interferes, resulting in a high processing load. As a result, in some cases, a corrupted image could be displayed.

[0005] Therefore, an object of the present invention is to provide an image processing program, an image processing device, an image processing system, and an image processing method that are capable of expressing an object having a sense of transparency with a low processing load. [Means for solving the problem]

[0006] In order to achieve the above object, for example, the following configuration example can be given.

[0007] (Configuration 1) Configuration 1 is an image processing program for drawing an object arranged in a virtual space, which causes a computer to function as a management means, a virtual camera control means, and a drawing means. The management means manages a first texture including first color information that is associated with the surface of the object based on a first correspondence relationship, a second texture including second color information that is associated with the surface of the object based on a second correspondence relationship, and a third texture including height information that is associated with the surface of the object based on a third correspondence relationship. The virtual camera control means controls the line of sight direction of a virtual camera arranged in the virtual space. The drawing means (1) controls a texture corresponding to the surface of the object acquired based on the first correspondence relationship. and (2) second color information of a second texture corresponding to the surface of the object, which is acquired based on a corrected correspondence relationship obtained by correcting the second correspondence relationship by a correction based on the height information specified based on the relationship between the line of sight of the virtual camera and the orientation of the surface of the object and the third correspondence relationship. Then, the drawing means draws the surface of the object using the drawing color.

[0008] According to the above configuration example, it is possible to render a transparent object in the process of determining the rendering color of each pixel, without actually performing a transparency process that makes the surface of the object transparent. Therefore, it is possible to render a transparent object with a low processing load.

[0009] (Configuration 2) In configuration 2, in configuration 1, the drawing means may obtain the second color information by shifting the position in the second texture referenced when drawing from a position identified based on the second correspondence relationship to a position further shifted based on the correction.

[0010] According to the above configuration example, the second correspondence relationship is corrected for each pixel, which means that it is possible to perform drawing that accurately reflects the positional relationship between the virtual camera and the object for each pixel.

[0011] (Configuration 3) In configuration 3, in the above configuration 2, the drawing means may determine the direction and amount of shifting of the position in the second texture referenced when drawing, based on the relative relationship between the line of sight direction of the virtual camera and the orientation of the surface of the object.

[0012] According to the above configuration example, the direction and amount of shifting are determined taking into consideration the relationship between the direction of the virtual camera (line of sight) and the direction of the object's surface. This makes it possible to provide an image that is less unnatural in appearance when expressing a transparent object. (Configuration 4) Configuration 4 is based on configuration 2, and the drawing means may determine the amount of shift such that the amount of shift becomes smaller as the line of sight direction of the virtual camera approaches a direction opposite to a normal direction of the object surface.

[0013] According to the above configuration example, for example, the closer the line of sight of the virtual camera is to the horizontal with respect to the object (surface), the larger the shift amount becomes. This allows for a richer expression of transparency that takes into account the unevenness and height difference of the object surface.

[0014] (Configuration 5) In a fifth aspect of the present invention, in any one of the first to fourth aspects, the image processing program may further cause the computer to function as a light source arrangement means for arranging a virtual light source in the virtual space. The drawing means may determine a drawing color by combining at least the first color information and the second color information in a predetermined ratio, and in this case, when the light source is located on the back side of the surface of the object, the drawing color may be determined so that the ratio of the second color information is higher than when the light source is located on the front side of the object.

[0015] According to the above configuration example, the synthesis ratio of the colors related to the first color information and the second color information is determined according to the positional relationship between the light source and the virtual camera. Specifically, if the positional relationship with the light source is a backlit relationship, the synthesis is performed so that the ratio of the colors related to the second color information is higher than in the case of front-lit. Therefore, in the case of backlit, the colors related to the second color information can be expressed more strongly than in the case of front-lit. By taking into account such a positional relationship with the light source, it is possible to express a richer sense of transparency. Effect of the Invention

[0016] According to this embodiment, it is possible to express objects with a sense of transparency with low processing load. [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is a block diagram showing an example of the configuration of a game device 2. [Diagram 2] FIG. 1 is a diagram for explaining an overview of processing according to the present embodiment; [Diagram 3] FIG. 1 is a diagram for explaining an overview of processing according to the present embodiment; [Figure 4] FIG. 1 is a schematic diagram showing an example of a first texture; [Diagram 5] FIG. 13 is a schematic diagram showing an example of a second texture; [Figure 6] Example of a screen when the process according to this embodiment is not used [Figure 7] Example of a screen when the process according to this embodiment is not used [Figure 8] Example of a screen when the process according to this embodiment is not used [Figure 9] Example of a screen when the process according to this embodiment is used [Figure 10] Example of a screen when the process according to this embodiment is used [Figure 11] Example of a screen when the process according to this embodiment is used [Figure 12] FIG. 1 is a diagram for explaining disparity mapping. [Figure 13] FIG. 1 is a diagram for explaining an overview of processing according to the present embodiment; [Figure 14] Example of a screen when the process according to this embodiment is not used [Figure 15] Example of a screen when the process according to this embodiment is not used [Figure 16] Example of a screen when the process according to this embodiment is not used [Figure 17] Example of a screen when the process according to this embodiment is used [Figure 18] Example of a screen when the process according to this embodiment is used [Figure 19] Example of a screen when the process according to this embodiment is used [Figure 20] An example of programs and data stored in the storage unit 84 of the game device 2 [Figure 21] A flowchart showing details of the process according to the present embodiment. [Figure 22] Flowchart showing details of object drawing process DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] An embodiment will be described below.

[0019] [Hardware configuration of information processing device] First, an information processing device for executing information processing according to the present embodiment will be described. The information processing device is, for example, a smartphone, a stationary or portable game device, a tablet terminal, a mobile phone, a personal computer, a wearable terminal, etc. Furthermore, the information processing according to the present embodiment can also be applied to a game system composed of the above-mentioned game devices and a predetermined server. In the present embodiment, a stationary game device (hereinafter simply referred to as a game device) will be described as an example of the information processing device.

[0020] FIG. 1 is a block diagram showing an example of the internal configuration of a game device 2 according to this embodiment. The game device 2 includes a processor 81. The processor 81 is an information processing section that executes various types of information processing executed in the game device 2, and includes, for example, a CPU (Central Processing Unit). The processor 81 may be configured with only a Graphics Processing Unit (GPU) or may be configured with a System-on-a-chip (SoC) including multiple functions such as a CPU function and a Graphics Processing Unit (GPU) function. The processor 81 executes various information processing by executing an information processing program stored in the storage unit 84. The storage unit 84 may be an internal storage medium such as a flash memory or a Dynamic Random Access Memory (DRAM), or may be configured to use an external storage medium or the like that is inserted into a slot (not shown).

[0021] The game device 2 also includes a controller communication unit 86 for allowing the game device 2 to perform wired or wireless communication with the controller 4. Although not shown in the drawings, the controller 4 is provided with various buttons such as a cross key and ABXY buttons, an analog stick, etc.

[0022] Furthermore, the game device 2 is connected to a display unit 5 (e.g., a television or the like) via an image / audio output unit 87. The processor 81 outputs images and sounds generated (for example, by executing the above-mentioned information processing) to the display unit 5 via the image / audio output unit 87.

[0023] An example of the process according to this embodiment will be described below. The process according to this embodiment relates to drawing of a (three-dimensional) object. More specifically, it relates to a drawing process for expressing an object with a sense of transparency. Moreover, the drawing process may be performed as a part of a game process, for example.

[0024] Generally, a technique of using a texture image to express the texture of the surface of an object (so-called texture mapping) is known. In this embodiment, at least two textures are used to express the transparency of an object. The first texture (hereinafter, the first texture) is conceptually an image for expressing the surface of an object (so-called the foreground color). The second texture (hereinafter, the second texture) is conceptually an image assumed to be located inside an object, and is an image assumed to be seen through the object surface. In other words, it is an image for expressing the inside of the object that is visible through the transparent surface (so-called the background color of the transparent surface). When drawing an object (each pixel corresponding to the object), the color of the second texture is blended with the pixels of the first texture in the part where the transparency is desired. By adjusting the blending ratio at this time, it is possible to express the image related to the second texture as being visible through the corresponding part of the first texture image. For example, if the blending ratio of the color of the first texture is increased, a semi-transparent feeling in which the image of the second texture is slightly visible through the first texture can be expressed. Additionally, by increasing the blending rate of the second texture, you can make the object's surface appear more transparent.

[0025] Although the above-mentioned methods can express a certain degree of transparency on the surface of an object, this embodiment makes it possible to express an object with a higher level of transparency by the process described below. Although details will be described later, in this embodiment, a process is performed on only the second texture described above, using a so-called parallax mapping technique, to render it so that it appears more three-dimensional. This gives the surface of the object a higher level of transparency, and also makes it possible to more effectively express the sense of depth and three-dimensionality of the inside of the object that is (or is intended to be) visible through the object. Furthermore, this type of transparent object can be expressed with a lower processing load than when transparency processing is used.

[0026] 2 to 11, examples of object images when the processing according to this embodiment is not used and examples of object images when the processing according to this embodiment is used are shown. Here, as an example, an example of drawing a vertically long rectangular pillar-shaped object as shown in FIG. 2 is shown. The pillar object has a square opening on the first surface and a round opening on the second surface. In addition, the virtual camera is assumed to move from a state in which the first surface is imaged from a height slightly looking down on the first surface to a state in which the virtual camera moves around to the second surface (see FIG. 3). In this example, it is assumed that the inner walls of these openings are represented as being see-through.

[0027] As the first texture, a texture as shown in FIG. 4 is prepared, and as the second texture, It is assumed that the textures shown in FIG. 5 are prepared as the pillar object. Although each of the drawings is a schematic illustration, the first texture is an image corresponding to the surface portion of the pillar object, and can be said to be a basic image. In FIG. 4, the first texture is an image expressing a square opening and a round opening. For ease of explanation, it is assumed that the color of the pillar object in the first texture is a predetermined single color. In other words, it is assumed that the first texture is a single-color image.

[0028] On the other hand, the second texture is an image that imagines an inner wall seen through the opening. In this example, the second texture is an image in which the portion corresponding to the "edge" of the opening is expressed using a color different from the above-mentioned predetermined single color. For example, the predetermined single color in the first texture is orange, and the "edge" portion of the second texture is white. Also, the color of the second texture other than the "edge" is the same as the first texture.

[0029] An example of a case where a pillar object is drawn using the above texture without performing the processing according to this embodiment is shown. Figs. 6 to 8 are examples of screens in such a case. In Fig. 6, the virtual camera is capturing an image of a square opening from a height slightly looking down from the upper right. Fig. 6 shows that for the "edge" portion (pixel) of the square opening, a color obtained by blending the color of the texel on the first texture corresponding to that portion (hereinafter, the first color) with the color of the texel on the second texture corresponding to that portion (hereinafter, the second color) is used as the drawing color. Also, it shows that the round opening in the upper right is drawn with a drawing color obtained by blending the second color with the first color for the "edge" portion.

[0030] When the virtual camera moves from the state shown in Fig. 6 around to the second surface side, images like those shown in Fig. 7 and Fig. 8 are displayed. In all of the figures, the "edges" of each opening are drawn with a drawing color that is a blend of the first color and the second color. Therefore, in these figures, the "edges" of each opening are expressed with a slightly transparent feel.

[0031] Next, examples of screens of a pillar object when the processing according to this embodiment is applied are shown in Figs. 9 to 11. In these figures, the parts shown with a hatched pattern indicate the parts (pixels) where the second color of the "edge" part in the second texture is blended and drawn. When the processing according to this embodiment is applied, the area around the "edge" of each opening where the second color of the "edge" part is blended and drawn is larger (wider) than in Figs. 6 to 8. In addition, these figures show that the size and range of the area where the second color of the "edge" part is blended and drawn changes according to the change in the position (imaging direction and height) of the virtual camera. By performing such drawing, the appearance of the inner wall part of the opening that can be seen through (the part with a different drawing color) can be changed according to the change in the viewpoint. This makes it possible to express the part that can be seen through (or is desired to be seen through) with a more three-dimensional feeling and a sense of depth, and to provide an image that gives a sense of transparency. In this example, the expression gives a stronger sense of transparency around the opening than in the above case. In addition, the processing related to the expression does not actually make the object transparent, but is merely processing to make it look like that. In other words, since so-called transparency processing, etc. is not performed, there is no need to control the drawing order of objects, etc., and the processing load is light. Note that the processing according to this embodiment is processing in a so-called fragment shader (pixel shader).

[0032] Next, the principle and overview of the object rendering process according to this embodiment will be described. In this embodiment, in addition to the two textures described above, In addition, a third texture (hereafter referred to as the third texture) that can be used for the first texture is used. Each texture is explained below.

[0033] First, the first texture is a texture that conceptually corresponds to the "surface of an object." In this embodiment, the first texture is, for example, a so-called albedo image (an image not affected by a light source, also called an albedo map, etc.).

[0034] The second texture is an image of something that is desired to be "see-through" as described above. In this embodiment, a normal RGB image is used as an example, but in other embodiments, the content may be, for example, a color brightness or darkness specified using RGB values. In this case, when viewed as an RGB image, it becomes, for example, a grayscale image.

[0035] The third texture is a texture used when using the above-mentioned parallax mapping technique, specifically, it is called a "height map" that is also used in bump mapping. The height map is information (height data) indicating the height (unevenness) of the object surface stored in image data in RGB format. For example, the height map is image data (e.g., a grayscale image) in which data indicated by values ​​within a range of 0 to 1 is stored, with the highest height being 1 (white) and the lowest height being 0 (black).

[0036] Here, the parallax mapping technique is a known technique, so a detailed description will be omitted, but the processing of this embodiment will be briefly described. First, parallax mapping is a lighting technique that expresses the three-dimensional effect of the unevenness of the object surface and the height difference by combining height information associated with pixels with a texture that has no unevenness. The height map is information that specifies the height of the object surface. Parallax mapping is a technique that shifts the texture coordinates that are referenced when drawing an object, taking this height into consideration. This makes it possible to perform shading that takes the height into consideration. This parallax is determined based on the angle difference between the line of sight of the virtual camera and the direction (normal) of the object's surface (polygon). The direction in which the coordinates are shifted (hereinafter, shift direction) and the amount of shift (hereinafter, shift amount) change depending on the positional relationship (parallax) between the virtual camera and the object surface. For example, the closer the line of sight of the virtual camera is to the horizontal with respect to the object (polygon) surface, the larger the shift amount. Conversely, the shift amount becomes smaller as the line of sight of the virtual camera moves in a direction parallel to the front direction of the object's surface (the shift amount is 0 when the image is captured from directly in front). The shift direction is determined depending on whether the virtual camera has moved up, down, left, or right, based on the image captured from the front of the object. For example, when the virtual camera has moved to the right, the shift direction is determined to be to the left. In other words, the parallax mapping technique is a technique for shifting the texture coordinates to be referenced, taking into consideration the direction of the virtual camera and the height indicated by the height map. For example, as shown in FIG. 12, there is an object coordinate that is the gaze point of the virtual camera when the height is not taken into consideration, and the shift of the gaze point position is calculated based on the height information (height map) related to this coordinate and the parallax. Then, the texture coordinates corresponding to the coordinates reflecting this shift are referenced.

[0037] In this embodiment, by applying the above-mentioned parallax mapping technique, an object is rendered by the following process. First, prior to the rendering process of an object, correspondence information indicating which part of the texture each part of the object's surface (mesh) uses, that is, texture coordinates corresponding to the object surface, is specified in advance. That is, a first correspondence indicating first texture coordinates corresponding to each part of the object's surface, a second correspondence indicating second texture coordinates corresponding to each part of the object's surface, and a third correspondence indicating third texture coordinates corresponding to each part of the object's surface are set. Below, these correspondences are collectively referred to as "basic correspondence definitions."

[0038] Next, the colors to be used for drawing each pixel are obtained from each texture. For the first color, the color of the texture coordinates of the first texture (hereinafter, first texture coordinates) corresponding to the pixel to be drawn (pixel to be drawn) is obtained in accordance with the basic correspondence definition.

[0039] On the other hand, for the second color, the texture coordinates of the second texture (hereinafter, second texture coordinates) determined according to the basic correspondence definition are not used as they are, but the color obtained from the coordinates shifted using the parallax mapping technique (hereinafter, shifted coordinates) is used as the second color. In other words, in this embodiment, for the second color, the second correspondence is corrected using the parallax mapping technique, and then the second texture coordinates for obtaining (referring to) the second color are determined. For example, for the second texture coordinates, assume that the texture coordinates A shown in FIG. 13 become the second texture coordinates based on the basic correspondence definition. In this case, the coordinates are shifted to the texture coordinates B by the parallax mapping technique, and the color of the texture coordinates B can be determined as the second color. As a result, as shown in FIGS. 9 to 11, an image is expressed in which the color of the "edge" of the opening is blended with the first color, and the part drawn with the blended color (the shaded part in FIGS. 9 to 11) changes depending on the direction and height of the virtual camera. For example, in the example of the above figure, the "edge" part (color) may be stretched, so to speak. Also, the amount of stretching can be expressed as changing according to the change in the position of the virtual camera.

[0040] Furthermore, in this embodiment, when blending the first color and the second color, processing is performed taking into consideration the positions of the virtual camera, the object, and the light source. Specifically, the blending ratio of the first color and the second color is changed depending on whether the light source is closer to the viewpoint than the object when viewed from the virtual camera (in the case of front lighting) or the light source is deeper in the line of sight than the object (in the case of back lighting). In this embodiment, in the case of front lighting, the blending is performed so that the ratio of the first color is higher than the second color. In the case of back lighting, the blending is performed so that the ratio of the second color is higher than the first color. As a result, in the case of front lighting, it is possible to express that light is reflected on the surface of the object. In addition, in the case of back lighting, it is possible to express that the light is transmitted and the inside of the object is more illuminated, and the transparency of the object can be expressed more strongly.

[0041] In the above, a pillar object with an opening is described as an example, but the above process is also useful when expressing the transparency of an object such as "ice" that has transparency in the real world. Figs. 14 to 16 show an example of an ice block object drawn without using the above process. Figs. 17 to 19 show an example of an ice block object drawn by applying the above process. The virtual camera movement assumed in the changes in these figures is assumed to be a movement that wraps around from right to left and approaches the ice block object slightly. Regarding the texture used for the ice block object, the first texture is an image created mainly with a light blue color, for example, and is an image that expresses the surface of the ice block. The second texture is an image created mainly with a dark blue or purple color, and is an image that expresses the inside of the ice block that can be seen through. In Figs. 14 to 19, the part related to the second texture is shown by a hatched pattern.

[0042] In Figs. 14 to 16, the portion relating to the second texture is displayed as an image that is enlarged as the virtual camera approaches, but the shape itself does not change significantly. On the other hand, in Figs. 17 to 19, the shape of the portion (particularly the width) changes more significantly as the virtual camera moves than in Figs. 14 to 16. In other words, the image of the inside of the ice (second texture) moves out of sync with the change in the representation (movement) of the ice surface (first texture) that accompanies the movement of the virtual camera. Therefore, for the part related to the second texture where the surface of the ice is (appears to be) transparent, the shape (display range) can be expressed as changing according to the movement of the virtual camera. This allows for an expression that gives a greater sense of three-dimensionality to the ice (inside), and in turn allows for a richer expression of transparency.

[0043] Incidentally, in the above-mentioned pillar object, if it is assumed that there is no unevenness in the surface portion other than the opening, the height map is set to a value indicating the same height uniformly. As a result, the above-mentioned shift amount is uniform. On the other hand, if it is assumed that the object has (is assumed to have) unevenness on the surface, the above-mentioned shift amount may differ depending on the pixel. For example, in the above-mentioned ice block object example, it is assumed that the ice surface has unevenness. In this case, a height map corresponding to the unevenness of the ice surface is prepared. That is, the unevenness of the ice surface is taken into account (reflected) in the second texture coordinate (shift coordinate) acquired as the second color. Therefore, the above-mentioned shift amount is not uniform, and may differ for each pixel. As a result, in the above-mentioned Figs. 17 to 19, the outer periphery of the shaded portion is expressed as if it has steps according to the unevenness (height) of the surface of the ice block object.

[0044] In this manner, in this embodiment, the color of the second texture is blended with the color of the first texture to express a transparent object. In this case, the technique of parallax mapping is applied only to the second texture, and the texture coordinates referred to as the second color are shifted before the rendering color is determined. This makes it possible to make the appearance of the image corresponding to the inside of the object appear to change three-dimensionally according to the movement of the virtual camera. As a result, it is possible to express the three-dimensional effect of the inside of an object that is (appears to be) see-through, and it is possible to express a more transparent object with a small processing load.

[0045] [Details of the processing in this embodiment] Next, the processing of this embodiment will be described in more detail with reference to Figures 20 to 22. Note that, although it is assumed that this processing is executed as part of, for example, game processing, in the following description, only the processing related to drawing of objects as described above will be described, and detailed description of other game processing will be omitted.

[0046] [About data usage] First, various data used in the processing according to this embodiment will be described. Fig. 20 shows an example of programs and data stored in the storage unit 84 of the game device 2. The storage unit 84 stores a game program 301, object data 302, first texture data 303, second texture data 304, third texture data 305, virtual camera control data 306, light source data 307, and the like.

[0047] The game program 301 is a program for executing a game including the object drawing process according to this embodiment.

[0048] The object data 302 is data related to an object to be rendered. Specifically, the object data 302 includes polygon data of the object. The object data 302 also includes position information and orientation information indicating the placement position of the object in the virtual space.

[0049] The first texture data 303, the second texture data 304, and the third texture data 305 are image data of the first texture, the second texture, and the third texture, respectively. These texture data are prepared for each of the objects.

[0050] The virtual camera control data 306 is data for controlling the movement of the virtual camera in a virtual space, etc. The virtual camera control data 306 includes information indicating the position, line of sight direction (imaging direction), angle of view, etc. of the virtual camera.

[0051] The light source data 307 is data that defines the position and light intensity of a light source in a virtual space.

[0052] In addition, various data necessary for object drawing processing is appropriately stored in the storage unit 84 as required.

[0053] [Drawing process details] Next, the details of the processing according to this embodiment will be described with reference to a flowchart. In this embodiment, one or more processors read and execute the above program stored in one or more memories, thereby realizing the flowchart shown below. Note that this flowchart is merely an example of the processing process. Therefore, the processing order of each step may be changed as long as the same result is obtained. In addition, the values ​​of the variables and the threshold values ​​used in the judgment step are merely examples, and other values ​​may be adopted as necessary.

[0054] Fig. 21 is a flowchart showing details of the process according to this embodiment. Note that the process loop of steps S4 to S7 in Fig. 21 is repeatedly executed for each frame.

[0055] 21, first, in step S1, the processor 81 places an object to be drawn in a virtual space. The processor 81 also places a light source in the virtual space. Furthermore, in step S2, the processor 81 places a virtual camera in the virtual space.

[0056] Next, in step S3, processor 81 reads polygon data from the object data 302. Furthermore, it reads first texture data 303, second texture data 304, and third texture data 305 associated with the object. At this time, the above-mentioned basic correspondence definition is set. That is, the correspondence between the surface of the object and the texture coordinates of each of the above-mentioned textures is set.

[0057] Next, in step S4, the processor 81 controls the virtual camera. That is, the virtual camera is moved and its line of sight (imaging direction) is set. The control may be a control for moving the virtual camera based on a user's operation on the controller 4, or a control for automatically moving the virtual camera without a user's operation.

[0058] Next, in step S5, the processor 81 executes an object drawing process. Fig. 22 is a flowchart showing the details of the object drawing process. In Fig. 22, first, in step S21, the processor 81 determines whether or not all polygons constituting the object to be drawn have been drawn. If not all polygons have been drawn yet (NO in step S21), in step S22, the processor 81 selects a polygon to be drawn next from among the polygons not yet drawn.

[0059] Next, in step S23, the processor 81 determines whether or not all of the pixels corresponding to the current polygon to be rendered have been rendered. If the result of this determination is that all of the pixels related to the polygon to be rendered have been rendered (YES in step S23), the process returns to step S21 and the processing is repeated. On the other hand, if all of the pixels have not yet been rendered (NO in step S23), in step S24, the processor 81 selects a pixel to be rendered next from among the unrendered pixels.

[0060] Next, in step S25, the processor 81 acquires first color information indicating the color of the first texture coordinates corresponding to the pixel to be rendered, based on the correspondence between the object surface and the first texture based on the basic correspondence definition.

[0061] Next, in step S26, the processor 81 acquires height information of the third texture coordinates corresponding to the pixel to be rendered, based on the correspondence between the object surface and the third texture (height map) based on the basic correspondence definition.

[0062] Next, in step S27, the processor 81 acquires second color information, which is information on the second color, from the second texture using the parallax mapping technique. Specifically, the processor 81 first identifies second texture coordinates corresponding to the pixel to be rendered based on the correspondence between the object surface and the second texture based on the basic correspondence definition. Furthermore, the processor 81 calculates the shift amount based on the height information and the position and orientation of the virtual camera. The shift amount is calculated, for example, by the following formula. second texture coordinates−(X and Y components of gaze direction vector×height×predetermined coefficient) Equation 1 Then, processor 81 acquires color information related to the second texture coordinates shifted by the calculated shift amount as second color information.

[0063] Next, in step S28, the processor 81 blends the first color with the second color to determine the rendering color of the current rendering target pixel. At this time, the processor 81 determines the blending ratio of the first color and the second color based on the light source data 307 and taking into consideration the positional relationship between the rendering target object and the light source. In this embodiment, this ratio is calculated as a blending ratio. As described above, if the positional relationship is front-lit, the blending ratio is calculated so that the ratio of the first color is higher than that of the second color. Also, if the positional relationship is back-lit, the blending ratio is calculated so that the ratio of the second color is higher than that of the first color. Then, the processor 81 blends the first color and the second color based on the calculated blending ratio to determine the rendering color. Then, the current rendering target pixel is rendered with the determined rendering color (for example, writing is performed in the frame buffer). Thereafter, the process returns to step S23 and is repeated.

[0064] On the other hand, if the result of the determination in step S21 above is that rendering has been completed for all polygons constituting the object to be rendered (YES in step S21), the object rendering process ends.

[0065] Returning to FIG. 21, after the object drawing process, in step S6, the processor 81 outputs to the display unit 5 an image reflecting the above process.

[0066] Next, in step S7, the processor 81 determines whether or not the condition for terminating the processing according to this embodiment is satisfied. If the condition is not satisfied (NO in step S7), the process returns to step S4 and is repeated. If the condition is satisfied (YES in step S7), the processor 81 terminates the processing according to this embodiment.

[0067] This concludes the detailed description of the processing according to this embodiment.

[0068] In this manner, in this embodiment, when an object is rendered by blending the colors of the first texture and the second texture, the parallax mapping technique is applied only to the second texture, and the coordinates of the texture from which the second color is obtained are shifted. As a result, an image with a greater sense of three-dimensionality and depth can be expressed for the image portion relating to the second texture, taking into account the unevenness of the object's surface and its positional relationship with the virtual camera. This makes it possible to express a transparent object, as if the inside of the object is visible through the transparent object, and to express the three-dimensionality and depth of the interior in a more emphasized manner. Furthermore, this type of processing is known as a freehand rendering. Because this is done at the fragment / pixel shader stage, the processing load is relatively light, making it possible to express transparent objects with a lower processing load than with conventional transparency processing using, for example, alpha blending.

[0069] [Variations] In the above embodiment, an example was shown in which an image assumed to be inside an object (an image to be seen through) was used as a normal RGB image as the second texture. As mentioned above, the second texture may be an RGB image that contains information indicating color brightness or color density. In this case, the second texture can be conceptually said to define the amount of light passing through the object surface corresponding to each pixel (transparency) (however, unlike the above process, the process of actually making the light transparent is not performed). When using such a second texture, for example, the "brightness information" of the second color is obtained from the second texture coordinates (shift coordinates) determined using the above parallax mapping technique. Then, the second color may be determined by multiplying the "brightness information" by a predetermined color to be expressed. In addition, the number of colors to be multiplied may be two or more.

[0070] Furthermore, the first texture may be processed using a normal map, a roughness map, etc. in addition to the albedo map. This allows for more diverse expression of the object surface while also allowing the object to have a sense of transparency.

[0071] In addition, regarding the blending of the first color and the second color, the above example shows an example of calculating a blend ratio, but the blend ratio may be specified using other parameters as long as the parameters can specify the ratio between the two colors.

[0072] In the above embodiment, the height information is used as a height map (third texture). In other embodiments, the height information may be embedded in the α value in the image data in RGBα format, for example, and may be integrated into one of the above first textures. In this case, only two textures are required.

[0073] In the above embodiment, the series of processes is executed in a single device. In another embodiment, the series of processes may be executed in an information processing system including a plurality of information processing devices. For example, in an information processing system including a terminal device and a server device capable of communicating with the terminal device via a network, a part of the series of processes may be executed by the server device. Furthermore, in an information processing system including a terminal device and a server device capable of communicating with the terminal device via a network, a main part of the series of processes may be executed by the server device, and a part of the processes may be executed by the terminal device. In the above information processing system, the server system may be configured by a plurality of information processing devices, and the processes to be executed on the server side may be shared and executed by the plurality of information processing devices.

[0074] Alternatively, a so-called cloud gaming configuration may be adopted. For example, the game device 2 may be configured to send operation data indicating a user's operation to a predetermined server, and various processes may be executed in the server, and the execution results may be streamed to the game device 2 as video and audio. [Explanation of symbols]

[0075] 2. Gaming Devices 4. Controller 5 Display section 81 Processor 84 Memory section 87 Video and audio output section

Claims

1. An image processing program for drawing an object arranged in a virtual space, Computer, a management means for managing a first texture in which first color information is associated with each coordinate on the surface of the object, a second texture in which second color information is associated with each coordinate on the surface of the object, and a third texture in which height information is associated with each coordinate on the surface of the object; In drawing the object, determining a first color based on first color information corresponding to coordinates of a surface of the object to be rendered, regardless of a line of sight direction of a virtual camera; determining a second color based on second color information corresponding to coordinates of the surface of the object corrected based on a relationship between a line of sight direction of the virtual camera and a normal vector of the object and height information of the third texture corresponding to the coordinates of the surface of the object to be rendered; determining a drawing color for each coordinate on the surface of the object based on the first color and the second color; drawing means for drawing a surface of the object; This is an image processing program that acts as a

2. 2. The image processing program according to claim 1, wherein the drawing means determines the second color based on second color information corresponding to coordinates of the surface of the object corrected based on a relative relationship between a line of sight direction of the virtual camera and a normal vector of the object and the height information.

3. 3. The image processing program according to claim 2, wherein the drawing means determines the coordinates of the surface of the object such that the amount of correction becomes smaller as the line of sight direction of the virtual camera approaches a direction opposite to a normal direction of the surface of the object.

4. the image processing program further causes the computer to function as a light source arrangement unit that arranges a virtual light source in the virtual space, The drawing means includes: determining the drawing color by combining at least the first color information and the second color information at a predetermined ratio; 2. The image processing program according to claim 1, wherein when the light source is located behind the surface of the object, the drawing color is determined so that the ratio of the second color information is higher than when the light source is located on the surface side of the object.

5. An image processing device that renders an object arranged in a virtual space, a management means for managing a first texture in which first color information is associated with each coordinate on the surface of the object, a second texture in which second color information is associated with each coordinate on the surface of the object, and a third texture in which height information is associated with each coordinate on the surface of the object; In drawing the object, determining a first color based on first color information corresponding to coordinates of a surface of the object to be rendered, regardless of a line of sight direction of a virtual camera; determining a second color based on second color information corresponding to coordinates of the surface of the object corrected based on a relationship between a line of sight direction of the virtual camera and a normal vector of the object and height information of the third texture corresponding to the coordinates of the surface of the object to be rendered; determining a drawing color for each coordinate on the surface of the object based on the first color and the second color; and a drawing means for drawing the surface of the object.

6. An image processing system for rendering an object arranged in a virtual space, comprising: a management means for managing a first texture in which first color information is associated with each coordinate on the surface of the object, a second texture in which second color information is associated with each coordinate on the surface of the object, and a third texture in which height information is associated with each coordinate on the surface of the object; In drawing the object, determining a first color based on first color information corresponding to coordinates of a surface of the object to be rendered, regardless of a line of sight direction of a virtual camera; determining a second color based on second color information corresponding to coordinates of the surface of the object corrected based on a relationship between a line of sight direction of the virtual camera and a normal vector of the object and height information of the third texture corresponding to the coordinates of the surface of the object to be rendered; determining a drawing color for each coordinate on the surface of the object based on the first color and the second color; and a drawing means for drawing the surface of the object.

7. An image processing method executed by a computer of an image processing device that renders an object arranged in a virtual space, comprising: The computer includes: managing a first texture to which first color information is associated for each coordinate on the surface of the object, a second texture to which second color information is associated for each coordinate on the surface of the object, and a third texture to which height information is associated for each coordinate on the surface of the object; In drawing the object, determining a first color based on first color information corresponding to coordinates of a surface of the object to be rendered, regardless of a line of sight direction of a virtual camera; determining a second color based on second color information corresponding to coordinates of the surface of the object corrected based on a relationship between a line of sight direction of the virtual camera and a normal vector of the object and height information of the third texture corresponding to the coordinates of the surface of the object to be rendered; determining a drawing color for each coordinate on the surface of the object based on the first color and the second color; A method of image processing for rendering a surface of the object.

Images