Image processing device, printing system, and image processing program

Abstract

To provide a technique that can reproduce the texture of a print medium.SOLUTION: An image processing device includes: an image data obtaining unit that obtains image data of an image to be printed on a print medium; a parameter obtaining unit that obtains a parameter which is for physical base rendering with the print medium being as a 3D object and which includes a normal map representing a normal direction of a concavo-convex surface of the print medium relating to a reflection direction of light incident to the print medium; a correcting unit that corrects the intensity of an applied effect of the normal map associated with the 3D object; and a rendering executing unit that generates a rendering image representing the print medium on which the image is printed by executing the physical base rendering using the image data and the parameter.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, a printing system, and an image processing program. Background Art

[0002] Patent Document 1 discloses a method of generating parameters for reproducing the texture of a printing sheet based on the brightness of each pixel of a captured image obtained by capturing the printing sheet with a digital camera, and causing a display device to display a printed matter preview that reproduces the reflection of lighting on the printing sheet. Prior Art Literature Patent Literature

[0003] Patent Document 1 Japanese Unexamined Patent Publication No. 2014-194713 Summary of the Invention Problem to be Solved by the Invention

[0004] When a captured image is used to generate parameters for reproducing the texture of a print medium, the printed matter preview is affected by the shooting environment such as the orientation of the camera and the shape of the lighting. Therefore, when the observation environment to be reproduced in the printed matter preview is different from the shooting environment, the texture of the print medium may not be sufficiently reproduced. Accordingly, there is a need for a technology that can reproduce the texture of a print medium without restricting the observation environment. Means for Solving the Problem

[0005] According to a first embodiment of this disclosure, an image processing apparatus is provided. This image processing apparatus includes: an image data acquisition unit that acquires image data of an image to be printed on a printing medium; a parameter acquisition unit that acquires parameters for physically based rendering the printing medium as a three-dimensional (hereinafter referred to as 3D) object, which include a normal map representing the normal direction of the uneven surface of the printing medium with respect to the reflection direction of light incident on the printing medium; a correction unit that corrects the strength of the application effect of the normal map associated with the 3D object; and a rendering execution unit that generates a rendered image representing the printing medium on which the image has been printed by performing physically based rendering using the image data and the parameters.

[0006] A second embodiment of the present disclosure provides a printing system, comprising the image processing apparatus according to the first embodiment, a display unit for displaying the rendered image generated by the image processing apparatus, and a printing apparatus for printing the image data.

[0007] According to a third embodiment of this disclosure, an image processing program is provided. This image processing program causes a computer to perform the following functions: acquire image data of an image to be printed on a print medium; acquire parameters for physically based rendering the print medium as a 3D object, which include a normal map representing the normal direction of the surface of the print medium with respect to the reflection direction of light incident on the print medium; correct the strength of the application effect of the normal map associated with the 3D object; and generate a rendered image representing the print medium on which the image has been printed by performing the physically based rendering using the image data and the parameters. [Brief explanation of the drawing]

[0008] [Figure 1] A schematic diagram showing an image processing apparatus of an embodiment. [Figure 2A] A flowchart illustrating the color conversion process. [Figure 2B] A flowchart illustrating other configuration examples for color conversion processing. [Figure 2C] A flowchart illustrating other configuration examples for color conversion processing. [Figure 3] An explanatory diagram showing the logical configuration of the rendering execution unit of the embodiment. [Figure 4] A schematic diagram illustrating an example of how an image can be displayed on a printed medium. [Figure 5] An explanatory diagram showing the relationship between the light source, viewpoint, and the angles of the surfaces of a 3D object. [Figure 6] A flowchart illustrating the display processing routine. [Figure 7] A schematic diagram illustrating how the display on a printed medium changes as the angle of the surface of the printed medium relative to the light source changes. [Figure 8] A diagram illustrating the configuration of the pixel pipeline. [Figure 9] A perspective view showing an example of an observation environment. [Figure 10] A side view showing an example of an observation environment. [Figure 11] A schematic diagram illustrating the distribution of the application ratio of normal maps for glossy paper. [Figure 12] A schematic diagram illustrating the distribution of the application ratio of normal maps for matte paper. [Figure 13] An explanatory diagram showing an example of a preview image for glossy paper. [Figure 14] An explanatory diagram showing an example of a preview image for matte paper. [Figure 15] A schematic diagram showing an embodiment of the printing system. [Figure 16] An explanatory diagram showing examples of how printed results appear in other printing media. [Modes for carrying out the invention]

[0009] A. First Embodiment: (A1) Hardware configuration: A schematic configuration of an image processing apparatus 100 according to the present embodiment is shown in FIG. 1. This image processing apparatus 100 performs image processing for previewing a state where an image is printed on a predetermined print medium. The image processing apparatus 100 not only performs image processing, but also displays the processing result as a preview image. As illustrated, the image processing apparatus 100 includes a color management system 111 that mainly performs color conversion, a rendering execution unit 121 that executes rendering of a print medium, a memory 135 including a first storage unit 131 and a second storage unit 132, a communication unit 141 that exchanges data with an external site 200 via a network NW such as the Internet, and an image display unit 151 that displays a preview image. Note that a program for executing each process described later is stored in the memory 135 or the like of the image processing apparatus 100, and each function of the image processing apparatus 100 is realized when a CPU or a GPU executes the program stored in the memory.

[0010] Hereinafter, the color management system may be abbreviated as CMS for simplicity. The CMS 111 acquires image data ORG representing an image to be printed (hereinafter referred to as an original image). The CMS 111 may acquire the image data ORG by wired or wireless communication from an image forming apparatus that has generated the image data ORG, may acquire the image data ORG from a storage medium such as a memory card storing the image data ORG, or may acquire the image data ORG via a network. The CMS 111 may acquire the image data ORG generated in the image processing apparatus 100. The image data ORG acquired by the CMS 111 is supplied to a printing apparatus (not shown) connected to the image processing apparatus 100 via wired or wireless communication, and is printed on a print medium by the printing apparatus. The image data ORG may be supplied from the image processing apparatus 100 to the printing apparatus via a storage medium such as a memory card. Note that the CMS 111 may be referred to as an image data acquisition unit.

[0011] CMS 111 performs color conversion on image data ORG, and converts it into colors expressed on a print medium by printing. The image data ORG subjected to color conversion is referred to as managed image data MGP. Details of the CMS 111 processing will be described later. The managed image data MGP is used as a texture of a print medium represented by a 3D object. An input profile IP, a media profile MP, a common color space profile CP, and the like are input to the CMS 111. The input profile IP converts a device-dependent color system such as RGB into L, which is a device-independent color system * a * b * (hereinafter simply abbreviated as Lab) and the like. The media profile MP is a profile representing color reproducibility when printing on a specific print medium under printing conditions such as a specific printing resolution by a specific printing device such as a printer, and is a profile for converting color values between a device-independent color system and a device-dependent color system. The media profile MP also includes information such as print settings of the printing device other than the print medium. For this reason, when attempting to cover all combinations of printing device (printer) × print medium × print settings, the number of types of media profiles MP increases. Therefore, when the dependency on printing conditions is small, or when it is not desired to increase the number of profiles, the media profile MP may be configured as a combination of printing device (printer) × print medium. As described above, since the characteristics of the printing device and the characteristics of the print medium itself are involved in the color of the image on the print medium, the media profile MP may be hereinafter referred to as a print profile MP.

[0012] Applying the input profile IP to the image data ORG, and then applying the print profile MP, yields color values ​​that would appear under specific printing conditions, i.e., color values ​​dependent on the printing device and medium. Applying the print profile MP to these image color values ​​to convert them from a device-dependent color system to a device-independent color system, and then applying the common color space profile CP, converts them to a representation in the second color space used during rendering (in this case, the sRGB color space). Because the image data ORG has been converted once using the print profile MP to color values ​​dependent on the characteristics of the printing device and medium, the image data ORG is color-converted to a range of actually printable color values. The common color space profile CP is used to convert the image data ORG to color values ​​in the color space used during rendering. The sRGB color space is a typical common color space, but AdobeRGB, Display-P3, etc., may also be used.

[0013] As described above, CMS111 uses each profile to convert image data ORG, which is expressed in the first color space, a device-dependent color system, into image data (managed image data) MGP, which is expressed in the sRGB color space, the second color space used during rendering. Here, the converted image data MGP is not limited to color values ​​in the sRGB color space, but can be expressed in any color space that the rendering execution unit 121 can handle. For example, if the rendering execution unit 121 employs a configuration that allows rendering using color values ​​or spectral reflectance in Lab or XYZ color spaces, the image data can be converted to the color values ​​used when displaying on the image display unit 151 during the lighting processing (described later) performed within the rendering execution unit 121, or in the post-processing unit (described later) located after the rendering execution unit 121.

[0014] Memory 135 stores the first data FD in the first storage unit 131 and the second data SD in the second storage unit 132. The first data FD and the second data SD are parameters necessary to represent a printed medium on which an image has been printed using physically based rendering. The first data FD includes 3D object information relating to the form of a 3D object representing the printed medium and placed in the virtual space, camera information relating to the position of a camera placed in the virtual space, lighting information relating to the position and color of a light source placed in the virtual space, and background information relating to the background represented in the virtual space. The second data SD includes data for representing the texture of the printed medium on the 3D object. These first data FD and second data SD are used during rendering by the rendering execution unit 121.

[0015] For the first data FD and second data SD, representative data used more than a predetermined frequency can be stored non-volatilely in the first storage unit 131 and second storage unit 132 in advance, and can be selected as needed and referenced by the rendering execution unit 121. For printing media that are not normally used and are used infrequently, such as texture data for special materials such as cloth, cans, and plastic sheets, the data may be stored at an external site 200 and retrieved via the communication unit 141 when needed. For the first data FD, such as lighting information, the user may specify it individually during rendering, but representative camera angles and light sources may be stored in the first storage unit 131 in advance and used. The camera angle is the position and direction from which the target printing medium is viewed, and corresponds to the position and direction of the virtual viewpoint of the user viewing the virtual space. For this reason, the camera is sometimes described as the viewpoint or direction of the line of sight, and is referred to as the "viewpoint" or "view". The part of the image processing device 100 that acquires parameters from outside is sometimes called the parameter acquisition unit.

[0016] The image display unit 151 displays the image of the print medium rendered by the rendering execution unit 121, along with the background and other elements. The image display unit 151 reads the image data for display from the frame memory FM provided in the rendering execution unit 121 and displays it. The image display unit 151 may be provided in the image processing device 100, or it may be provided separately from the image processing device 100. The image processing device 100 may be implemented as a dedicated machine, or it may be implemented by running an application program on a computer. Of course, the computer includes terminals such as tablets and mobile phones. Since the processing of the rendering execution unit 121 requires considerable resources and computing power, the rendering execution unit 121 alone may be run by a CPU capable of high-speed processing or a dedicated GPU, and the image processing device 100 may be configured with the rendering execution unit 121 located at a separate site on the network. The image display unit 151 is sometimes referred to as the display unit.

[0017] The color conversion process performed by CMS111 will be explained using Figure 2A. The figure is a flowchart showing the process by which CMS111 converts the original image data ORG into color data in a common color space for rendering. When the color conversion process starts, first the original image data ORG and the input profile IP are input, and the original image data ORG, which is represented in a device-dependent color system (e.g., RGB color system), is converted into color data in a device-independent color system (e.g., Lab or XYZ color system) (step S110). Next, it is determined whether a media profile MP is available (step S120), and if a media profile MP exists, it is applied, and the color conversion is performed to the range of colors that can be expressed by printing, taking into account the combination of printing device (printer) and printing medium as printing conditions (step S130). If there is no media profile MP, the process in step S130 is not performed. After that, the color values ​​are converted to the color space of the common color space, which is the second color space used during rendering, using the common color space profile CP (step S150). In this embodiment, sRGB is used as the common color space. The managed image data MGP obtained in this way is set to the albedo color, which is the texture of the 3D object (step S160), and this processing routine is terminated.

[0018] In step S130, setting the rendering intent for color conversion in the media profile to absolute allows the color of the print medium itself (background color) to be reflected. If the color value of the image to be converted in step S150 is outside the sRGB color space gamut, it may be approximated to a value within the sRGB color space, or it may be treated in a way that allows it to take values ​​outside the sRGB color space. Generally, the RGB values ​​of image data are stored as 8 bits for each color, i.e., integers from 0 to 255. However, if pixel values ​​are represented as floating-point numbers from 0.0 to 1.0 instead, values ​​outside the sRGB color space can be handled as negative values ​​or values ​​greater than 1.0.

[0019] Color conversion by CMS111 is not limited to the configuration shown in Figure 2A; it can also be performed using configurations such as those shown in Figures 2B and 2C. Figure 2B shows the color conversion processing routine when correction data DPD for the image display unit 151 is provided. The display device correction data DPD is data used to correct the shift in the display colors of the image display unit 151 relative to the common color space sRGB. In the color conversion process shown in Figure 2B, color conversion using the media profile MP (step S130) is followed by color conversion processing using the display device correction data DPD (step S140).

[0020] Alternatively, a composite correction data SPD may be prepared in advance by combining the display device correction data DPD and the common color space profile CP, and the color conversion using the composite correction data SPD (step S155) may be performed instead of the color conversion using the common color space profile CP (step S150). An example of the color conversion process in this case is shown in Figure 2C. Note that the correction for the shift in the display color of the image display unit 151 may be performed in the post-processing unit PST after the render backend, as shown in Figure 3, which will be described later, instead of being performed in the CMS 111.

[0021] The rendering execution unit 121 renders the managed image data MGP output by the CMS 111 after color conversion, displaying on the image display unit 151 how the printed medium on which the original image data ORG is printed would look in a virtual space. An example configuration of the rendering execution unit 121 is shown in Figure 3. This rendering execution unit 121 shows a typical configuration for physically based rendering processing, and other configurations can also be adopted. The rendering execution unit 121 of this embodiment employs a pipeline configuration including a vertex pipeline VPL and a pixel pipeline PPL, and performs physically based rendering at high speed. The vertex pipeline VPL includes a vertex shader VS and a geometry shader GS. A configuration without using the geometry shader GS is also possible.

[0022] The vertex shader VS converts the coordinates of the vertices of a 3D object (print medium) on the print medium to the coordinates of the 3D space to be rendered. The coordinate transformation comprehensively includes coordinate transformations such as the coordinates of the model (in this case, the print medium) → world coordinates → view (camera) coordinates → clip coordinates, but the transformation to view coordinates is performed by the geometry shader GS. In addition, the vertex shader VS also performs shading and calculates texture coordinates (UV). In performing these processes, the vertex shader VS and geometry shader GS refer to 3D object information TOI, camera information CMR, lighting information LGT, and background information BGD stored in the first memory unit 131.

[0023] 3D object information (TOI) is information about the shape of 3D objects placed in the virtual space. 3D objects are composed of multiple polygons and represent the shape of a printed medium. Since the surface of a real printed medium has minute irregularities that affect its texture, it is preferable that these minute irregularities be represented on the surface of the 3D object. However, reducing the size of each polygon constituting the 3D object to represent minute irregularities results in a huge number of polygons and a heavy computational load. Therefore, normal maps and height maps are sometimes used to represent minute irregularities on 3D objects. Normal maps and height maps are included in the texture parameters described later. Camera information (CMR) represents the position and orientation of the camera in the virtual space. Lighting information (LGT) includes at least one piece of information such as the position, angle, intensity, and color temperature of the light source in the virtual space. Note that multiple light sources may be placed in the virtual space; in this case, the effects of the multiple light sources can be calculated separately and then superimposed on the 3D object.

[0024] Background information (BGD) does not necessarily have to be used for rendering, but it is information used to represent the background in the virtual space. Background information (BGD) includes information about objects such as walls, floors, and furniture placed in the virtual space, and these objects are rendered by the rendering execution unit 121 in the same way as print media. In addition, since lighting hits these background objects and illuminates the print medium, it is also treated as part of the lighting information. By performing rendering using this various information, a three-dimensional preview becomes possible. The vertex information calculated by the vertex shader VS is passed to the geometry shader GS.

[0025] The geometry shader GS is used to manipulate the set of vertices within a 3D object. The geometry shader GS allows for increasing or decreasing the number of vertices at runtime, and changing the types of primitives that make up a 3D object. One example of increasing or decreasing the number of vertices is culling, which excludes vertices that are not visible to the camera based on the camera's position and orientation. The geometry shader GS also generates new primitives from existing primitives such as points, lines, and triangles. The geometry shader GS receives primitives from the vertex shader VS, either the entire primitive or a primitive containing information about adjacent primitives. The geometry shader GS processes the input primitives and outputs a rasterized primitive.

[0026] The output of the vertex pipeline (VPL), specifically the primitives processed by the geometry shader (GS), is rasterized by the rasterizer (RRZ) to obtain pixel-level data, which is then passed to the pixel pipeline (PPL). In this embodiment, the pixel pipeline (PPL) comprises a pixel shader (PS) and a render backend (RBE).

[0027] The pixel shader PS manipulates rasterized pixels, and simply put, calculates the color of each pixel. Based on information input from the vertex shader VS and geometry shader GS, it performs processes such as compositing textures and applying surface colors. The pixel shader PS maps managed image data MGP, which is obtained by converting image data ORG using CMS111 based on various profiles, onto the print medium as a 3D object. At this time, the lighting processing function provided in the pixel shader PS performs lighting processing and maps the managed image data MGP based on the object's light reflection model, the illumination information LGT mentioned above, and the texture parameter TXT, which is one of the second data SD stored in the second storage unit 132. The reflection model used in the lighting processing is one of the mathematical model calculation formulas for simulating illumination phenomena in the real world. The reflection model used in this embodiment will be explained in detail later.

[0028] Pixel manipulation processes become computationally intensive and time-consuming when the number of rasterized pixels increases, such as when the output resolution is high. Therefore, compared to vertex-level processing, it can take longer and result in inefficient pipeline processing. In this embodiment, the pixel shader PS processing program is optimized for execution on a GPU with high parallel processing capabilities, enabling advanced effects, including texture rendering, to be achieved in a short time.

[0029] The pixel information obtained through the processing of the pixel shader PS is further evaluated by the render backend (RBE) to determine whether or not to write it to the frame memory (FM) for display. Only when the render backend (RBE) determines that it is safe to write the pixel data to the frame memory (FM) is the pixel data saved as something to be rendered. Well-known tests used to determine whether to write include the "alpha test," "depth test," and "stencil test." The render backend (RBE) executes the configured test from among these tests and writes the pixel data to the frame memory (FM).

[0030] With the above processing completes the rendering pipeline, the post-processing unit (PST) then performs processing on the data stored in the frame memory (FM) to improve its appearance. Examples of such processing include anti-aliasing, which smooths out unwanted edges in the image. Other processes include ambient occlusion, screen-space reflection, and depth of field. The post-processing unit (PST) should be configured to perform the necessary post-processing.

[0031] The rendering execution unit 121 completes the above processing, and the rendering is finished. The result is output as a render result RRD. In practice, the data written to the frame memory FM is read out in accordance with the display cycle of the image display unit 151 and displayed as a render result RRD. An example of a render result RRD is shown in Figure 4. In this example, the image display unit 151 displays a print medium PLb as a 3D object placed in a virtual space, a light source LG, and a background object Bob such as furniture that exists as one of the backgrounds.

[0032] Figure 5 illustrates the relationship between the print medium PLb placed in a virtual space, the light source LG, and the viewpoint (camera) VP. The relationship between the light source LG, viewpoint VP, and print medium PLb is three-dimensional within the virtual space VSP, but the figure shows the virtual space VSP as the xz plane. x is the coordinate of the point where each of the vectors described below converges. The figure illustrates the positional relationship between the light source LG and viewpoint VP that illuminate the print medium PLb, which is the object of rendering, at a given coordinate x. The figure shows the light source direction vector ωl from coordinate x to the light source LG, the viewpoint direction vector ωv from coordinate x to the viewpoint VP, and the half vector HV of both. Furthermore, the symbol Np represents the normal vector assuming that the print medium PLb is a perfect plane PLp, and the symbol Nb represents the normal vector at coordinate x of the actual print medium PLb, which is not a perfect plane. Note that in Figure 4, the rendering result of the print medium PLb is illustrated assuming that the viewpoint VP (camera) is located almost directly in front of the print medium PLb.

[0033] In the image processing device 100 of this embodiment, the position and angle of the print medium in the virtual space can be freely changed, and the appearance of the print medium and the image on the print medium can be checked. This is because, as shown in Figure 6, the image processing device 100 repeatedly performs a series of processes: operating a pointing device on the image displayed on the image display unit 151 (step S210), if there is a change in the instruction from the pointing device (step S220: "YES"), the rendering execution unit 121 performs the rendering process again (step S230), and the result of the processing is displayed on the image display unit 151 (step S240). The pointing device may be a 3D mouse or a tracking ball, or it may be a type that is operated with a finger or stylus on a multi-touch panel provided on the image display unit 151. For example, if a multi-touch panel is provided on the surface of the image display unit 151, the print medium PLb and the light source LG may be moved directly with a finger, or two fingers may be used to rotate the print medium PLb or change the distance between the light source LG and the print medium PLb in three dimensions.

[0034] When the position and angle of the print medium PLb or light source LG in this virtual space are changed, the rendering execution unit 121 performs rendering processing each time, and the rendered result RRD is displayed on the image display unit 151. An example of such a display is shown in Figure 7. As shown in the figure, when the position and angle of the print medium PLb or light source LG in the virtual space are changed, the print medium on which the image is printed is physically rendered each time, and the actual print medium on which the image is printed is displayed in a manner that is close to how it would appear in real space.

[0035] In particular, in this embodiment, in addition to converting the colors of the image to be printed on the print medium to the actual colors of the printed image using a color management system (CMS), during the lighting process during rendering, [1] The printing medium on which the image is printed is treated as a 3D object. [2] The texture parameter TXT is used to consider the surface texture of the print medium. Therefore, the reproducibility of the printed material displayed on the image display unit 151 is extremely high. The following describes the processes of [1] and [2].

[0036] [1] How a 3D object appears in a virtual space can be represented using the bidirectional reflectance distribution function (BRDF) and luminance of the reflected light at each part of the object. The bidirectional reflectance distribution function (BRDF) indicates the angular distribution characteristics of the reflected light when light is incident from a specific angle. Luminance is the brightness of the object. Together, these are called the illumination model. An example of the reflection model adopted in this embodiment is shown below. The BRDF can be expressed as a function f(x,ωl,ωv), and the luminance as a function L(x,ωv), as shown in the following equations (1) and (2). f(x,ωl,ωv)=kD / π+kS*(F*D*V) …(1) L(x,ωv)=f(x,ωl,ωv)*E⊥(x)*n·ωl …(2) x: coordinate within the plane, ωv: viewpoint direction vector, ωl: light source direction vector kD: Diffuse Albedo, kS: Specular Albedo F: Fresnel term, D: Normal distribution function, V: Geometric decay term E⊥(x): Illuminance incident perpendicular to coordinate x, n: normal vector

[0037] The first term of the BRDF, kD / π, is the diffuse reflection component and is based on the Lambert model. The second term is the specular reflection component and is based on the Cook-Torrance model. In equation (1), kd / π is sometimes called the diffuse reflection term, and kS*(F*D*V) is sometimes called the specular reflection term. The Fresnel term F, the normal distribution function D, and the geometric decay term V are well-known models and calculation methods, so their explanation is omitted. As for the BRDF, any function appropriate to the reflection characteristics of the 3D object surface and the purpose of rendering may be used. For example, the Disney Principled BRDF may be used. In this embodiment, the BRDF is used as a function to represent light reflection, but the Bidirectional Scattering Surface Reflectance Distribution Function (BSSRDF) may also be used as a function to represent light reflection.

[0038] As can be seen from equations (1) and (2) above, the calculation of the above reflection model requires the normal vector n, the light source direction vector ωl, and the viewpoint direction vector ωv. Printed media are treated as 3D objects composed of multiple minute polygons for rendering purposes, but the normal vector n, which reflects the minute irregularities on the surface of the printed media, is calculated from the polygon normal Np and the normal map described later. Therefore, the vertex pipeline VPL calculates the polygon normal Np and the UV coordinates that determine the reference position of the normal map, and inputs these, along with the light source direction vector ωl and the viewpoint direction vector ωv, into the pixel pipeline PPL. In the pixel pipeline PPL, the pixel shader PS references the normal map, which is given as one of the texture parameters, using the UV coordinates, and calculates the normal vector n from the referenced normal map value and the polygon normal Np.

[0039] In this embodiment, as described above, the printing medium on which the image is printed is treated as a 3D object, and physically based rendering is performed using equations (1) and (2) above. The light source direction vector ωl and the viewpoint direction vector ωv are calculated each time the user changes the position or angle of the printing medium PLb or light source LG in the virtual space using a pointing device, as shown in Figure 7.

[0040] Regarding [2]: In this embodiment, the surface texture of the printing medium is considered using the texture parameter TXT. The texture parameter TXT may include the following, but it is not necessary to consider all of them; at least one of the parameters listed below, for example, smoothness, should be considered. • Smoothness (S) or Roughness (R): This parameter indicates the smoothness of the surface of a 3D object. Smoothness S is generally specified within the range of 0.0 to 1.0. Smoothness S affects the normal distribution function D and the geometric attenuation term V in equation (1) BRDF mentioned above. A larger value results in stronger specular reflection and a glossy appearance. Roughness R can be used instead of smoothness S. The two are convertible using the formula S = 1.0 - R. Note that smoothness is sometimes referred to as "smoothness," and roughness as "roughness."

[0041] ·Metallic M (metallic): This indicates the degree to which the surface of a 3D object is metallic. The higher the metallicity of the surface, the larger the value of metallicity M. When metallicity M is high, the object's surface reflects light from the surroundings more easily, and the reflection of the surrounding scenery makes it easier to hide the color of the object itself. Metallicity M affects the Fresnel term F. The Fresnel term F can be expressed as equation (3) below using the Schlick approximation. F(ωl,h)=F0+(1-F0)(1-ωl·h) 5 …(3) Here, h is the half-vector of the viewpoint direction vector ωv and the light source direction vector ωl, and F0 is the specular reflectance when perpendicularly incident. The specular reflectance F0 can be directly specified as the specular color of the specularly reflected light, or it can be given by linear interpolation (referred to here as the lerp function) using the metallicity M in equation (4). F0 = lerp(0.04, tC, M) …(4) Here, tC is the texture color (albedoColor) of the 3D object. Note that the value 0.04 in equation (4) represents typical RGB values ​​for nonmetals. The same applies to the texture color tC.

[0042] • Normal Map: The normal map represents the normal vectors of the minute surface irregularities on the printing medium. By associating (applying) the normal map to a 3D object, the normal vectors of the minute surface irregularities on the printing medium can be assigned to the 3D object. The normal map can influence the Fresnel term F, normal distribution function D, and geometric decay term V of the BRDF. It should be noted that normal maps can be generated from general RGB images using known techniques, and there are also providers who offer services to generate normal maps from RGB images. Therefore, users of the image processing device 100 can either generate the normal map themselves or obtain one by using the aforementioned service.

[0043] • Other texture parameters: Other parameters that can function as texture parameters include specular color, and clear coat layer parameters that indicate the presence or absence of a clear coat layer on the surface of the printing medium, its thickness, or its transparency.

[0044] As explained above, [1] Treat the printing medium on which the image is printed as a 3D object, [2] The texture parameter TXT is used to consider the surface texture of the print medium. As a result, the image processing apparatus 100 of this embodiment can display the appearance of the printed medium with an image on the image display unit 151 with a high degree of freedom and high reproducibility. As illustrated in Figure 4, when viewed from a direction directly facing the printed medium, the texture of the printed medium's surface and the roughness resulting from the fine irregularities on the surface of the printed medium are displayed. As illustrated in Figure 7, when the printed medium is rotated and viewed from an oblique direction, the illumination from the light source LG is reflected on the surface of the printed medium, and the resulting highlight area HLT is displayed. Note that the illumination light is not limited to illumination directed directly at the printed medium, such as a spotlight, but also includes sunlight, indirect lighting, and indirect light.

[0045] When glossy paper is observed with the naked eye, it appears flat at first glance. However, when the glossy paper is observed from the direction of specular reflection of light reflected from its surface, the shape of the highlight areas (HLT) on the surface of the glossy paper appears wavy. This is because, under the observation conditions described above, minute irregularities in the substrate of the glossy paper beneath the clear coat layer become visible. Furthermore, in glossy paper, the appearance of the substrate irregularities differs between the highlight areas (HLT) and areas outside the highlight areas. The substrate irregularities are clearly visible in the highlight areas (HLT), but not so clearly visible outside the highlight areas.

[0046] On the other hand, matte paper and plain paper do not have a clear coat layer, and their surfaces are less smooth than glossy paper, resulting in a smaller specular reflection component. Therefore, on matte paper and plain paper, the highlight (HLT) is not clearly visible, and the appearance of surface irregularities is almost uniform. However, even on matte paper and plain paper, the appearance of surface irregularities is angle-dependent. Specifically, on matte paper and plain paper, surface irregularities are less likely to appear under observation conditions where the paper surface is observed from the specular reflection direction, or under observation conditions where light from the light source (LG) is incident perpendicularly on the paper surface, while surface irregularities are more likely to appear under other observation conditions.

[0047] Thus, the way surface irregularities appear differs depending on the type of printing medium. In this embodiment, the differences in how surface irregularities appear for each type of printing medium are reproduced by applying a normal map to a 3D object in the pixel pipeline PPL and correcting the strength of the normal map application effect at each point on the 3D object.

[0048] Figure 8 is an explanatory diagram showing the configuration of the Pixel Pipeline PPL. In this embodiment, the Pixel Pipeline PPL comprises a selection unit SL, a first pixel shader PS1, a second pixel shader PS2, and a render backend RBE. The first pixel shader PS1 is provided with a first correction unit CR1, and the second pixel shader PS2 is provided with a second correction unit CR2. The first correction unit CR1 and the second correction unit CR2 each correct the strength of the normal map application effect at each point on the 3D object. The first correction unit CR1 and the second correction unit CR2 use different methods to correct the strength of the normal map application effect. The selection unit SL selects the method for correcting the strength of the normal map application effect by selecting one of the first pixel shader PS1 and the second pixel shader PS2 to perform processing. The render backend RBE receives information from the one of the first pixel shader PS1 and the second pixel shader PS2 that has performed processing.

[0049] In this embodiment, the selection unit SL selects whether to correct the strength of the normal map application effect by the first correction unit CR1 of the first pixel shader PS1 or by the second correction unit CR2 of the second pixel shader PS2, depending on the smoothness S included in the texture parameter. Specifically, the selection unit SL causes the first correction unit CR1 to perform correction if the smoothness S is above a threshold, and causes the second correction unit CR2 to perform correction if the smoothness S is below a threshold. The threshold is set to, for example, 0.5. Note that the smoothness S of glossy paper is above the threshold, while the smoothness S of matte paper and plain paper is below the threshold. Therefore, the method of correcting the strength of the normal map application effect differs between glossy paper and matte or plain paper.

[0050] A method of correcting the strength of the application effect of a normal map by the first correction unit CR1 of the first pixel shader PS1 will be described. The first correction unit CR1 performs correction for glossy paper. First, the first correction unit CR1 calculates a temporary specular reflection component (ST=F*D*V) using the polygon normal vector Np, and then calculates the normal vector n used for BRDF and luminance calculation using the following equation (5). n=lerp(Np,Nb,saturate(ST)) …(5) Here, n is the normal vector used for BRDF and luminance calculation, Np is the polygon normal vector, Nb is the normal vector of the normal map, and ST is the temporary specular reflection component. lerp(Np,Nb,saturate(ST)) is a function that linearly interpolates between Np and Nb by saturate(ST). saturate(ST) is a function that clamps ST to the range of 0.0 to 1.0. When ST>1.0, saturate(ST)=1.0, and when 0.0≦ST≦1.0, saturate(ST)=ST.

[0051] From equation (5), if the provisional specular reflection component exceeds 1.0, the direction of the normal vector n coincides with the direction of the normal vector Nb of the normal map, and if the provisional specular reflection component is 0.0, the direction of the normal vector n coincides with the direction of the polygon's normal vector Np. As the specular reflection component increases, the direction of the normal vector n becomes closer to the direction of the normal vector Nb of the normal map, and conversely, the closer the specular reflection component is to 0.0, the closer the direction of the normal vector n becomes to the direction of the polygon's normal vector Np. In other words, the first correction unit CR1 corrects the strength of the normal map application effect so that the effect of applying the normal map is stronger for pixels with a large specular reflection component, and weaker for pixels with a small specular reflection component. The strength of the normal map application effect is also called the normal map application ratio. The normal map application ratio is expressed as a value between 0.0 and 1.0. The closer the normal map application ratio is to 1.0, the closer the direction of the normal vector n becomes to the direction of the normal vector Nb in the normal map. The closer the normal map application ratio is to 0.0, the closer the direction of the normal vector n becomes to the direction of the polygon's normal vector Np. For example, when the normal map application ratio is 0.0, the direction of the normal vector n is the same as the direction of the polygon's normal vector Np. When the normal map application ratio is 1.0, the direction of the normal vector n is the same as the direction of the normal vector Nb in the normal map. When the normal map application ratio is 0.5, the direction of the normal vector n is exactly halfway between the direction of the polygon's normal vector Np and the direction of the normal vector Nb in the normal map.

[0052] This section describes how to adjust the strength of the normal map application effect by the second correction unit CR2 of the second pixel shader PS2. The second correction unit CR2 performs corrections for matte paper and plain paper. The second correction unit CR2 calculates the normal vector n used for calculating BRDF and luminance using the following equation (6). n = lerp(Nb, Np, nh*nl) …(6) Here, n is the normal vector used in the calculation of BRDF and brightness, Nb is the normal vector of the normal map, Np is the normal vector of the polygon, nh is the dot product of the polygon normal vector Np and the half vector HV, and nl is the dot product of the polygon normal vector Np and the light source direction vector ωl. nh*nl takes values ​​from 0.0 to 1.0. lerp(Nb,Np,nh*nl) is a function that linearly interpolates Nb and Np by nh*nl.

[0053] From equation (6), when the value of nh*nl is 1.0, the direction of the normal vector n coincides with the direction of the polygon's normal vector Np, and when the value of nh*nl is 0.0, the direction of the normal vector n coincides with the direction of the normal vector Nb of the normal map. As the value of nh*nl increases, the direction of the normal vector n becomes closer to the direction of the polygon's normal vector Np, and conversely, as the value of nh*nl approaches 0.0, the direction of the normal vector n becomes closer to the direction of the normal vector Nb of the normal map.

[0054] The smaller the angle between the polygon's normal vector Np and half-vector HV, the larger the dot product nh between the polygon's normal vector Np and half-vector HV. Therefore, the smaller the angle between the polygon's normal vector Np and half-vector HV, the larger the value of nh*nl becomes, and the weaker the effect of the normal map application becomes. In other words, the second correction unit CR2 corrects the strength of the normal map application effect so that the effect of the normal map application becomes weaker for pixels with a small angle between the polygon's normal vector Np and half-vector HV. The smaller the angle between the polygon's normal vector Np and half-vector HV, the closer the positional relationship between the viewpoint VP and light source LG is to the positional relationship of specular reflection. Therefore, when correction is performed by the second correction unit CR2, the closer the positional relationship between the viewpoint VP and light source LG is to the positional relationship of specular reflection, the weaker the effect of the normal map application becomes.

[0055] Furthermore, the smaller the angle between the polygon's normal vector Np and the light source direction vector DL, the larger the dot product nl between the polygon's normal vector Np and the light source direction vector DL. Therefore, the smaller the angle between the polygon's normal vector Np and the light source direction vector DL, the larger the value of nh*nl becomes, and the weaker the effect of applying the normal map becomes. In other words, the second correction unit CR2 corrects the strength of the normal map's application effect so that the effect of applying the normal map becomes weaker for pixels with a small angle between the polygon's normal vector Np and the light source direction vector DL. A small angle between the polygon's normal vector Np and the light source direction vector DL ​​means that the degree of alignment between the polygon's surface PLp and the light source LG is high. Therefore, when correction is performed by the second correction unit CR2, the higher the degree of alignment between the polygon's surface PLp and the light source LG, the weaker the effect of applying the normal map becomes.

[0056] Figure 9 is a perspective view showing an example of an observation environment. Figure 10 is a side view showing an example of an observation environment. Figure 11 is a schematic explanatory diagram showing the distribution of the application ratio of the normal map in the case of glossy paper, and Figure 12 is a schematic explanatory diagram showing the distribution of the application ratio of the normal map in the case of matte paper. Figure 13 is an explanatory diagram showing an example of a preview image in the case of glossy paper, and Figure 14 is an explanatory diagram showing an example of a preview image in the case of matte paper. Figures 9 and 10 show how a 3D object OBJ illuminated by a light source LG is observed from a viewpoint VP positioned diagonally above the 3D object OBJ. The 3D object OBJ represents a sheet of printing paper placed on a flat surface such as a desk. The light source LG is positioned directly above the 3D object OBJ. The light source LG is a rod-shaped fluorescent lamp positioned parallel to the surface of the 3D object OBJ. Figures 12 and 14 show preview images under the observation environment shown in Figures 9 and 10.

[0057] As shown in Figure 11, when correction is performed by the first correction unit CR1, the application ratio of the normal map is higher for pixels with a strong specular reflection component, i.e., pixels corresponding to the highlight area. On the other hand, the application ratio of the normal map is lower for pixels with a small specular reflection term, i.e., pixels outside the highlight area. In Figure 11, the first region R1 in the center of the 3D object OBJ is the highlight area, and the application ratio of the normal map is 1.0 in the first region R1. The application ratio of the normal map decreases as you move away from the first region R1, becoming 0.6 in the second region R2 surrounding the first region R1, and 0.0 in the third region R3 surrounding the second region R2. As a result, as shown in Figure 13, the unevenness of the substrate is clearly visible in the highlight area, while the unevenness of the substrate is not clearly visible outside the highlight area, reproducing the texture unique to glossy paper. Furthermore, if the position and orientation of the viewpoint (VP), the position and orientation of the light source (LG), or the orientation of the 3D object (OBJ) are changed, the location where the bumpy pattern appears will also change.

[0058] As shown in Figure 12, when correction is performed by the second correction unit CR2, the application ratio of the normal map decreases in pixels with a strong specular reflection component, i.e., pixels corresponding to the highlight area. Furthermore, the application ratio of the normal map decreases for pixels where the polygon face PLp of the 3D object OBJ is directly facing the light source LG. In Figure 12, the first region R1 in the center of the 3D object OBJ is the highlight area, and the application ratio of the normal map in the first region R1 is 0.8. The application ratio of the normal map increases as you move away from the first region R1, becoming 0.9 in the second region R2 surrounding the first region R1, and 1.0 in the third region R3 surrounding the second region R2. As a result, as shown in Figure 14, the appearance of surface irregularities is almost uniform. Under observation conditions where the paper surface is observed from the specular reflection direction, or where light from the light source LG is incident perpendicularly on the paper surface, surface irregularities are less noticeable, while under other observation conditions, surface irregularities are more noticeable, thus reproducing the texture unique to matte paper and plain paper. Note that in Figures 11 and 13, the application ratio of the normal map changes in three stages, but in reality, the application ratio of the normal map changes more finely.

[0059] As described above, the image processing apparatus 100 of this embodiment corrects the strength of the application effect of the normal map associated with the 3D object and then performs physically based rendering to generate a rendered image. Therefore, the texture of the printed medium can be reproduced without restrictions on the viewing environment. In particular, in this embodiment, the image processing apparatus 100 uses different methods for correcting the strength of the application effect of the normal map depending on whether the smoothness S of the printed medium is above a threshold or below a threshold. As a result, the texture unique to glossy paper with high smoothness S, and the texture unique to matte paper or plain paper with low smoothness S can be realistically reproduced.

[0060] In the embodiment described above, the pixel pipeline PPL is provided with two pixel shaders PS1 and PS2, but in other embodiments, the number of pixel shaders PS provided in the pixel pipeline PPL may be just one. In this case, the correction unit of the pixel shader PS may correct the strength of the normal map application effect according to the following equation (7). n = lerp(n1, n2, S) …(7) Here, n1 is the normal vector calculated using the correction method when the smoothness S of the printing medium is above the threshold, n2 is the normal vector calculated using the method when the smoothness S of the printing medium is below the threshold, and S is the smoothness of the printing medium. This method can also reproduce the difference between the texture of glossy paper and matte paper as described above. Note that if there is only one pixel shader PS in the pixel pipeline PPL, the selection section SL does not need to be provided.

[0061] B. Second Embodiment: The second embodiment is in the form of a printing system 300. As shown in Figure 15, this printing system 300 comprises the image processing device 100 described above, an image preparation device 310, and a printing device 320. In this embodiment, the image preparation device 310 is a computer used by the user and is a device that prepares image data ORG, which is image data represented in a first color space. This image preparation device 310 may have the function of creating images, or it may simply store image data and provide it to the image processing device 100 as needed. The image preparation device 310 is connected via a network NW, similar to the site 200, so that the image processing device 100 can acquire the image data ORG, but it may also be directly connected to the image processing device 100 by wired or wireless connection.

[0062] In this embodiment, the printing device 320 is connected to the image preparation device 310 via a network NW, and receives instructions from the image preparation device 310 to print the image data ORG output by the image preparation device 310 onto the printing medium PRM. Prior to printing by the printing device 320, the user of the printing system 300 has the image data ORG acquired by the image processing device 100, and as described in the first embodiment, treats the printing medium PRM as a 3D object and performs lighting processing using second data SD including texture parameters to render the printing medium PRM, including the image data printed on it.

[0063] The user can check the rendering result on the image display unit 151 and, if necessary, change the viewpoint, light source position, light source intensity, white balance, etc., to check how the print medium PRM looks. After that, the image data ORG is output from the image preparation device 310 to the printing device 320 via the network NW, and the image data ORG is printed onto the print medium PRM. Prior to printing, the user can check how the image looks on the print medium PRM through physically based rendering by the image processing device 100. As a result, the user can check the differences in texture depending on the type of print medium PRM, including the smoothness (roughness) of the print medium PRM surface, before printing. By looking at the rendering result displayed on the image display unit 151, it is also possible to change the colors of the image data ORG, change the type of print medium PRM used, change the printing device 320 used for printing, or change its ink set, in order to obtain the desired print result.

[0064] When the image processing device 100 is used together with the printing device 320, the printing device 320 may be equipped with a printing condition setting unit 315 in the computer that issues printing instructions, for example, in this embodiment, the image preparation device 310, which sets printing conditions that affect how the image printed on the printing medium appears on the printing medium. This makes it possible to set the profile necessary for color conversion from printing conditions such as the selection of a paper tray containing a predetermined printing medium, the selection of the ink set to be used, and the selection of the type of printing device to be used, and to define first and second data to be referenced based on the printing conditions, thereby easily realizing various settings. In addition to these conditions, the printing condition setting unit 315 may also set the observation state of the printing medium on which the image is printed in the virtual space, lighting information which is information about the lighting of the printing medium in the virtual space, object identification information which identifies 3D objects in the virtual space, and background information which identifies the background in the virtual space.

[0065] The printing medium used by the printing device 320 may be something other than paper. For example, it may be a textile printer that prints on fabric or a printing device that prints on solid objects such as cans or bottles. In addition to a configuration that prints directly onto the target object, it is also possible to use a printing device that prints on a transfer medium such as transfer paper and then transfers the ink formed on the transfer medium to the fabric or solid object that is the printing medium. One example of such a transfer-type printing device is a dye-sublimation printing device. In such a transfer-type configuration, the printing medium is the final printed product to which the transfer was made. In such cases, texture parameters related to the structure and texture of the surface of the printing medium, such as fabric, metal, glass, or plastic, should be prepared according to the properties of the printing medium, and physically based rendering should be performed by the image processing device 100. Even in a transfer-type printing device, the texture parameters used represent the texture of the final printed product, not the transfer medium. An example of the display on the image display unit 151 when printed on such fabric or cans is shown in Figure 16. For ease of understanding, the diagram shows both the object OBJt printed on the T-shirt and the object OBJc printed on the can together, but normally, one print medium at a time is displayed. Of course, multiple rendering units can be prepared and multiple physically based rendering results can be displayed simultaneously.

[0066] C. Other embodiments: (1) The disclosure can also be implemented in the following forms. One other embodiment is in the form of an image processing apparatus. This image processing apparatus includes: an image data acquisition unit that acquires image data of an image to be printed on a printing medium; a parameter acquisition unit that acquires parameters for physically based rendering the printing medium as a 3D object, which include a normal map representing the normal direction of the uneven surface of the printing medium with respect to the reflection direction of light incident on the printing medium; a correction unit that corrects the strength of the application effect of the normal map associated with the 3D object; and a rendering execution unit that generates a rendered image representing the printing medium on which the image has been printed by performing physically based rendering using the image data and the parameters.

[0067] This approach corrects the strength of the effect of the normal map associated with the 3D object before performing physically based rendering to generate the rendered image. As a result, the texture of the printed material can be reproduced without limitations on the viewing environment.

[0068] Such image processing devices may be configured solely for the image processing described above, or they may include a function for saving images intended for printing. Alternatively, they may include a function for creating images intended for printing, or they may be configured as devices for printing images. The image processing device may be implemented using a computer equipped with a GPU, or it may be configured as a distributed system with the necessary functions located at multiple sites and capable of collaboration. When configured as a distributed system, the processing load on terminals is reduced, making it easier to perform the image processing described above on mobile devices such as tablets, further improving user convenience.

[0069] Various existing configurations can be used for such rendering execution units. Generally, rendering may be performed by dividing it into multiple elements, such as viewpoint transformation to convert 3D world coordinates to a coordinate system viewed from the viewpoint, culling to remove vertices unnecessary for rendering from 3D objects, clipping to remove invisible coordinates, and rasterization. These processes may be configured to be suitable for processing on a dedicated GPU and may be implemented by a pipeline configuration that includes a vertex pipeline for processing the vertices of 3D objects and a pixel pipeline for processing each rasterized pixel.

[0070] (2) In such a configuration, the parameters may include the smoothness of the surface of the printing medium with respect to the dispersion of the reflection direction of light incident on the printing medium. This method allows you to impart a texture to a 3D object that corresponds to the smoothness of the surface of the printing medium, thus making the texture of the 3D object represented in the rendered image closer to the texture of the actual printing medium.

[0071] (3) In the configuration of (2) above, the correction unit may correct the strength of the application effect of the normal map according to the degree of smoothness. This approach adjusts the strength of the normal map's effect according to the smoothness of the print medium, making the texture of the 3D object in the rendered image closer to the texture of the actual print medium.

[0072] (4) In the configurations of (2) to (3) above, the correction unit may, if the smoothness is greater than or equal to a predetermined value, correct the normal map so that the effect of applying the normal map becomes stronger as the specular reflection component of the function representing the reflection of light increases. This allows us to make the texture of a 3D object more closely resemble that of a real-world printing medium, such as glossy paper, when reproducing a highly smooth printing medium in a 3D object.

[0073] (5) In the configurations of (2) to (4) above, if the smoothness is less than a predetermined value, the correction unit may correct the normal map so that the smaller the angle between the polygon normal, which is the normal of the polygons constituting the 3D object, and the half vector between the light source direction vector and the viewpoint direction vector, the weaker the effect of applying the normal map. This allows you to make the texture of a 3D object more closely resemble the texture of a real-world printing medium, such as matte paper or plain paper, when reproducing a printing medium with low smoothness.

[0074] (6) In the configurations of (2) to (5) above, if the smoothness is less than a predetermined value, the correction unit may correct the normal map so that the effect of applying the normal map is weakened as the angle between the polygon normal, which is the normal of the polygons constituting the 3D object, and the light source direction vector becomes smaller. This allows you to make the texture of a 3D object more closely resemble the texture of a real-world printing medium, such as matte paper or plain paper, when reproducing a printing medium with low smoothness.

[0075] (7) In the configurations of (2) to (6) above, the correction unit may, if the smoothness is less than a predetermined value, correct the normal map so that the smaller the angle between the polygon normal, which is the normal of the polygons constituting the 3D object, and the half vector between the light source direction vector and the viewpoint direction vector, the weaker the effect of applying the normal map, and the smaller the angle between the polygon normal and the light source direction vector, the weaker the effect of applying the normal map. This allows you to make the texture of a 3D object more closely resemble the texture of a real-world printing medium, such as matte paper or plain paper, when reproducing a printing medium with low smoothness.

[0076] (8) In the configuration of (2) to (7) above, a plurality of correction units having different methods for correcting the strength of the application effect of the normal map may be provided, and a selection unit may be provided for selecting from the plurality of correction units which correction unit performs the correction of the strength of the application effect of the normal map according to the smoothness. This makes it easy to switch the method for correcting the strength of the normal map's application effect depending on the smoothness of the print medium.

[0077] (9) In the configurations of (1) to (8) above, the correction unit may be provided in the shader that adds shading to the 3D object in the physically based rendering. This allows you to adjust the strength of the normal map's effect in the shader that applies shadows to 3D objects. As a result, you can change the shadows applied to 3D objects by adjusting the strength of the normal map's effect whenever the light source, viewpoint, and the position and orientation of the 3D object change.

[0078] (10) Another embodiment of the present disclosure may be configured as a printing system. This printing system comprises an image processing device one of the above (1) to (9), a display unit for displaying the rendered image generated by the image processing device, and a printing device for printing the image data. This allows the printer to display how the image will look on the printed medium before printing, enabling users to check the display before printing. Consequently, discrepancies between the image to be printed and the appearance on the printed medium are minimized, reducing the need for repeated trial and error by adjusting the original image and printing conditions, and potentially saving time and money on test prints.

[0079] (11) Another configuration of the present disclosure is an image processing program. This image processing program causes a computer to perform the following functions: acquire image data of an image to be printed on a print medium; acquire parameters for physically based rendering the print medium as a 3D object, which include a normal map representing the normal direction of the surface of the print medium with respect to the reflection direction of light incident on the print medium; correct the strength of the application effect of the normal map associated with the 3D object; and generate a rendered image representing the print medium on which the image has been printed by performing the physically based rendering using the image data and the parameters. In this way, the computer-equipped device can be easily given the image processing functions described in (1) to (9).

[0080] (12) In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software. At least some of the configurations implemented by software can also be implemented by discrete circuit configurations. Furthermore, if some or all of the functions of this disclosure are implemented by software, the software (computer program) may be provided in the form of being stored on a computer-readable recording medium. "Computer-readable recording medium" is not limited to portable recording media such as flexible disks and CD-ROMs, but also includes various internal storage devices in a computer such as RAM and ROM, and external storage devices fixed to a computer such as hard disks. In other words, "computer-readable recording medium" has a broad meaning that includes any recording medium on which data packets can be fixed rather than temporary.

[0081] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0082] 100…Image processing device, 111…Color management system, 121…Rendering execution unit, 131…First storage unit, 132…Second storage unit, 135…Memory, 141…Communication unit, 151…Image display unit, 200…Site, 300…Printing system, 310…Image preparation device, 315…Printing condition setting unit, 320…Printing device

Claims

1. An image data acquisition unit that acquires image data of an image to be printed on a print medium, A parameter acquisition unit acquires parameters for physically based rendering the print medium as a 3D object, which include a normal map representing the normal direction of the uneven surface of the print medium with respect to the reflection direction of light incident on the print medium. A correction unit for correcting the strength of the application effect of the normal map associated with the 3D object, A rendering execution unit that generates a rendering image representing the print medium on which the image has been printed by performing the physically based rendering using the image data and the parameters, An image processing device equipped with the following features.

2. The parameters include the smoothness of the surface of the printing medium with respect to the dispersion of the reflection direction of light incident on the printing medium. The image processing apparatus according to claim 1.

3. The correction unit corrects the strength of the application effect of the normal map according to the degree of smoothness. The image processing apparatus according to claim 2.

4. The correction unit, when the smoothness is greater than or equal to a predetermined value, corrects the normal map so that the effect of applying the normal map becomes stronger as the specular reflection component of the function representing light reflection increases. The image processing apparatus according to claim 2.

5. The correction unit, when the smoothness is less than a predetermined value, corrects the normal map's application effect as the angle between the polygon normal (which is the normal of the polygons constituting the 3D object) and the half-vector (the ratio of the light source direction vector to the viewpoint direction vector) decreases. The image processing apparatus according to claim 2.

6. If the smoothness is less than a predetermined value, the correction unit corrects the angle between the polygon normal (which is the normal of the polygon constituting the 3D object) and the light source direction vector so that the effect of applying the normal map weakens as the angle becomes smaller. The image processing apparatus according to claim 2.

7. The correction unit, when the smoothness is less than a predetermined value, corrects the normal map so that the effect of applying the normal map weakens as the angle between the polygon normal (which is the normal of the polygons constituting the 3D object) and the half-vector (the ratio of the light source direction vector to the viewpoint direction vector) decreases, and the effect of applying the normal map weakens as the angle between the polygon normal and the light source direction vector decreases. The image processing apparatus according to claim 2.

8. The correction unit comprises a plurality of correction units, each having a different method for correcting the strength of the application effect of the normal map. The system further includes a selection unit that selects from among a plurality of correction units which correction unit performs correction on the strength of the application effect of the normal map, according to the degree of smoothness. The image processing apparatus according to claim 2.

9. The correction unit is provided in the shader that applies shading to the 3D object in the physically based rendering. The image processing apparatus according to claim 1.

10. An image processing apparatus according to any one of claims 1 to 9, A display unit that displays the rendered image generated by the image processing device, A printing device for printing the aforementioned image data, A printing system equipped with the following features.

11. A function to acquire image data of images to be printed on print media, A function to obtain parameters for physically based rendering the aforementioned printing medium as a 3D object, which include a normal map representing the normal direction of the uneven surface of the printing medium with respect to the reflection direction of light incident on the printing medium, A function to correct the strength of the application effect of the normal map associated with the 3D object, A function to generate a rendered image representing the printed medium on which the image has been printed by performing the physically based rendering using the image data and the parameters, An image processing program that causes a computer to perform an image processing operation.

Images