Image processing device, image printing system, and image processing program

Abstract

When previewing the state of print media, the appropriate method for representing textures and relief is selected while considering the load on the system. [Solution] Image data of an image to be printed on a print medium is input, and printing conditions, which are the conditions for printing on the print medium, are obtained. From the obtained printing conditions, a method for representing relief when processing the image data is selected, and the printed material is previewed using the selected relief representation method.

Description

Technical Field

[0001] The present disclosure relates to a technique for appropriately performing concavo-convex representation on the surface of a printed medium in a preview of the printed medium.

Background Art

[0002] When previewing the appearance of a three-dimensional object, in Patent Document 1 below, according to the height of the viewpoint, normal representation, shadow representation, and parallax representation are added to perform rendering processing. If the object is viewed from the front, the viewpoint becomes close to perpendicular to the image plane. In this case, since the concavo-convex on the surface of the object is difficult to recognize, the processing is limited to the degree of normal representation processing, and the processing of the added representation is not performed. On the other hand, when the object is viewed from an oblique direction or the direction of the line of sight approaches parallel to the image plane, the processing of normal representation by a bump map, shadow representation by a horizon map, and parallax representation by a height map is sequentially added for rendering. Thus, in Patent Document 1, it is said that by adding the representation processing according to the position of the viewpoint, a high-definition and realistic representation can be efficiently performed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, Patent Document 1 merely determines the content of the processing for expressing unevenness based on the direction and height of the viewpoint relative to the object, that is, whether it is viewed from directly in front or from an angle, and there may be cases where the expression of unevenness is not sufficiently achieved. For example, even when viewed from the front, unevenness can be recognized if light shines on it from an angle, but this case cannot be adequately handled. Furthermore, when previewing the unevenness of the surface of a printing medium in which an image is formed by an image-forming material such as ink, there has been no prior consideration of what kind of unevenness expression should be selected and used. [Means for solving the problem]

[0005] This disclosure can be implemented in the following forms or applications:

[0006] (1) The present disclosure can be implemented as an image display device. This image display device includes an input unit for inputting image data of an image to be printed on a printing medium, an acquisition unit for acquiring printing conditions for the printing medium, a selection unit for selecting a method for expressing relief when processing the image data from the acquired printing conditions, and a display execution unit for displaying a preview of the printed material using the selected method for expressing relief.

[0007] (2) The present disclosure can also be implemented as an image printing system. This image printing system comprises an input unit for inputting image data of an image formed on a printing medium using an image forming material; an acquisition unit for acquiring printing conditions which are the conditions for printing on the printing medium; a selection unit for selecting a method for expressing relief when processing the image data from the acquired printing conditions; a display execution unit for displaying a preview of the printed material using the selected method for expressing relief; and a printing device for printing the previewed image on the printing medium.

[0008] (3) The disclosure can also be implemented as an image processing program that generates a rendering image of a printed medium on which an image is printed. This image processing program uses a computer to implement the following functions: inputting image data of an image formed on the printed medium using an image forming material; acquiring printing conditions which are the conditions for printing on the printed medium; selecting a method for expressing relief when processing the image data from the acquired printing conditions; and displaying a preview of the printed material using the selected method for expressing relief. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic diagram showing an image processing apparatus of an embodiment. [Figure 2] A schematic diagram showing the processing overview, focusing on the configuration of the selection unit. [Figure 3A] This diagram illustrates the contents of the print condition database used to select the appropriate processing method for representing relief and textures. [Figure 3B] An explanatory diagram to illustrate the different methods used to represent uneven surfaces. [Figure 4] An explanatory diagram showing an overview of the tessellation process in the embodiment. [Figure 5] An explanatory diagram illustrating the overview of smoothness map generation in the embodiment. [Figure 6] A flowchart illustrating an example of an image processing routine. [Figure 7] A schematic diagram of the rendering execution unit. [Figure 8] An explanatory diagram showing an example of a preview screen. [Figure 9] An explanatory diagram showing the relationship between the light source, viewpoint, and the angles of the surfaces of a 3D object. [Figure 10] An explanatory diagram illustrating how ink appears when printed on a print medium. [Figure 11] An explanatory diagram illustrating how the appearance of a displacement model differs depending on the number of vertices. [Figure 12] An explanatory diagram showing a triangular polygon and a method for increasing its number of vertices. [Figure 13]An explanatory diagram that illustrates and contrasts the appearances due to differences in concavo-convex expressions, taking the protrusions formed with clear ink as an example. [Figure 14] An explanatory diagram showing another configuration example of a rendering unit using displacement processing including polygon division processing. [Figure 15] A schematic configuration diagram of an image printing system.

Mode for Carrying Out the Invention

[0010] A. First Embodiment: (A1) Overall Configuration of Image Processing Apparatus 100: The overall configuration of the image processing apparatus 100 according to the first embodiment is shown in FIG. 1. This image processing apparatus 100 performs image processing for previewing how an image is printed on a predetermined printing medium. As shown in the figure, this image processing apparatus 100 includes, inside, a color management system (hereinafter simply referred to as CMS) 20, an acquisition unit 60 that acquires printing conditions PJ from the outside, a concavo-convex expression processing unit 111 that selects a method of concavo-convex expression for processing image data and generates necessary maps and the like from the acquired printing conditions, a rendering execution unit 121, a memory 135, a communication unit 141, and a rendering image display unit 151. The CMS 20 color-converts the original image to be printed previewed into the object color expressed on the printing medium. The converted image data is called managed image data MGP. Details of the CMS processing will be described later. The concavo-convex expression processing unit 111 includes a printing condition database 50 that stores the relationship between various printing conditions and a plurality of concavo-convex expressions, and a selection unit 40 that selects a method of concavo-convex expression. Further, the printing conditions PJ input by the acquisition unit 60 from the outside include the type PC of the printing medium, the printing method PM, and the like. The printing method PM includes information related to the thickness of the ink layer, such as the type of ink and the ink amount, which are one of the image forming materials. Details of the printing conditions PJ will be described later.

[0011] The image processing device 100 receives image data (ORG) from an external source, converts it to managed image data (MGP) using the CMS 20, and outputs it to the rendering execution unit 121. The image processing device 100 not only performs image processing but also performs rendering using the rendering execution unit 121 and displays the processing result as a preview image on the rendering image display unit 151. The rendering execution unit 121 and the rendering image display unit 151 correspond to the "display execution unit that displays a preview of the printed material using the selected method of expressing relief." The programs that perform each of the processes described later are stored in the memory 135 of the image processing device 100, and the functions of the image processing device 100 are realized when the CPU or GPU executes the programs stored in memory.

[0012] Memory 135 stores these programs, as well as various maps such as the first data FD, the second data SD, and a normal map used for the relief representation described later. The first and second data FD and SD are parameters necessary for physically rendering and displaying the printing medium on which the image is printed as a 3D object. In particular, the first data FD is data related to the form of the printing medium under a light source in the virtual space, and includes 3D object information of the printing medium, camera information CMR such as the position from which the printing medium is viewed, lighting information LGT such as the position and color of the lighting, and background information BGD indicating information about the background on which the printing medium is placed. The second data SD is data related to image formation on the surface of the printing medium, and includes, for example, texture data representing the texture of the surface of the printing medium. These first data FD and second data SD are used during rendering in the rendering execution unit 121.

[0013] For the first data FD and the second data SD, for typical data with a usage frequency of a predetermined level or higher, they may be non-volatilely stored in the memory 135 in advance, selected as needed, and made available for reference by the rendering execution unit 121. As for printing media such as those not commonly used, for example, those with a low usage frequency, such as texture data when using special materials like cloth fabric, cans, or plastic sheets, they may be stored in an external site 190 and obtained via the communication unit 141 when necessary. The first data FD such as lighting information may be individually specified by the user during rendering, but for typical camera angles and light sources, they may be stored in the memory 135 in advance and used as well. The camera angle refers to the position and direction of looking at the target printing media, corresponding to the position of the virtual viewpoint and the direction of the line of sight of the user looking at the virtual space. Therefore, the camera may be described as the viewpoint or view in terms of the direction of the viewpoint and the line of sight.

[0014] The image display unit 151 displays the image of the printing media rendered by the rendering execution unit 121 together with the ink on the surface, the background, etc. The image display unit 151 reads the image data for display from the frame memory FM provided in the rendering execution unit 121 and performs the display. The image display unit 151 may be provided in the image processing apparatus 100, or may be provided separately from the image processing apparatus 100. Note that the image processing apparatus 100 may be realized as a dedicated machine, or may be realized by executing an application program on a computer. Of course, the computer includes terminals such as tablets and mobile phones. Since the processing of the rendering execution unit 121 requires a considerable amount of resources and computing power, only the rendering execution unit 121 may be executed by a CPU capable of high-speed processing or a dedicated GPU, and the rendering execution unit 121 may be configured at another site on the network to form the image processing apparatus 100.

[0015] The various maps stored in memory 135 set the appearance of the printing medium and the ink printed on its surface when a method for representing the unevenness of the printing medium is selected according to the printing conditions PJ. These maps include normal maps, height maps, displacement maps, smoothness maps, etc., as will be described later. Maps can be generated once the printing conditions PJ and image data ORG are defined, and are stored in memory 135 so that they can be referenced at any time from the rendering execution unit 121. The generation and usage of each map will be described later.

[0016] (A2) Configuration and function of the relief expression processing unit 111: The relief processing unit 111 processes the relief representation of an image on a print medium. The detailed configuration of this relief processing unit 111 is shown in Figure 2. Before explaining the functions of the print condition database 50 and the selection unit 40 that constitute the relief processing unit 111, we will briefly describe some typical relief representation processes used in rendering.

[0017] [1] Bump Mapping: A general term for techniques used to add bump and relief details to 3D models. Alternatively, it refers to techniques for creating pseudo-shading of bumps and reliefs. In practice, Normal Mapping, which is explained below, is often used. [2] Normal Mapping: Generates an RGB image (normal map) corresponding to the XYZ values ​​of the normal vector of the 3D model surface. The processing load in rendering is relatively small. Unlike bump mapping, it uses vector data, allowing for detailed calculation of light intensity in a particular direction. This can be calculated from the height map described later. [3] Parallax Mapping. This technique creates the illusion of depth by shifting the coordinates from which textures are obtained according to the height difference between convex and concave areas. The rendering load is moderate. A height map is used to determine the parallax. Even when applying Normal Mapping or Parallax Mapping to an object, such as a sphere, its outline remains that of the original sphere. For this reason, these processes, which do not alter the shape, are sometimes called texture mapping. [4] Displacement Mapping: Shifts the vertex coordinates of polygons according to the height of the bumps and irregularities. This has a heavy rendering load. Unlike any of the texture mappings mentioned above, it deforms the polygons, so the outline of the model changes. A Height Map is used to perform this.

[0018] The above are representative methods for representing surface texture. Generally, the texture representation becomes more precise and closer to reality as the methods are listed above, but the processing load increases. A few additional notes regarding the terminology used in the above explanation, as well as other maps and numerical values ​​used in surface texture representation, are provided below. [5] Height Map: An image that represents the height of an object. Generally, it is used as a grayscale image where white indicates a higher position. It can be used in the Parallax Mapping and Displacement Mapping described above. [6] Height Scale: A numerical value that represents the degree of height. A Height Map is a grayscale image represented by values ​​between 0 and 255 if it's 8-bit, and it doesn't store absolute height values. The Height Scale can be used to adjust the degree to which Parallax and Displacement Mapping are applied. [7] Smoothness Map: A numerical value representing the smoothness of the surface. The texture is expressed on a scale from 0.0 (rough, coarse) to 1.0 (smooth, glossy). Printed areas on glossy paper, for example, may have a value close to 1.0. On the other hand, without clear ink, such as printing on matte paper with pigment-based ink, may have a value close to 0.0.

[0019] The configuration and operation of the relief expression processing unit 111 will be explained using Figure 2. First, the acquisition unit 60 acquires the printing conditions PJ from an external source. The printing conditions PJ includes the printing medium type PC and the printing method PM. The printing medium type PC is information about the type of printing medium, such as plain paper, photographic paper, fine art paper, thin cloth (e.g., silk), or cloth (e.g., cotton). The printing method PM relates to the printing method and includes information related to the height of the relief in the ink layer, such as whether the ink type, which is one of the image forming materials, is dye, pigment, or UV-curing ink, and information about the number of times the ink is ejected. Other image forming materials include DTFilm and 3D printer filament. This information about image forming materials corresponds to the conditions of the image forming material. The conditions of the image forming material are classified according to the type of image forming material and the amount of image forming material. These correspond to the height of the relief on the printing medium where the image is formed. The classification of the conditions of the image forming material will be explained further later with reference to Figure 3A.

[0020] The user of the image processing device 100 inputs these print conditions PJ. For example, the input can be done by displaying a dialog box that allows the user to select from multiple types of print media as the print media type PC, or a dialog box that allows the user to select from multiple types of ink and print cycles as the print method PM, allowing the user to select the print media type PC or print method PM. Alternatively, the print method PM can be set all at once by specifying the model number of the printer.

[0021] Once these print condition PJs are obtained, the print condition database 50 is then referenced based on these print condition PJs. An example of the print condition database 50 is shown in table format in Figure 3A. Based on this print condition database 50, the selection unit 40 selects the method for expressing relief to be performed in the rendering execution unit 121 from among normal processing, parallax processing, and displacement processing. For example, if the print medium type PC is plain paper and the print method PM is one ejection of dye or pigment ink, normal processing is selected. Alternatively, if the print medium type PC is fine art paper and the print method PM is one ejection of dye or pigment ink, both normal processing and parallax processing are selected.

[0022] In the printing conditions database 50 shown in Figure 3A, the method of expressing relief is selected based on the type of printing medium and printing method, such as the type of ink and the number of prints. The reason for selecting the method of expressing relief under specific printing conditions is that, for printing media, the depth of the surface relief to be expressed (the height difference between the top surface of the protrusions and the bottom surface of the recesses) is largely determined by the type of medium, or the height of the ink layer that rises on the surface of the printing medium is largely determined by the classification of the conditions of the image-forming material, such as the type of ink and the amount of ink used. The amount of ink is determined by the number of prints and the ink duty cycle in a single print. In the example shown in Figure 3A, for example, in the case of UV ink, the number of ink ejections is classified as 1, 2-3, and 4 or more. These classifications correspond to the actual height of the relief on the printing medium where the image is formed. Therefore, by selecting the method of expressing relief from the printing medium type PC and printing method PM, it is possible to select the appropriate relief expression during preview.

[0023] If the depth of the surface irregularities on the printing medium or the thickness of the ink layer is less than the first threshold, normal processing is selected. This thickness comparison can be done using actual measurements, but it can also be defined using a height scale, for example, normal processing is performed if it is less than 0.2, and normal processing and parallax processing are performed if it is greater than or equal to 0.2. In terms of actual dimensions, for an ink layer, if the thickness is about 0.1 to 0.2 mm, it is desirable to perform parallax processing along with normal processing. Of course, the thickness of the ink layer is affected not only by the amount of ink per unit area, but also, in the case of UV ink, by the curing method and the wettability of the ink. Therefore, when selecting a method for expressing surface irregularities based on printing conditions PJ, it is desirable to measure the actual thickness of the ink layer and calibrate the correspondence between the printing conditions PJ and the selected surface irregularity expression. Furthermore, there is no limit to one threshold for such judgments; multiple thresholds may be set. For example, if the depth of the surface irregularities on the printing medium or the thickness of the ink layer is greater than the first threshold, i.e., a second threshold (1.0 or greater on the height scale), displacement processing may be selected.

[0024] In addition to these selection methods, the method of creating raised relief can also be switched depending on, for example, the number of times UV ink is printed. UV ink can be built up thicker by repeatedly printing in the same place. Therefore, the method of creating raised relief can be selected according to the number of prints as follows. <1> If the number of prints is only once, select normal processing. <2> If the number of prints is 2-3, select normal vector processing and parallax processing. <3> If the number of prints is four or more, select displacement processing. Furthermore, conditions such as ink volume and print resolution may be added in addition to the number of prints. For example, if the judgments in (1) to (3) above were created under the condition that the ink duty cycle for a single print is a maximum of 200%, then if the ink duty cycle is 400%, parallax processing may be applied even if the number of prints is only one.

[0025] In Figure 3A, for DTFilm, displacement processing is selected regardless of the number of prints. This is because DTFilm uses adhesive to bond the printed layer, creating a step between the printing medium and the printed layer, which results in a stepped appearance in the image's contours. If powder is used, the thickness increases accordingly, resulting in a stepped appearance.

[0026] If normal processing or parallax processing is selected, the height map generation unit CHM generates a height map HM using the second data, such as texture parameters, according to the white ink image Wid or clear ink image Cid included in the image data ORG. Here, the white ink image Wid or clear ink image Cid are given as examples of images included in the image data ORG, but the ink color is arbitrary. The clear ink image Cid is used when it is printed over other inks to enhance the smoothness of the surface, or when physical protrusions are created on the printing medium, such as in Braille. In the figure, each generated map is enclosed in a double line.

[0027] Using the generated height map HM, the normal map generation unit CNM generates a normal map NM. The calculation formula for obtaining the normal map NM from the height map HM is well known. If normal processing is selected in the selection unit 40, this normal map NM is output to the rendering execution unit 121. The rendering execution unit 121 performs normal processing PON using the normal map NM and renders the appearance of the surface of a printed medium with an image printed with a specific ink for preview. On the other hand, if normal processing and parallax processing are selected by the selection unit 40, this normal map NM and the previously obtained height map HM are output to the rendering execution unit 121. The rendering execution unit 121 performs parallax processing POP using the normal map NM and height map HM and renders the appearance of the surface of a printed medium with an image printed with a specific ink for preview. The details of the rendering execution unit 121 will be explained in detail later.

[0028] Based on the printing conditions PJ, when displacement processing is selected in the selection unit 40, the displacement map generation unit CDM generates a displacement map DM using second data, such as texture parameters, according to the white ink image Wid or clear ink image Cid included in the image data ORG. Unlike the normal map NM and height map HM described above, the displacement map DM is a map that represents the actual three-dimensional shape itself.

[0029] In each of the processes described above, the print medium or the ink that forms the image being previewed is treated as a collection of polygons. However, the displacement map DM differs from the normal processing PON and parallax processing POP in that it achieves a three-dimensional representation by displacing the vertices of the polygons that make up the surface of the object. While the normal processing PON and parallax processing POP achieve a three-dimensional representation in the preview, they do not change the shape of the polygons themselves to create a three-dimensional object. This point will be explained using Figure 3B. The top row of the figure shows an example of the relief representation when previewed. In this example, the object OJT placed on the print medium MT has a hemispherical shape like an inverted bowl with a depressed center. Let's assume that this object OJT is formed of clear ink, for example. When viewing this object OJT from directly above, the appearance differs only in the degree of relief representation whether using normal processing with a normal map NM, parallax processing with a normal map NM and a height map HM, or displacement processing with a displacement map DM.

[0030] On the other hand, when viewed from a diagonal direction in front of the object OJT, the displacement processing using the displacement map DM shown in the middle of the figure actually displaces the contour of the object OJT, resulting in a three-dimensional representation of the object OJT's contours, and creating parts that are hidden by the object OJT from the viewpoint. Because the light source is in the foreground, the shadows cast by the object are not shown in the illustration, but the shadows cast by the object are displayed in the displacement processing. In contrast, with normal processing using a normal map NM, for example, although shading is applied, the shape does not change, so it appears as a simple shaded plane, as shown in the bottom of the figure.

[0031] In displacement processing, which uses a displacement map (DM) to represent surface irregularities, the vertices of polygons are displaced in response to the irregularities caused by the printing medium and ink. This process of displacing polygon vertices can sometimes degrade the reproducibility of the original surface irregularities. This is because, if the polygons are relatively large, it may not be possible to match the position where the thickness of the ink layer actually changes with the position of the movable vertices of the polygons. For this reason, a process is sometimes performed to ensure shape reproducibility by dividing the polygons to be displaced, reducing the size of each polygon and increasing the number of movable vertices. This process is called tessellation processing (POT). As outlined in Figure 2, the degree of polygon division is determined from the resolution of the displacement map (DM) and the information about the polygons contained in the first data used for the original image representation, and this is output to the displacement processing (POD) along with the displacement map (DM). In this way, the reproducibility of the original surface irregularities can be maintained even by the displacement processing (POD).

[0032] The details of the tessellation processing POT are shown in Figure 4. As shown in the figure, the tessellation processing POT divides the polygon according to the resolution of the displacement map DM based on the basic polygon data contained in the first data FD, calculates the vertex data of the divided polygons, and stores it as divided polygon DDP. The vertex information of this divided polygon DDP and the displacement map DM are used in the displacement processing POD in the rendering execution unit 121, realizing the movement of the vertices of the divided polygons in the vertex shader VS, and realizing the unevenness representation of the printed medium and printed ink. The number of polygon divisions DDP should be determined by balancing the reproducibility of the original shape and the processing load in the vertex shader VS. For example, the basic number of divisions may be set to SDD (a value of 20 as an example), and it may be determined by the following formula (1) from the resolution Gdm of the displacement map DM and the display resolution Gds of the image display unit 151. DDP = SDD × Gdm / Gds …(1) If the resolution Gdm of the displacement map DM is 360 dpi, then the resolution of the display unit such as the image display unit 151 is generally around 72 dpi, DDP = 20 × 360 / 72 =100 This is how it works. Of course, the user may adjust the basic division number SDD by looking at the preview image displayed on the image display unit 151 to set the desired division number DDP. Alternatively, the number of divisions may be determined according to the distance between the viewpoint and the object being previewed, as will be described later. The smaller the distance to the viewpoint, the larger the number of divisions, as this allows for a more detailed preview of the shape of the printed medium and the printed ink. Therefore, it is desirable to increase the number of divisions to improve the reproducibility of the shape.

[0033] From the perspective of shape reproducibility, the smoothness of the object's surface can also be important. As previously explained, a smoothness map is used to reproduce surface smoothness. The handling of this smoothness map is shown in Figure 5. The smoothness map SM is generated by the smoothness map generation unit CSM included in the unevenness expression processing unit 111. The smoothness map generation unit CSM receives mainly clear ink image data Cid from the image data ORG, and uses this with second data SD, which includes printing conditions PJ and texture parameters, to generate the smoothness map SM. This smoothness map SM is used in the smoothness processing POS in the rendering execution unit 121.

[0034] The normal processing PON, parallax processing POP, and displacement processing POD described above are shown side by side in the rendering execution unit 121 in the figure, but since the selection unit 40 makes the selection, in reality only the processing selected based on the printing conditions PJ is executed in the rendering execution unit 121, and the relief is expressed by the printing medium MT and the printed ink. In Figure 2, the selection unit 40 selects the method for expressing relief based on the printing conditions PJ and generates the necessary maps according to that selection, but it is also possible to generate all the maps to be used in the rendering execution unit 121 after acquiring the printing conditions PJ etc. by the acquisition unit 60, and the rendering execution unit 121 uses the necessary maps (normal map NM, height map HM, displacement map DM, etc.) according to the relief expression method (normal processing, parallax processing, displacement processing, etc.) selected by the selection unit 40.

[0035] (A3) Overview of processing in the image processing device 100: In addition to the functions of the relief expression processing unit 111 described above, the image processing device 100 executes the image processing routine shown in Figure 6 for preview purposes. When this process starts, the image processing device 100 first performs the print condition acquisition process (step S71). This process is performed by the acquisition unit 60, which has already been described. Here, the image data ORG, which will be printed, is also acquired in conjunction with the acquisition of the print conditions. The image processing device 100 displays the necessary dialog boxes, etc., via the acquisition unit 60, and inputs the print conditions PJ and the image data ORG.

[0036] Next, the image processing device 100 performs a selection process to select a method for representing relief (step S81). This process is performed by the selection unit 40, which has already been described. As a result of the processing by the selection unit 40, in this embodiment, the method for representing relief on the printing medium and printed ink is selected from normal processing, parallax processing, and displacement processing. Note that the selection of relief representation methods is not limited to these three. If the image processing device 100 can only execute two of these three methods, it may select from those two, or it may select from four or more methods, including other methods. Based on the above processing (steps S71, S81), it is determined whether the preview conditions have been finalized (step S91), and the above processing is repeated until the conditions are finalized. If the conditions are finalized, the process proceeds to the color conversion processing routine in step S100.

[0037] The color conversion process (step S100) is performed by the CMS20 of the image processing device 100. This process converts the original image data ORG into color data in a common color space for rendering by the CMS20. Step S100 includes the processes described below (steps S110 to S160). When the color conversion process starts, the original image data ORG and the input profile IP are first input, and the original image data ORG, which is represented in a device-dependent color system (e.g., RGB color system), is converted into color data in a device-independent color system (e.g., Lab or XYZ color system) (step S110). Next, it is determined whether a media profile MP is available (step S120), and if a media profile MP is available, it is applied, and the color conversion is performed to the range of colors that can be expressed by printing, taking into account the combination of printing device (printer) and printing medium as printing conditions (step S130). If there is no media profile MP, the process in step S130 is not performed. Subsequently, the color values ​​are converted to the color space of the common color space, which is the second color space used during rendering, using the common color space profile CP (step S150). In this embodiment, sRGB is used as the common color space. The managed image data MGP obtained in this way is set to the albedo color, which is the texture of the 3D object (step S160), and the color conversion process (step S100) is terminated.

[0038] The role and function of the profiles used in each of the processes described above will be explained in more detail. The input profile IP converts the color system of the input side, which is dependent on the device, such as RGB data, to the L color system, which is independent of the device. * a * b *It is used to perform conversions to formats such as (hereinafter simply abbreviated as Lab). A media profile MP is a profile that represents the color reproduction when printed on a specific printing device, such as a printer, on a specific printing medium, under specific printing conditions such as printing resolution. It is a profile that converts color values ​​between a device-independent color system and a device-dependent color system. A media profile MP also includes information other than the printing medium, such as the printing settings of the printing device. Therefore, if we try to cover all combinations of printing device (printer) × printing medium × printing settings, the number of media profile MPs will increase. So, when the dependency on printing conditions is small, or when we do not want to increase the number of profiles, the media profile MP is configured as a combination of printing device (printer) × printing medium. Furthermore, multiple printing media with similar color characteristics may be grouped together and the same media profile MP may be applied to them. In this case, it is desirable to group printing media that not only have similar color characteristics but also the same or similar duty cycle limits according to the ink receiving capacity of the printing medium, and use a common media profile MP. Thus, since the color of an image on a print medium is influenced by both the characteristics of the printing device and the characteristics of the printing medium itself, the media profile MP is sometimes referred to as the print profile MP below.

[0039] Applying the input profile IP to the image data ORG, and then applying the print profile MP, yields color values ​​that would appear under specific printing conditions, i.e., color values ​​dependent on the printing device and medium. Applying the print profile MP to these image color values ​​to convert them from a device-dependent color system to a device-independent color system, and then applying the common color space profile CP, converts them to a representation in the second color space used during rendering (in this case, the sRGB color space). Because the image data ORG has been converted once using the print profile MP to color values ​​dependent on the characteristics of the printing device and medium, the image data ORG is color-converted to a range of actually printable color values. The common color space profile CP is used to convert the image data to color values ​​in the color space used during rendering. The sRGB color space is a typical common color space, but AdobeRGB, Display-P3, etc., may also be used.

[0040] As described above, CMS20 uses each profile to convert image data ORG, which is expressed in the first color space, a device-dependent color system, into image data (managed image data) MGP, which is expressed in the sRGB color space, the second color space used during rendering. Here, the converted image data is not limited to the color values ​​of the sRGB color space, but can be expressed in any color space that the rendering execution unit 121 can handle. For example, if the rendering execution unit 121 employs a configuration that allows rendering using color values ​​of Lab or XYZ color spaces, the image data can be converted to the color values ​​used when displaying it on the image display unit 151 during the lighting processing (described later) performed within the rendering execution unit 121, or in the post-processing unit (described later) located after the rendering execution unit 121.

[0041] In step S130, setting the rendering intent for color conversion in the media profile to absolute allows the color of the print medium itself (background color) to be reflected. Note that if the color value of the image targeted for color conversion in step S150 is outside the sRGB color space gamut, it may be approximated to a value within the sRGB color space, or it may be treated in a way that allows it to take values ​​outside the sRGB color space. Generally, RGB values ​​of image data are stored as 8 bits per color, i.e., integers between 0 and 255, but this is not limited to this; any method that allows specifying a wider color range is acceptable. For example, by adopting signed integers or increasing the number of bits representing RGB values, negative values ​​and values ​​exceeding 255, i.e., values ​​outside the sRGB color space, can be easily handled. Alternatively, pixel values ​​can be represented as floating-point numbers between 0.0 and 1.0, and values ​​outside the sRGB color space can be handled as negative values ​​or values ​​exceeding 1.0.

[0042] Color conversion by CMS20 is not limited to the illustrated configuration; other methods are also possible. For example, color conversion using display device correction data DPD may be performed after color conversion using media profile MP (step S130). Alternatively, composite correction data SPD may be prepared by pre-combining such display device correction data DPD and common color space profile CP, and color conversion using composite correction data SPD may be performed instead of color conversion using common color space profile CP (step S150). Alternatively, a device link profile may be prepared by combining input profile IP and media profile MP, and the device link profile may be applied to perform the conversion all at once instead of applying input profile IP and media profile MP individually. Note that correction for the shift in display color of the image display unit 151 may be performed by the post-processing unit PST after the render backend, as shown in Figure 7, which will be described later, instead of by CMS20.

[0043] (A4) Rendering process: After the color conversion process described above (step S100), a rendering process (step S170) is performed, and the result of the process is displayed as a preview image on the image display unit 151. The rendering process is performed by the rendering execution unit 121. The rendering execution unit 121 places ink based on the managed image data MGP output by the CMS20 after color conversion onto each printing medium MT and renders it, thereby displaying on the image display unit 151 how the printing medium MT on which the original image data ORG is printed and the ink printed on its surface will appear in virtual space. An example configuration of the rendering execution unit 121 is shown in Figure 7. This rendering execution unit 121 shows a typical configuration for performing physically based rendering, and other configurations can also be adopted. The rendering execution unit 121 of this embodiment employs a pipeline configuration including a vertex pipeline VPL and a pixel pipeline PPL, and performs physically based rendering at high speed. The vertex pipeline VPL includes a vertex shader VS and a geometry shader GS. A configuration without using the geometry shader GS is also possible.

[0044] The vertex shader VS converts the coordinates of the vertices of the 3D object, the print medium MT, from the print medium's coordinates to the 3D space being rendered. This coordinate transformation comprehensively includes conversions from the model's (in this case, the print medium's) coordinates to world coordinates, then to view (camera) coordinates, and finally to clip coordinates. However, conversions to view coordinates are handled by the geometry shader GS. The vertex shader VS also performs shading and calculates texture coordinates (UV). Furthermore, as explained using Figure 4, the vertex shader VS references the displacement map DM and moves the vertices of the subdivided polygons created by the tessellation process POT. During these processes, the vertex shader VS and geometry shader GS refer to print medium data TOI, camera information CMR, lighting information LGT, and background information BGD, among others.

[0045] The print media data TOI is information regarding the shape of the print media as a 3D object. The print media and printed ink used in this embodiment have a three-dimensional shape, and their surfaces are not flat. Therefore, the print media MT and ink are treated as having a three-dimensional shape with minute irregularities on their surface. Basically, they are treated as a collection of minute polygons. If the surface of the print media is represented by minute polygons, the number of polygons becomes enormous. For this reason, to represent the surface irregularities, it is also chosen to handle the surface of the print media using textures such as the normal map NM and height map HM, which have already been described. Textures such as the normal map NM and height map HM are provided as texture parameters. Camera information CMR is virtual information about the position and direction in which the camera is placed relative to the print media. Lighting information LGT includes at least one piece of virtual information such as the position, angle, intensity, and color temperature of the light source in the virtual space where the print media is placed. Note that multiple light sources can be set, in which case the effects of the multiple light sources can be calculated separately and superimposed on the 3D object.

[0046] Background information (BGD) is optional, but it contains information about the background on which the print medium, as a 3D object, is placed in the virtual space. Background information (BGD) includes information about objects such as walls and tables placed in the virtual space, and these objects are rendered in the rendering execution unit 121 just like the print medium. In addition, since lighting hits these background objects and illuminates the print medium, it is also treated as part of the lighting information. By performing rendering using this various information, a three-dimensional preview becomes possible. The vertex information calculated by the vertex shader VS is passed to the geometry shader GS.

[0047] The geometry shader GS is used to manipulate the set of vertices within an object. The geometry shader GS allows for increasing or decreasing the number of vertices at runtime, and changing the types of primitives that make up a 3D object. One example of increasing or decreasing the number of vertices is culling, which excludes vertices that are not visible to the camera based on the camera's position and orientation. The geometry shader GS also generates new primitives from existing primitives such as points, lines, and triangles. The geometry shader GS receives primitives from the vertex shader VS, either the entire primitive or a primitive containing information about adjacent primitives. The geometry shader GS processes the input primitives and outputs a rasterized primitive.

[0048] The output of the vertex pipeline (VPL), specifically the primitives processed by the geometry shader (GS), is rasterized by the rasterizer (RRZ) to obtain pixel-level data, which is then passed to the pixel pipeline (PPL). In this embodiment, the pixel pipeline (PPL) comprises a pixel shader (PS) and a render backend (RBE).

[0049] The pixel shader PS manipulates rasterized pixels, and simply put, calculates the color of each pixel. Based on information input from the vertex shader VS and geometry shader GS, it performs processes such as compositing textures and applying surface colors. The pixel shader PS maps managed image data MGP, which is obtained by converting image data ORG using CMS20 based on various profiles, onto the print medium as a 3D object. At this time, the lighting processing function provided in the pixel shader PS performs lighting processing based on the object's light reflection model, the illumination information LGT mentioned above, and the texture parameter TXT, which is one of the second data SD stored in the second storage unit 132, and maps the managed image data MGP. The texture parameter TXT is also specified for the surface of the print medium MT, but it is important in the areas where the image ORG is printed on the print medium MT. If printing on the print medium MT is done by a transfer method, the parameter used represents the texture when the ink printed on the transfer paper is transferred to the surface of the print medium MT. When printing is performed by directly ejecting UV ink onto the printing medium (MT), a parameter is used that represents the texture after the ink has been cured by ultraviolet light. The reflection model used in the lighting process is one of the mathematical models used to simulate illumination phenomena in the real world. The reflection model used in this embodiment will be explained in detail later.

[0050] Pixel manipulation processes become computationally intensive and time-consuming when the number of rasterized pixels increases, such as when the output resolution is high. Therefore, compared to vertex-level processing, it can take longer and result in inefficient pipeline processing. In this embodiment, the pixel shader PS processing program is optimized for execution on a GPU with high parallel processing capabilities, enabling advanced effects, including texture rendering, to be achieved in a short time.

[0051] The pixel information obtained through the processing of the pixel shader PS is further evaluated by the render backend (RBE) to determine whether or not to write it to the frame memory (FM) for display. Only when the render backend (RBE) determines that it is safe to write the pixel data to the frame memory (FM) is the pixel data saved as something to be rendered. Well-known tests used to determine whether to write include the "alpha test," "depth test," and "stencil test." The render backend (RBE) executes the configured test from among these tests and writes the pixel data to the frame memory (FM).

[0052] With the above processing completes the rendering pipeline, the post-processing unit (PST) then performs processing on the data stored in the frame memory (FM) to improve its appearance. Examples of such processing include anti-aliasing, which smooths out unwanted edges in the image. Other processes include ambient occlusion, screen-space reflection, and depth of field. The post-processing unit (PST) should be configured to perform the necessary post-processing.

[0053] The rendering execution unit 121 completes the above processing, and the rendering is finished. The result is output as the render result RRD. In practice, the data written to the frame memory FM is read out in accordance with the display cycle of the image display unit 151 and displayed as the render result RRD (Figure 6, step S160). An example of the render result RRD is shown in Figure 8. In this example, the image display unit 151 displays, as 3D objects placed in a virtual space, the ink-printed medium OJ1, the light source LG, and the background object Bob, which is part of a table and exists as one of the backgrounds.

[0054] Figure 9 illustrates the relationship between a print medium MT placed in a virtual space, a light source LG, and a viewpoint (camera) VP. The relationship between the light source LG, viewpoint VP, and the print medium MT is three-dimensional within the virtual space VSP, but the figure shows the virtual space VSP as the xz plane. x is the coordinate of the point where the vectors described below converge. The figure illustrates the positional relationship between the light source LG and viewpoint VP that illuminate the print medium MT, on which the image ORG to be rendered is printed, at a given coordinate x. The figure shows the light source direction vector ωl from coordinate x to the light source LG, the viewpoint direction vector ωv from coordinate x to the viewpoint VP, and the half vector HV of both. Furthermore, the symbol Np represents the normal vector assuming that the surface of the print medium MT is a virtual plane PLp when viewed microscopically, and the symbol Nb represents the normal vector at coordinate x of the actual plane PLb of the actual print medium MT, which is not a plane when viewed microscopically.

[0055] In the image processing apparatus 100 of this embodiment, the position and angle of the print medium in the virtual space can be freely changed, and the appearance of the print medium and the image on the print medium can be checked. As shown in Figure 8, this can be achieved by a series of processes in which, when the position and angle of the three-dimensional print medium MT are changed by operating a pointing device on the image displayed on the image display unit 151 and dragging the object with the pointer PTD displayed on the screen by the pointing device, the rendering execution unit 121 performs rendering processing each time, and the processing result is displayed on the image display unit 151. The pointing device may be a 3D mouse or a tracking ball, or it may be a type that is operated with a finger or stylus on a multi-touch panel provided on the image display unit 151. For example, if a multi-touch panel is provided on the surface of the image display unit 151, the print medium MT and light source LG may be moved directly with a finger, or two fingers may be used to rotate the print medium MT or change the distance between the light source LG and the actual plane PLb of the print medium in three dimensions.

[0056] When the position and angle of the print medium MT or light source LG in this virtual space are changed, the rendering execution unit 121 performs rendering processing each time, and the rendered result RRD is displayed on the image display unit 151. When the position and angle of the print medium MT or light source LG in the virtual space are changed, the print medium on which the image is printed is physically rendered each time, and the actual print medium on which the image is printed is shown in a manner close to how it would appear in real space.

[0057] In particular, in this embodiment, in addition to converting the colors of the image to be printed on the print medium to the actual colors of the printed image using a color management system (CMS), during the lighting process during rendering, [1] The printing medium on which the image is printed and the ink used for printing are treated as 3D objects. [2] The texture parameter TXT is used to consider the texture of the surface of the print medium and the ink printed on the surface. Therefore, the reproducibility of the printed medium displayed on the image display unit 151 is extremely high. The following describes the processes of [1] and [2].

[0058] [1] How a 3D object appears in a virtual space can be represented using the bidirectional reflectance distribution function (BRDF) and luminance of the reflected light at each part of the object. The bidirectional reflectance distribution function (BRDF) indicates the angular distribution characteristics of the reflected light when light is incident from a specific angle. Luminance is the brightness of the object. Together, these are also called the illumination model. An example of the reflection model adopted in this embodiment is shown below. The BRDF can be expressed as a function f(x,ωl,ωv), and the luminance as a function L(x,ωv), as shown in the following equations (2) and (3). f(x,ωl,ωv)=kD / π+kS*(F*D*V) …(2) L(x,ωv)=f(x,ωl,ωv)*E⊥(x)*n·ωl …(3) x: coordinate within the plane, ωv: viewpoint direction vector, ωl: light source direction vector kD: Diffuse Albedo, kS: Specular Albedo F: Fresnel term, D: Normal distribution function, V: Geometric decay term E⊥(x): Illuminance incident perpendicular to coordinate x, n: normal vector

[0059] The first term of the BRDF, kD / π, is the diffuse reflection component and is a Lambertian model. The second term is the specular reflection component and is a Cook-Torrance model. In (2), kD / π is sometimes called the diffuse reflection term and kS*(F*D*V) is sometimes called the specular reflection term. As the models and calculation methods for the Fresnel term F, the normal distribution function D, and the geometric decay term V are well known, explanations are omitted. For the BRDF, any function appropriate to the reflection characteristics of the 3D object surface and the purpose of rendering may be used. For example, the Disney Principled BRDF may be used. In this embodiment, the BRDF is used as a function to represent the reflection of light, but the Bidirectional Scattering Surface Reflectance Distribution Function (BSSRDF) may also be used as a function to represent the reflection of light.

[0060] As can be seen from equations (2) and (3) above, the calculation of the above reflection model requires the normal vector n, the light source direction vector ωl, and the viewpoint direction vector ωv. Printed media are treated as 3D objects composed of multiple minute polygons for rendering purposes, but the normal vector n, which reflects the minute irregularities on the surface of the printed media, is calculated from the polygon normal Np and the normal map described later. Therefore, the vertex pipeline VPL calculates the polygon normal Np and the UV coordinates that determine the reference position of the normal map, and inputs these, along with the light source direction vector ωl and the viewpoint direction vector ωv, into the pixel pipeline PPL. In the pixel pipeline PPL, the pixel shader PS references the normal map, which is given as one of the texture parameters, using the UV coordinates, and calculates the normal vector n from the referenced normal map value and the polygon normal Np.

[0061] In this embodiment, as described above, the print medium MT on which the image ORG is printed is treated as a 3D object, and physically based rendering is performed according to equations (2) and (3) above. The light source direction vector ωl and the viewpoint direction vector ωv are calculated each time the user changes the position and angle of the real plane PLb of the print medium or the light source LG in the virtual space using a pointing device, as shown in Figure 9.

[0062] Regarding [2]: In this embodiment, the texture of the surface of the printing medium and the texture of the printed ink are considered using the texture parameter TXT. The texture parameter TXT may include the following, but it is not necessary to consider all of them; at least one of the parameters listed below, for example, smoothness, should be considered. • Smoothness (S) or Roughness (R): This parameter indicates the smoothness of the surface of a 3D object. Smoothness S is generally specified within the range of 0.0 to 1.0. Smoothness S affects the normal distribution function D and the geometric attenuation term V of the BRDF in equation (2) above. A larger value results in stronger specular reflection and a glossy appearance. Roughness R can be used instead of smoothness S. The two are convertible using the formula S = 1.0 - R. Note that smoothness is sometimes referred to as "smoothness," and roughness as "roughness."

[0063] ·Metallic M (metallic): This indicates the degree to which a 3D object's surface is metallic. A higher metallicity (M) value indicates a higher surface metallicity. When metallicity (M) is high, the object's surface reflects ambient light more easily, reflecting the surrounding scenery and obscuring the object's own color. Metallicity (M) affects the Fresnel term (F). The Fresnel term F can be expressed as equation (4) below using the Schlick approximation. F(ωl,h)=F0+(1-F0)(1-ωl·h) 5 …(4) Here, h is the half-vector of the viewpoint direction vector ωv and the light source direction vector ωl, and F0 is the specular reflectance when perpendicularly incident. The specular reflectance F0 can be directly specified as the specular color of the specularly reflected light, or it can be given by linear interpolation (referred to here as the lerp function) using the metallicity M in equation (5). F0 = lerp(0.04, tC, M) …(5) Here, tC is the texture color (albedoColor) of the 3D object. The value 0.04 in equation (5) is a representative value for each RGB component, showing typical values ​​for nonmetals. The same applies to the texture color tC. Here, we refer to it as metallicity, but materials with high reflectivity and gloss, such as the surface of porcelain, are treated as having high metallicity M even if they are not metals.

[0064] • Normal Map: The normal map represents the normal vectors of the minute surface irregularities on the printed material. By associating (applying) the normal map to a 3D object, the normal vectors of the minute surface irregularities on the printed material can be assigned to the 3D object. The normal map can influence the Fresnel term F, normal distribution function D, and geometric decay term V of the BRDF.

[0065] • Other texture parameters: Other parameters that can function as texture parameters include specular color, and clear coat layer parameters that indicate the presence or absence of a clear coat layer on the surface of the printing medium, its thickness, or its transparency.

[0066] As explained above, [1] Treating the printing medium on which the image is printed as a 3D object, [2] The texture parameter TXT is used to consider the texture of the surface of the printing medium and the surface of the printed ink. As a result, the image processing device 100 of this embodiment can display the appearance of the printed medium MT on which the image ORG is printed on the image display unit 151 with a high degree of freedom and high reproducibility.

[0067] (A5) Effects of the first embodiment: According to the image processing apparatus 100 of the first embodiment described above, image data ORG of an image to be printed on a printing medium MT is acquired, and the appearance when printed on this printing medium MT with ink is rendered using a relief expression method selected based on the printing conditions PJ. Therefore, when printing once on plain paper with dye or pigment ink, the relief on the surface of the printing medium MT and the relief caused by the ink printing are not very large, so rendering is performed by normal processing. This reduces the processing load. Moreover, it can adequately express the relief on the surface of the paper and the relief caused by the ink. On the other hand, when printing on art fine paper with dye or pigment ink, or when performing two to three ink ejection and UV curing with UV ink, rendering is performed by normal processing using a normal map NM and parallax processing using a height map HM. This is because in such printing, the ink may rise significantly from the surface of the printing medium MT, for example, by about 0.2 mm.

[0068] On the other hand, when printing with UV ink four or more times and curing with UV, or when printing using DTFilm, the ink thickness becomes even higher, so rendering is performed using displacement processing POD with a displacement map DM. This increases the rendering load, but the uneven shape of the printing medium MT surface, including the ink thickness, is clearly represented. In this way, the method of representing the uneven shape is switched depending on the printing conditions PJ (see Figure 3A), so it is possible to balance the processing load with the reproducibility of the preview.

[0069] Figure 10 shows an example of how a print medium MT and the ink printed on it appear. The top row of the figure shows an example of relief representation using a normal map NM. In this example, the viewpoint is above the front of the print medium MT (Z direction), and the light source LG is positioned in both the Z and Y directions, i.e., diagonally above. The surface of the print medium MT has raised FP corresponding to Braille made of clear ink. The middle row of the figure shows the same object, but with the viewpoint positioned diagonally above. Both figures were rendered using normal processing PON with a normal map NM, and the relief is represented by shading. As will be explained later in comparison with the results of rendering using displacement processing POD with a displacement map DM, because the relief is represented by shading, depending on the viewing angle, as explained using Figure 3B, it may appear as if the plane is simply painted in different colors, or as if it represents recesses rather than protrusions.

[0070] The bottom row of Figure 10 shows an example of rendering results using displacement processing POD with a displacement map DM. Here, the string "UV Printing" (UVP) is formed on the black printing medium MT through four prints with white UV ink. This example employs displacement processing using a displacement map DM, and as shown in Figure 4, tessellation processing POT is performed to divide the polygons that make up the ink shape. Tessellation processing POT has already been explained, but the division of polygons affects the results of the subsequent rendering process. Therefore, this point will be explained using Figure 11.

[0071] The upper part of Figure 11 shows how the same text UVP appears when constructed with fewer polygons and rendered using displacement processing POD. Because there are fewer polygons and fewer vertices that can be displaced, the text UVP appears flat, and the expression of thickness and unevenness caused by the ink is weak. In general, if the printing medium MT is a paper with a smoothness greater than that of ordinary paper, such as photographic paper, then when constructing its surface with polygons, polygons of a size approximately equal to the resolution of the image display unit 151 are sufficient. This is because the unevenness of the surface of printing paper, etc., is small. This is not the case when the printing medium MT is a cloth or the like.

[0072] Figure 12 shows an example of the simplest polygon. Here, the vertical and horizontal resolution of the printing medium MT is assumed to be 72 × 72 dpI, and to represent this with a 3D model polygon, two triangular polygons PG, shown in the upper left of the figure, are placed in one of the sections that make up the plane. As a result, the printing medium MT, such as photographic paper, can be represented with six vertices. In reality, two of the six vertices can be shared, so it effectively becomes four vertices. An example in which a plane is constructed by arranging 10 of these combinations of two triangular polygons in the X and Y directions is also shown. In the displacement processing POD, the vertex positions of these triangular polygons are displaced in the Z direction, and the unevenness is represented by rendering.

[0073] To represent the thickness of ink on the printing medium (MT), the vertices of the polygons where the ink is printed are moved. However, if the number of polygons is small, the movement of vertices along the boundaries of characters is minimal, and as exemplified in the upper part of Figure 11, the relief is almost invisible. Therefore, the polygons are divided using tessellation processing (POT) to increase their number, and then displacement processing (POD) is performed. An example of the rendering of the result is shown in the lower part of the figure. Here, the number of polygons has been increased 16 times. An example of the method for increasing the number of polygons by dividing them is shown in the lower part of Figure 12. By bisecting each side of the original triangular polygon PG, it can be divided into four similar triangular polygons pg1. Furthermore, by bisecting each side of these smaller polygons pg1, they can each be divided into four smaller polygons mg1. Compared to the original polygon PG, these smaller polygons mg1 are 1 / 16th the size.

[0074] The preview screen shown in the lower part of Figure 11 shows the rendering result after increasing the number of polygons by 16 times and processing with displacement processing POD. At this stage, although the number of polygons has increased, it is still insufficient in terms of resolution, and when the vertices of the polygons located at the positions corresponding to the string UVP are distorted, the modeling becomes distorted from the original shape as shown in the figure. Therefore, if the small polygon mg1 is further divided into four, forming minute polygons ng1, the size becomes 1 / 64 of the original triangular polygon PG. Dividing the polygon to this size provides a sufficient number of vertices to reproduce the shape of the string UVP in displacement processing POD using displacement map DM, and the rendering result has sufficient shape reproduction, as exemplified in the bottom part of Figure 10. Of course, it is also possible to further divide the minute polygon ng1 into four, using polygons that are 1 / 256 the size of the original triangular polygon PG. The division is not limited to four divisions; it can be two, three, or five or more divisions. Furthermore, the shapes of the resulting polygons do not need to be identical or similar; they can be of different sizes, shapes, or any other design. The final number of divisions relative to the original shape also does not need to be limited to powers of four as described above. As already explained using equation (1), it can be determined from the image resolution.

[0075] In this embodiment, as described above, the method for expressing surface texture is selected based on the printing conditions PJ, balancing the rendering processing load with the appearance of surface texture on the printing medium MT and ink. When the printing medium MT has thickness and texture, such as cloth, or when the ink has considerable thickness, such as in multiple UV ink printing or transfer printing using DTFilm, displacement processing using a displacement map DM is selected. In displacement processing, the vertices of polygons forming the surface of the printing medium MT or ink are moved, or, if necessary, tessellation processing POT is performed to increase the number of polygons and then the vertices of the polygons are moved. Therefore, compared to normal processing or parallax processing that simply express texture through shading, the 3D structure of the surface can be realistically represented.

[0076] Figure 13 is a schematic diagram illustrating the appearance of protrusions (FPs) formed on the printing medium (MT) by multiple passes of clear ink. The protrusions (FPs) formed at the edges of the printing medium (MT) are formed by stacking ink in a truncated square pyramidal shape. The area of ​​the ink layer gradually decreases from bottom to top (z direction). The upper part of the figure shows a preview screen when displacement processing is performed using a displacement map (DM), and the lower part shows a preview screen when normal processing is performed using a normal map (NM). As shown in the figure, when using a displacement map (DM), the vertices of the polygon are actually displaced, so when viewed from an oblique viewpoint, the protrusions (FPs) at the edges of the printing medium (MT) appear to extend beyond the printing medium (MT) (part F in the figure). In contrast, when using a normal map (NM), even though the formation position of the protrusions (FPs) is the same as in the upper figure, the protrusions (FPs) remain within the range of the printing medium (MT). This is because the surface texture representation using normal maps (NM) does not involve moving polygon vertices, meaning the ink is not actually rising from the surface of the printing medium (MT). Rather, it simply calculates the direction of reflected light and applies shading as if the surface were raised.

[0077] B. Second Embodiment: Next, the image processing apparatus 100 of the second embodiment will be described. The image processing apparatus 100 of the second embodiment is the same as the first embodiment except for the configuration of the vertex pipeline VPL of the rendering execution unit 121. The configuration of the vertex pipeline VPL of the second embodiment is shown in Figure 14. In the second embodiment, the tessellation process POT is performed within the rendering execution unit 121, specifically in the vertex pipeline VPL. As shown in the figure, the vertex pipeline VPL of the second embodiment includes a hull shader HS, a tessellator TS, and a domain shader DS from the vertex shader VS to the rasterizer RRZ.

[0078] The vertex shader VS receives polygon data contained in the first data FD. The coordinate data of the vertices of the numerous polygons forming the surface of the print medium MT is input to the vertex shader VS. The hull shader HS sets the number of divisions of the polygon in the vertex pipeline VPL. The hull shader HS receives the number of divisions from an external source, which may be a number of divisions determined based on the resolution of the displacement map DM, as already explained using Figure 4, or a number of divisions set by the user. The hull shader HS uses this number of divisions to instruct the tessellator TS to divide the initial polygon input to the vertex shader VS. The function of the tessellator TS is the same as that of the tessellation process POT explained in Figure 4.

[0079] This polygon subdivision results in numerous smaller polygons, and the domain shader DS is used to set the coordinates of the vertices of these subdivided polygons. At the same time, displacement processing POD is performed. The coordinates of the vertices of the numerous smaller subdivided polygons are set to match the unevenness of the surface of the printing medium MT that is to be represented, as well as the unevenness caused by the thickness of the ink being printed. As a result, the representation of unevenness is achieved not simply by shading based on light reflection from the surface normal, but by the displacement of the polygon vertices.

[0080] In the configuration shown as the second embodiment, the tessellation processing POT function is performed within the rendering execution unit 121, but the displacement processing POD using the displacement map DM can be performed in the same way as in the first embodiment. Therefore, by selecting a method for expressing unevenness based on the printing conditions PJ, all the same effects as in the first embodiment can be achieved, such as balancing the rendering processing load with the appearance of unevenness on the printing medium MT and ink.

[0081] C. Third Embodiment: In the first and second embodiments described above, the print medium MT and the image ORG printed thereon are virtual entities handled within the image processing device 100. Rendering is performed using various data, and the image, including its texture, is displayed as a preview image on the image display unit 151. In contrast, the third embodiment describes an image printing system 300 that actually performs printing using the image processing device 100. Figure 15 is a schematic diagram showing the configuration of this image printing system 300. The image printing system 300 comprises the image processing device 100 described in the first and second embodiments, an image display unit 151 that displays the preview image, and a printing device 200 that performs printing. The image processing device 100 receives image data ORG, printing condition data PJD specifying the type of print medium MT and printing method, and texture / environment parameters TXD via the input unit 110. The printing condition data PJD includes data stored in the printing condition database 50, such as the type of print medium MT, the type of ink used for printing, and the number of prints. Furthermore, the texture and environment parameter TXD includes various information necessary for previewing, such as the texture parameter TXT and lighting information LGT. The contents of this data have already been explained in the configuration and operation of the rendering execution unit 121.

[0082] The printing apparatus 200 prints images onto various printing media MT using dye inks, pigment inks, or, if necessary, UV inks. Printing media MT include not only plain paper but also fabrics. In the case of fabrics, direct printing is possible, but a practical method involves first printing onto transfer paper and then transferring it to the printing media MT. UV ink is a type of ink that hardens with ultraviolet (UV) light; similar inks, such as heat-curing inks, can also be used.

[0083] Transfer printing is performed using the following procedure. First, ink is applied to the recording surface of the transfer paper PRS by printing with the printing device 200. Then, with the recording surface of the transfer paper PRS pressed against the printing medium MT, both the transfer paper PRS and the printing medium MT are heated. As a result, an image is printed onto the object by thermal transfer. Transfer printing methods include sublimation transfer and DTFilm transfer. By applying such transfer printing, it becomes possible to print on various media other than paper as the printing medium MT. For example, it is possible to print on various media such as cloth, leather, vinyl fabric, or sewn products such as bags, shoes, and clothing made from these materials, as well as ceramics, plastics, glass, and wood. The shape of the printing medium MT is not limited to a sheet; it can be a plate, cylinder, or rectangle, or any shape on which the transfer sheet can be placed, and can be treated as the printing medium MT.

[0084] This embodiment describes an example of sublimation transfer. Rendering is performed using the image processing device 100, and a preview of the surface texture of the printing medium MT is performed. Once the desired printed material is obtained by confirming the preview, the user uses the printing device 200 to print the previewed image onto the transfer paper PRS. The printing device 200 inverts the print image data (PIM) received from the image processing device 100, horizontally flipping the print image. The horizontally flipped image is printed onto the recording surface of the transfer paper PRS. The surface of the transfer paper PRS is the recording surface. With the recording surface of the printed transfer paper PRS pressed against the surface of the printing medium MT, the printed transfer paper PRS and the printing medium MT are heated. For heat transfer, a heat press (HPM) is used according to the shape of the printing medium MT. A flat press is used when the object to be printed is fabric, etc. A mug press is used when the object to be printed is cylindrical, such as a mug or glass. A vacuum press is used when the object to be printed is a three-dimensional object such as a smartphone case. As a result of the transfer process, a product with the image printed on its surface is obtained.

[0085] D. Other embodiments: (1) This disclosure can be implemented as an image display device. This image display device comprises an input unit for inputting image data of an image to be printed on a printing medium, an acquisition unit for acquiring printing conditions for the printing medium, a selection unit for selecting a method for expressing relief when processing the image data from the acquired printing conditions, and a display execution unit for previewing the printed material using the selected method for expressing relief. In this way, a method for expressing relief can be selected from the printing conditions, and the printed material can be previewed using the selected method for expressing relief. The image data input unit inputs image data to be printed by various methods, such as by communication or via a memory card. The image data may be further edited within the image display device, or the printing position on the printing medium may be corrected. The image display device may be configured as a single unit, or it may be distributed on a network according to its function and cooperate to realize the preview of the printed material. The preview display only needs to be able to display the state of the printed material on various displays, and the display can be of any type, such as a liquid crystal display, an organic EL display, or a projector.

[0086] (2) In the above configuration, the printing conditions may include at least one of the type of printing medium and the classification of the image forming material conditions. This allows for the selection of a method for expressing surface irregularities that is suitable for the height of the irregularities formed by the image forming material, corresponding to the height of the irregularities on the surface of the printing medium which differs depending on the type of printing medium and the classification of the image forming material conditions. Various materials can be used as the image forming material, such as ink, DTFilm, transfer sheets, and 3D printer filaments. Various inks can be used, such as dye ink, pigment ink, photocurable ink (UV ink), and thermocurable ink.

[0087] (3) In the above configuration, the classification of the conditions for the image forming material may include the type of ink used for printing the image and the amount of ink. This makes it easy to select a method for representing the unevenness when processing the image data, corresponding to the height of the unevenness formed on the printing medium by the image forming material in relation to the image. The classification of the conditions for the image forming material can be set by the type of ink, such as dye, pigment, or photocurable ink, or by the amount of ink, such as the number of prints or the range of ink duty in a single print.

[0088] (4) In the above configuration, the classification of the type of printing medium and the classification of the image forming material conditions are associated with the height of the unevenness on the printing medium on which the image is formed, and the selection unit may select the unevenness representation according to at least one of the classification of the type of printing medium and the classification of the image forming material conditions. In this way, by obtaining the classification of the type of printing medium and the classification of the image forming material conditions as printing conditions, it is easy to realize an expression corresponding to the height of the unevenness. The classification of the type of printing medium and the classification of the image forming material conditions associated with the height of the unevenness may be measured in advance and linked to the printing conditions. In this way, it is easy to select a method of expressing unevenness corresponding to the classification of the type of printing medium and the classification of the image forming material conditions associated with the height of the unevenness.

[0089] (5) In the configurations of (1) to (4) above, the selection unit may select the method of expressing the unevenness according to the height of the unevenness on the printing medium on which the image is formed and a threshold value provided for selecting the method of expressing the unevenness. In this way, the selection of the unevenness expression can be easily made by simply comparing the two. The "height of the unevenness on the printing medium on which the image is formed" during the comparison can be determined by pre-associating the height of the unevenness that may occur depending on the type of printing medium and the conditions of the image forming material, which are the printing conditions, with the printing conditions, and then specifying the height of the unevenness according to the printing conditions acquired by the acquisition unit.

[0090] (6) In the configurations of (1) to (5) above, the acquisition unit may select one printing method from a plurality of printing methods as the printing condition, and the selection unit may select the method of relief expression based on the printing method selected from the plurality of printing methods. In this way, it is only necessary to select the printing method, and the selection of relief expression can be easily performed.

[0091] (7) In the configurations of (1) to (6) above, the selection unit may select at least one of the following as the method for representing the relief of relief: normal processing, parallax processing, and displacement processing. This allows for a wide range of processes, from processing with low processing load to relief of relief that corresponds to the actual shape, although the processing load is high. These processes may be performed by selecting only one of them, or they may be performed in combination, such as normal processing and parallax processing.

[0092] (8) In the configurations of (1) to (7) above, the method for representing bumps and ridges includes at least normal processing and parallax processing, and the selection unit may select normal processing as the method for representing bumps and ridges if the height of the bumps and ridges on the printing medium on which the image is formed is less than a predetermined first threshold, and select normal processing and parallax processing together as the method for representing bumps and ridges if the height of the bumps and ridges is equal to or greater than the first threshold. In this way, when the height of the bumps and ridges is small, normal processing, which has a low processing load, can be adopted as the method for representing bumps and ridges, and when the height of the bumps and ridges is large, normal processing and parallax processing can be selected together as the method for representing bumps and ridges, making it easier to balance processing load and accuracy of bump representation.

[0093] (9) In the configurations of (1) to (8) above, the selection unit may, if the height of the unevenness on the printing medium on which the image is formed is less than a predetermined second threshold, perform at least the normal processing among the normal processing and the parallax processing as the method for representing the unevenness, and if it is above the second threshold, select displacement processing as the method for representing the unevenness. In this way, when the height of the unevenness is small, processing including normal processing, which has a low processing load, can be adopted as the method for representing the unevenness, and when the height of the unevenness is even larger, displacement processing can be selected as the method for representing the unevenness, making it easier to balance the processing load when the height of the unevenness is relatively large with the accuracy of the representation of the unevenness.

[0094] (10) In the configurations of (1) to (9) above, the selection unit may select at least one of the normal processing and parallax processing as the method for representing the unevenness if the height of the unevenness on the printing medium on which the image is formed is less than a predetermined first threshold, select both normal processing and parallax processing as the method for representing the unevenness if the height of the unevenness is less than a second threshold greater than the first threshold but greater than or equal to the first threshold, and select displacement processing as the method for representing the unevenness if the height of the unevenness is greater than or equal to the second threshold. This makes it easy to accurately use normal processing, parallax processing and displacement processing depending on the height of the unevenness.

[0095] (11) In the configurations of (1) to (10) above, the method for representing the unevenness includes at least displacement processing, the display execution unit represents the printing medium and the image forming material with a plurality of polygons, and when performing the displacement processing, the polygons may be divided. This makes it easier to avoid situations where the shape reproduction is insufficient due to the size of the polygons.

[0096] (12) In the configurations of (1) to (11) above, the number of divisions of the polygon may be determined according to the resolution of the image. This makes it easier to determine the number of divisions.

[0097] (13) In the configurations of (1) to (12) above, the number of divisions of the polygon may be accepted from an external source. This allows the polygon size to be adjusted to suit the user's perception and reduces any visual incongruity regarding the shape.

[0098] (14) The present disclosure can also be implemented as an image printing system. This image printing system comprises an input unit for inputting image data of an image formed on a printing medium using an image forming material; an acquisition unit for acquiring printing conditions which are the conditions for printing on the printing medium; a selection unit for selecting a method of relief expression when processing the image data from the acquired printing conditions; a display execution unit for displaying a preview of the printed material using the selected relief expression method; and a printing device for printing the previewed image onto the printing medium. In this way, the appearance, including the image to be printed on the printing medium, can be previewed prior to printing by the printing device, and moreover, the printed material is previewed with relief expression that conforms to the printing conditions, making it easy to understand the printed material that will be printed under those printing conditions.

[0099] (15) The present disclosure can also be implemented as an image processing program that generates a rendering image of a print medium on which an image is printed. This image processing program uses a computer to implement the following functions: inputting image data of the image formed on the print medium using an image forming material; acquiring printing conditions which are the conditions for printing on the print medium; selecting a method for expressing relief when processing the image data from the acquired printing conditions; and displaying a preview of the printed material using the selected method of expressing relief. In this way, a preview of the printed material can be easily implemented using a computer, and since the printed material is previewed with relief that conforms to the printing conditions, it is easy to understand what the printed material will look like when printed under those printing conditions.

[0100] (16) In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software. At least some of the configurations implemented by software can also be implemented by discrete circuit configurations. Furthermore, if some or all of the functions of this disclosure are implemented by software, the software (computer program) may be provided in the form of being stored on a computer-readable recording medium. "Computer-readable recording medium" is not limited to portable recording media such as flexible disks and CD-ROMs, but also includes various internal storage devices in a computer such as RAM and ROM, and external storage devices fixed to a computer such as hard disks. In other words, "computer-readable recording medium" has a broad meaning that includes any recording medium on which data packets can be fixed rather than temporary.

[0101] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0102] 20...Color management system, 40...Selection unit, 50...Printing condition database, 60...Acquisition unit, 100...Image processing unit, 110...Input unit, 111...Relief expression processing unit, 121...Rendering execution unit, 135...Memory, 141...Communication unit, 151...Rendered image display unit, 190...Site, 200...Printing device, 300...Image printing system

Claims

1. An input unit for inputting image data of an image formed on a printing medium using an image forming material, An acquisition unit that acquires printing conditions, which are the conditions for printing on the aforementioned printing medium, A selection unit that selects a method for representing relief when processing the image data based on the acquired printing conditions, A display execution unit that displays a preview of the printed material using the selected method of expressing relief, An image processing device equipped with the following features.

2. The image processing apparatus according to claim 1, wherein the printing conditions include at least one of the type of printing medium and the classification of the conditions of the image forming material.

3. The image processing apparatus according to claim 2, wherein the classification of the conditions for the image forming material includes the type of ink used for printing the image and the amount of ink used.

4. An image processing apparatus according to claim 2, The classification of the type of printing medium and the conditions of the image-forming material corresponds to the height of the irregularities on the printing medium on which the image is formed. The selection unit selects the relief expression according to at least one of the classifications of the type of printing medium and the conditions of the image forming material. Image processing device.

5. The image processing apparatus according to claim 1, wherein the selection unit selects a method for representing the unevenness on the printing medium on which the image is formed, and a threshold value provided for selecting a method for representing the unevenness.

6. An image processing apparatus according to claim 1, The acquisition unit selects one printing method from several types of printing methods as the printing condition. The selection unit selects the method of expressing relief according to the printing method selected from the multiple types of printing methods. Image processing device.

7. The image processing apparatus according to any one of claims 1 to 6, wherein the selection unit selects at least one of normal processing, parallax processing, and displacement processing as the method for representing the unevenness.

8. An image processing apparatus according to claim 7, The method for representing uneven surfaces includes at least normal processing and parallax processing. The selection unit selects normal processing as the method for representing the unevenness if the height of the unevenness on the printing medium on which the image is formed is less than a predetermined first threshold, and selects normal processing and parallax processing together as the method for representing the unevenness if the height of the unevenness is equal to or greater than the first threshold. Image processing device.

9. The image processing apparatus according to claim 7, wherein the selection unit, when the height of the unevenness on the printing medium on which the image is formed is less than a predetermined second threshold, performs at least the normal processing among the normal processing and the parallax processing as the method for representing the unevenness, and when it is equal to or greater than the second threshold, selects displacement processing as the method for representing the unevenness.

10. An image processing apparatus according to claim 7, The aforementioned selection unit is If the height of the unevenness on the printing medium on which the image is formed is less than a predetermined first threshold, then at least one of the normal processing and the parallax processing is selected as the method for representing the unevenness. If the height of the unevenness is less than a second threshold greater than the first threshold and greater than or equal to the first threshold, then normal processing and parallax processing are selected together as the method for representing the unevenness. If the height of the unevenness is greater than or equal to the second threshold, displacement processing is selected as the method for representing the unevenness. Image processing device.

11. An image processing apparatus according to any one of claims 1 to 6, The method for representing unevenness includes at least displacement processing, The display execution unit represents the printing medium and the image forming material using a plurality of polygons, and performs polygon division processing when performing the displacement processing. Image processing device.

12. The image processing apparatus according to claim 11, wherein the number of divisions of the polygon is determined according to the resolution of the image.

13. The image processing apparatus according to claim 11, which receives the number of divisions of the polygon from an external source.

14. An input section for inputting image data onto a print medium, An acquisition unit that acquires printing conditions, which are the conditions for printing on the aforementioned printing medium, A selection unit that selects a method for representing relief when processing the image data based on the acquired printing conditions, A display execution unit that displays a preview of the printed material using the selected method of expressing relief, A printing device that prints the previewed image onto the printing medium, An image printing system equipped with this system.

15. An image processing program that generates a rendered image of a printed medium on which an image has been printed, A function for inputting image data of the image formed on the printing medium using an image forming material, A function to acquire printing conditions, which are the conditions for printing on the aforementioned printing medium, A function to select a method for representing relief when processing the image data based on the acquired printing conditions, A function to preview a printed document using the selected method for representing relief, An image processing program that uses a computer to achieve this.

Images