Image processing apparatus, image processing method, and program

Abstract

To reduce the frequency of re-generation or update of a learned model, while coping with color variations.SOLUTION: An image processing apparatus 100 uses a learned model to create a predicted image that is an image obtained by predicting an output image corresponding to an input image. The image processing apparatus adjusts the pixel value of a target image by using an adjustment parameter in order to reduce the difference between the output image and the predicted image. The image processing apparatus determines whether to update the learned model or update only the adjustment parameter without updating the learned model based on the difference between the output image and the predicted image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating image data using machine learning. [Background technology]

[0002] There is a technology that uses a trained model to generate image data that serves as a reference when inspecting a printed matter, for example. Patent Document 1 discloses a technology that automatically inspects or checks a printed matter by comparing image data generated using a trained model generated by applying deep learning with read image data obtained by reading a printed matter to be inspected. Specifically, the trained model disclosed in Patent Document 1 infers an image to be formed on a printed matter using data equivalent to input image data input to an image forming device for obtaining a printed matter as input, and generates and outputs image data that serves as a reference for inspection as an inference result.

[0003] Incidentally, as image forming apparatuses, inkjet type apparatuses that form an image on a recording medium by ejecting ink from a plurality of nozzles, and electrophotographic type apparatuses that form an image on a recording medium using a laser photosensitive body and a charged toner are both widely used. In electrophotographic image formation, it is known that the density or color of the formed image changes (hereinafter referred to as "color fluctuation") due to the amount of toner remaining in the apparatus, or environmental conditions such as the surrounding temperature or humidity. Similarly, in inkjet type image formation, it is known that color fluctuation occurs due to ink adhesion around the nozzles, aging of the piezo element or heater that controls the ink ejection, or environmental conditions such as the surrounding temperature or humidity. Such color fluctuation may cause the image generated by the trained model and the image formed on the printed matter to have different colors. In this case, for example, even if there is no problem with the visual color of the image formed on the printed matter, depending on the degree of color fluctuation, it may be determined to be defective in the above-mentioned inspection or inspection. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-186938 A Summary of the Invention [Problem to be solved by the invention]

[0005] When generating an image corresponding to color variation using a trained model, the technology disclosed in Patent Document 1 requires either generating a new trained model using the scanned image after the color variation or updating the trained model by additional learning of the existing trained model. However, generating or updating the trained model requires a huge amount of calculation or time. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes a generation means for generating a predicted image, which is an image obtained by predicting an output image corresponding to an input image, using a trained model, an adjustment means for adjusting pixel values ​​of a target image using adjustment parameters in order to reduce the difference between the output image and the predicted image, and a determination means for determining, based on the difference between the output image and the predicted image, whether to update the trained model or to update only the adjustment parameters without updating the trained model. Effect of the Invention

[0007] According to the present disclosure, it is possible to reduce the number of times a new trained model is generated or an existing trained model is updated while responding to color variations. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system and a hardware configuration of an image processing device according to a first embodiment. [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of an image forming unit in the image forming apparatus according to the first embodiment. [Diagram 3]1 is a block diagram showing an example of a functional configuration of an image forming apparatus according to a first embodiment. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 5] 4 is a block diagram showing an example of the functional configuration of a printing unit and an updating unit according to the first embodiment. FIG. [Figure 6] 5 is a flowchart showing an example of a processing flow of a calibration process in the image processing device according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a calibration chart according to the first embodiment. [Figure 8] 7 is a flowchart showing an example of the process flow in S604 shown in FIG. [Figure 9] 5A and 5B are diagrams illustrating an example of one-dimensional characteristics of calibration read data and calibration reference data according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of a processing flow of a calibration process in the image processing device according to the second embodiment. [Figure 12] 13 is a flowchart showing an example of a process flow of a calibration process in the image processing device according to Modification 1 of the second embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a display screen according to Modification 1 of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals.

[0010] [Embodiment 1] <Image processing system configuration> 1(a) is a block diagram showing an example of the configuration of an image processing system 1 according to the first embodiment. The image processing system includes an image processing device 100, an image forming device 104, and a reading device 105, which are connected to each other so as to be able to communicate with each other. An output device 109, an input device 110, and an external storage device 111 are also connected to the image processing device 100. Functions of the image processing device 100, the image forming device 104, and the reading device 105, as well as the output device 109, the input device 110, and the external storage device 111, will be described later.

[0011] <Hardware configuration of image processing device> 1(b) is a block diagram showing an example of a hardware configuration of an image processing device 100 according to embodiment 1. The image processing device 100 has a CPU 101, a RAM 102, a storage device 103, an output interface (I / F) 106, a general-purpose interface (I / F) 107, and a main bus 108. An output device 109 is connected to the image processing device 100 via the output I / F 106, and an input device 110 and an external storage device 111 are connected to the image processing device 100 via the general-purpose I / F 107.

[0012] The CPU (Central Processing Unit) 101 is a processor that controls each part in the image processing device 100. Here, the CPU 100 is described as the source of overall control of the image processing device 100, but the CPU 100 may be composed of multiple processors, and each processor may share processing to control the entire image processing device 100. Also, part of the control processing of the CPU 100 may be performed by hardware such as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array). The RAM (Random Access Memory) 102 functions as a main storage device, a work area, etc. of the CPU 101. The storage device 103 stores programs executed by the CPU 101 and data used when executing the programs.

[0013] The output I / F 106 is an image output interface such as a DVI (Digital Visual Interface). The output device 109 connected via the output I / F 106 is, for example, a liquid crystal display, and functions as a user interface that presents the status of each device of the image processing system 1, such as the image processing device 100, the image forming device 104, and the reading device 105, to the user. The general-purpose I / F 107 is a bus interface such as a USB (Universal Serial Bus) or an IEEE (Institute of Electrical and Electronics Engineers) 1394. Information on operations (instructions) by a user is received from an input device 110, such as a keyboard or a mouse, connected to the image processing device 100 via the general-purpose I / F 107. The image processing device 100 is also connected to an external storage device 111 via the general-purpose I / F 107, and the user can store data such as a log in the external storage device 111 and can cause the image processing device 100 to obtain desired data from the external storage device 111.

[0014] The main bus 108 connects each hardware of the image processing device 100 so as to be able to communicate with each other. The hardware configuration of the image processing device 100 is not limited to the above-mentioned configuration. For example, the image forming device 104 or the reading device 105 may be connected to the image processing device 100 via an I / F such as the general-purpose I / F 106, and the above-mentioned external devices such as the output device 109 may be present inside the image processing device 100 via the main bus 108. The output device 109 and the input device 110 may be integrated as a touch panel display or the like. The image processing device 100 may have an I / F for connecting to an external network and may be configured to be able to transmit and receive information to and from the external network via the I / F. The image processing device 100 may have a GPU (Graphics Processing Unit), which is a processor specialized for high-speed parallel calculation, and may be configured to perform part of the control processing of the CPU 100 by the GPU.

[0015] <Functions of each device> The image forming device 104 forms an image on a recording medium such as a printing paper based on the received print job. In the following, as an example, the image forming method in the image forming device 104 is described as an inkjet method. The image forming method in the image forming device 104 may be an electrophotographic method or another method. The reading device 105 obtains a read image by reading an image formed on a recording medium. Data of the read image obtained by the reading device 105 (hereinafter referred to as "read data") is transmitted to the image processing device 100. The read data is data representing the color and color density of each pixel in the read image. In the following, as an example, the reading device 105 is described as an inline scanner. The functions of the image processing device 100 will be described later.

[0016] <Hardware configuration of image forming device> 2 is a diagram showing a schematic diagram of an example of a hardware configuration of an image forming unit in the image forming apparatus 104 according to the first embodiment. As described above, the image forming apparatus 104 in this embodiment is an inkjet type that forms an image on a recording medium by ejecting ink, which is a color material, from nozzles onto the recording medium. In this embodiment, it is assumed that the image forming apparatus 104 uses four colors of ink, namely black (K), cyan (C), magenta (M), and yellow (Y), to form an image.

[0017] The image forming apparatus 104 includes print heads 201 to 204, as shown in FIG. 2(a) as an example. Each of the print heads 201 to 204 is fixed at a position shown in FIG. 2(a), for example, and ejects each of the inks K, C, M, or Y in a range corresponding to the width of the recording medium 206. Such an image forming apparatus using an ink ejection method is generally called a "full line type". Each of the print heads 201 to 204 is composed of a combination of multiple head modules. FIG. 2(b) shows an example of the configuration of the print heads 201 to 204. As shown in FIG. 2(b), for example, the print head 201 is composed of a combination of head modules 201a, 201b, and 201c. The head modules 201a, 201b, and 201c are arranged side by side in the main scanning direction, with adjacent head modules being arranged at different positions in the sub-scanning direction. Similarly, the print head 202 is composed of a combination of head modules 202a, 202b, and 202c, and the print head 203 is composed of a combination of head modules 203a, 203b, and 203c. Similarly, the print head 204 is composed of a combination of head modules 204a, 204b, and 204c.

[0018] Further, each head module is composed of a plurality of chip modules. FIG. 2(c) shows the configuration of the head module 201a as an example of the configuration of each head module. The head module 201a is composed of chip modules 201a-1 to 201a-5. Hereinafter, each chip module will be described as being connected to a board independent of each other. Each chip module has a plurality of nozzles (also called "ejection ports") therein. FIG. 2(d) shows an example of the chip module as viewed from the recording medium 206 side. As shown in FIG. 2(d), for example, each chip module has a total of 16 nozzles across two rows. Note that the number of nozzles is not limited to 16, and may be 15 or less or 17 or more. The nozzles are arranged so as to be equal to the recording resolution of the image forming device, and the recording resolution is 1200 dpi (dots per inch), for example.

[0019] The recording medium 206 is conveyed in the direction indicated by an arrow 207 shown in FIG. 2(a) (from top to bottom in FIG. 2(a)) by the rotation of a conveying roller 205 (and other rollers not shown in FIG. 2(a)) by the driving force of a motor not shown. While the recording medium 206 is conveyed and passes fixed positions of the recording heads 201 to 204, ink is ejected from a plurality of nozzles arranged in each of the recording heads 201 to 204 according to a print job, thereby forming an image on the recording medium 206. When ink is ejected once from each nozzle of the plurality of nozzles according to the print job, an image for one raster (also called a "row") corresponding to the nozzle row is formed. The image forming device 104 forms the entire image on the recording medium by repeating the ink ejection operation at high speed while conveying the recording medium.

[0020] The image forming apparatus 104 is not limited to an inkjet type full-line type apparatus. For example, the image forming apparatus 104 may be a so-called serial type inkjet type apparatus that forms an image while moving a recording head in the main scanning direction. The image forming apparatus 104 may be an electrophotographic type apparatus that forms an image using a laser photosensitive body and a charged toner, or a thermal transfer type apparatus that vaporizes solid ink by heat and transfers it to a recording medium. The image forming apparatus 104 may be an offset printing type apparatus that prints ink applied to a plate via an intermediate transfer body onto a recording medium.

[0021] Fig. 2(a) also illustrates the reading device 105. The reading device 105 shown as an example in Fig. 2(a) is a line sensor having a length corresponding to the width of the recording medium, and is installed downstream of the recording heads 201 to 204. The reading device 105 sequentially captures images of the conveyed recording medium as RGB information, and can acquire the images as two-dimensional read data consisting of pixel values ​​of three RGB channels.

[0022] The reading device 105 is not limited to the line sensor described above, and may be, for example, a device equipped with a carriage for moving the sensor in the main scanning direction and capable of capturing an image of any area of ​​the recording medium. The reading device 105 may also be provided as an external device of the image forming device 104, and may obtain the CIE L by measuring the recording medium on which an image is formed. * a * b * (hereinafter simply referred to as "Lab") color space values. In this case, the reading device 105 is composed of a light source, such as a colorimeter or a spectrodensitometer, and a measuring instrument for quantitatively determining the color of an object. Note that the Lab color space is an example of a device-independent color system, and the reading device 105 may measure or obtain values ​​of other color systems, such as the XYZ color system expressed by the tristimulus values ​​XYZ of an object.

[0023] <Functional configuration of the image forming apparatus> The image forming process in the image forming apparatus 104 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the image forming apparatus 104 according to the first embodiment. The image forming apparatus 104 has, as its functional configuration, an input unit 302, a color conversion unit 303, a halftone processing unit 304, and a drawing unit 305. Terminals 301 and 306 shown in Fig. 3 are data input / output terminals. The input unit 302 receives a print job transmitted from an external device such as the image processing apparatus 100 via the terminal 301. Hereinafter, it is assumed that the data format of the input image included in the print job received by the input unit 302 is expressed as a pixel value of three RGB channels, each pixel value being composed of 8 bits.

[0024] The color conversion unit 303 receives input image data included in the print job received by the input unit 302, and refers to profile information included in the input image data to convert each pixel value (RGB value) of the input image into a value in a color space corresponding to the ink color of the image forming device 104. For example, this conversion uses a color conversion lookup table (LUT) created in advance for each profile. Hereinafter, the color conversion unit 303 will be described as converting input image data expressed by 8-bit pixel values ​​of 3 RGB channels into image data expressed by 8-bit pixel values ​​of 4 CMYK channels (hereinafter referred to as "CMYK image data").

[0025] The halftone processing unit 304 performs halftone processing on the CMYK image data after color conversion by the color conversion unit 303 to generate halftone image data. The halftone processing is a process of converting the CMYK image data into a number of gradations that can be directly expressed by the image forming device 104, such as a binary value indicating whether or not to eject ink for each CMYK ink. In order to generate halftone image data, a technique such as an error diffusion method or a dither method can be used. Hereinafter, the halftone image data will be described as being expressed in a CMYK 4-channel binary format. The drawing unit 305 controls the ejection of ink from each of the recording heads 201 to 204 onto a recording medium based on the halftone image data generated by the halftone processing unit 304, and draws an image corresponding to the input image on the recording medium to form the image. The drawing unit 305 also outputs the halftone image data and information indicating the recording of the control via a terminal 306.

[0026] <Image processing device functions> The image processing device 100 according to this embodiment performs an inspection process when a recording medium on which an image has been formed by an image forming device 104 (hereinafter referred to as a "printed matter") is delivered to a customer who orders the printed matter. Here, the inspection is performed to ensure that the printed matter is free of defects and has no quality problems. The inspection process is performed automatically by comparing read data obtained by reading the printed matter to be inspected by a reading device 105 with reference data that is the inspection standard and has been determined in advance to be a non-defective product. The image data that is the basis of the reference data according to this embodiment is automatically generated using a trained model.

[0027] <About trained models> A trained model according to this embodiment (hereinafter, simply referred to as a trained model) will be described. The trained model is, for example, configured by a convolutional neural network (hereinafter, referred to as a "CNN (Convolutional Neural Network)") used in image processing technology to which deep learning is applied. CNN is a technology in which a unit process is a nonlinear operation performed on image data after convolution of an image (or after transposition convolution) using a filter generated by learning, and this unit process is repeatedly applied. A set of one or more nodes that perform this unit process is called a layer. The above-mentioned filter is also called a local receptive field. Image-like data obtained by convolving an image using a filter and then performing a nonlinear operation is called a feature map, and is called an n-th layer feature map (n is a natural number) and a feature map in an n-th layer filter, etc., depending on the order of the layers. In addition, a CNN that repeats convolution using a filter and nonlinear operation m (m is a natural number equal to or greater than n) times is called a CNN having an m-layer network structure, etc.

[0028] When the image data input to the trained model has multiple color channels such as RGB, or when the feature map is composed of multiple channels, that is, when the input data to the layer is multi-channel, the number of filters required corresponds to the number of channels. Hereinafter, a set of filters corresponding to each channel will be referred to as a filter set. One filter set outputs one feature map. Each layer has a number of filter sets corresponding to the number of feature maps to be output. That is, the convolution filter of a certain layer is expressed as a four-dimensional array consisting of the number of channels of the input image data or feature map and the number of channels of the output feature map in addition to information indicating the vertical and horizontal sizes of the input image data or feature map. The processing in each layer can be expressed by an equation in which a bias addition operation is added to the value after convolution. In general, the equation can be formulated as the following equation 1. (Formula 1) TIFF2024116688000002.tif13150

[0029] In Equation 1, W n is the nth layer filter, b n is the bias of the nth layer, f() is a nonlinear operation, and X n is the nth layer feature map, and * is the convolution operator. The (l,k) on the right indicates that it is the kth channel in the lth (l is a natural number equal to or less than L, and L is the number of output channels of this layer) filter or feature map. K is the number of channels input to this layer. The filters and biases are generated by learning, which will be described later, and are collectively called "network parameters." As the nonlinear operation f(), a sigmoid function, ReLU (Rectified Linear Unit), or hyperbolic tangent function (tanh), etc., are used. The nonlinear operation f() is also called an activation function. Note that each layer of the CNN network may process the feature map output from a distant layer as input, rather than only the feature map output from the previous layer. As an example of a network with such a structure, U-Net, ResNet, etc. are known.

[0030] In deep learning, the accuracy of inference processing is generally improved by making the CNN structure multi-layered (deep) so that convolution by filters is performed many times. Simply put, the reason that the accuracy of inference processing is improved by multi-layer CNN is that the nonlinear relationship between input and output can be expressed by the stacking of nonlinear operations in each layer by repeating nonlinear operations many times.

[0031] The learning method of CNN will be explained. CNN learning is performed using learning data consisting of a pair of input image data and output image (correct image) data. CNN learning refers to generating, using the learning data, a filter and bias that can convert input image data into correct image data corresponding to the input image, that is, network parameter values. First, the CNN network parameters are given arbitrary initial values. CNN learning is performed by updating the network parameters for the learning data so as to minimize an objective function as expressed by the following equation 2. (Formula 2) TIFF2024116688000003.tif13150

[0032] In Equation 2, L() is the objective function, which is a loss function that measures the error between the correct answer and its prediction. θ is the network parameter (filter and bias) and is the variable of the loss function L(). X i is the data of the i-th input image, Y i is the data of the i-th correct image. Furthermore, F() is a function that collectively represents the nonlinear operations and other calculations (Equation 1 above) performed in each layer of CNN. ||x||2 is the L2 norm of x. n is the number of training data used for training, and is the number of input images. Note that n may be the total number of training data, or may be the number of training data randomly extracted from all of the training data. This is because the number of training data is generally large.

[0033] In this way, the network parameters can be updated by a method such as mini-batch gradient descent using a randomly extracted portion of the training data. Mini-batch gradient descent is a method for updating the network parameters using a gradient calculated using a randomly extracted portion of the training data instead of calculating the loss using all the training data. According to the mini-batch gradient descent, the network parameters converge in fewer training iterations, so that the amount of calculation in the training can be reduced.

[0034] From the above definitions, Equation 2 is i is input to the network F() whose network parameter is θ, and the obtained predicted value is used as the correct image data Y i It should be noted that the objective function shown in Equation 2 is merely an example, and the objective function may be modified or added depending on the characteristics to be imitated.

[0035] The network parameter θ is updated by updating the value in a direction expected to reduce the error by going back through the network structure based on the differential value (gradient) of each parameter with respect to the error L, which is calculated at the same time as the parameter update. This type of update method is called the error backpropagation method. As this update method, that is, a method for minimizing the objective function, the momentum method, the AdaGrad method, the AdaDelta method, the Adam method, etc. are known. Basically, any method may be used as the minimization method. It is known that the convergence of each minimization method differs, resulting in differences in learning time.

[0036] A sufficient number of iterations of learning using the training data are used to obtain network parameters that reduce the output value of the objective function. In other words, a trained CNN model is generated that operates on the input image data of the training data and outputs predicted image data similar to the correct image data. The trained model thus generated can also be used as a converter for input image data other than the training data, converting the input image data into data similar to the correct image data.

[0037] In this embodiment, a trained model is generated in advance by learning using the following training data, and the generated trained model is stored in the image processing device 100. The input image data included in the training data is halftone image data, and the correct image data is read data obtained by reading, by the reading device 105, a recording medium on which an image has been formed by the image forming device 104 based on the halftone image data. In this case, the trained model is a model that converts a CMYK 4-channel binary image into an RGB 3-channel 8-bit image.

[0038] The trained model can generate reference data that serves as a standard for inspection by using, as input, halftone image data generated by the halftone processing unit 304 based on data of any image input to the image forming device 104 as data of the input image. Note that the input image data is not limited to CMYK 4-channel binary halftone image data, and may be, for example, CMYK 4-channel 8-bit image data. The same applies to the output image data. The trained model may be provided in a form in which the quality is guaranteed after being sufficiently verified by the manufacturer of the image processing device 100 or the image processing system 1, for example.

[0039] <Functional configuration of image processing device> The functional configuration of the image processing device 100 will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its functional configuration, a prediction unit 403, an adjustment unit 404, an inspection unit 405, a reading unit 406, a printing unit 410, a determination unit 420, and an update unit 430. Terminals 401, 402, and 407 are input / output terminals. The prediction unit 403 inputs halftone image data acquired via the terminal 401 to the above-mentioned trained model, and acquires data of a predicted image output from the trained model. The data of the predicted image is, for example, RGB 3-channel 8-bit image data.

[0040] The adjustment unit 404 adjusts pixel values ​​of the predicted image data acquired by the prediction unit 403 to generate reference image data (hereinafter referred to as "reference data"). In this embodiment, the adjustment process in the adjustment unit 404 refers to so-called gamma correction processing, in which pixel values ​​are converted into new pixel values ​​for each RGB channel. The reference data generated by the adjustment unit 404 is transmitted to the inspection unit 405. When it becomes necessary to generate new reference data, a signal indicating this and CMYK four-channel binary halftone image data are input to the terminal 401 as input image data. When there is an input to the terminal 401, the prediction unit 403 and adjustment unit 404 perform the above-mentioned processing.

[0041] The printing unit 410 controls the image forming apparatus 104 to cause the image forming apparatus 104 to output a calibration chart and a learning chart, etc., described later, as printed matter. Details of the processing by the printing unit 410 will be described later. The reading unit 406 controls the reading apparatus 105 to obtain read data, and transmits the obtained read data to the inspection unit 405. A signal is input to the terminal 402 in synchronization with image formation in the image forming apparatus 104, and the reading unit 406 performs the above-mentioned processing in response to the signal.

[0042] The inspection unit 405 compares the reference data with the read data to inspect the presence or absence of defects, scratches, etc. in the printed matter corresponding to the read data. For example, the inspection unit 405 first performs alignment to correct misalignment or tilt of the image between the reference data and the read data. Next, the inspection unit 405 takes the difference between the reference data after alignment and the read data, and performs inspection by determining whether or not there is a difference equal to or greater than a predetermined threshold value that indicates defects, scratches, etc., by filter processing or the like. The inspection unit 405 outputs a control signal indicating the inspection result via the terminal 407. Depending on the output control signal, control is performed such as changing the output location of the printed matter depending on whether the inspection is passed or failed, counting the number of passes or fails, and notifying the user as necessary. This allows the delivery of the printed matter to be performed efficiently.

[0043] The determination unit 420 determines whether to update the learned model or to update only the correction parameters (hereinafter referred to as "adjustment parameters") for gamma correction in the adjustment unit 404 without updating the learned model, based on the reference data and the read data. The details of the processing by the determination unit 420 will be described later. The update unit 430 updates at least one of the learned model and the adjustment parameters, based on the determination result by the determination unit 420. The details of the processing by the update unit 430 will be described later.

[0044] <Updating trained models and tuning parameters> In general, due to physical factors such as the members constituting the image forming apparatus 104, or environmental factors such as the temperature or humidity around the image forming apparatus, the image formed on the recording medium may have a variation in color development or color density. Hereinafter, such a variation in color development or color density will be referred to as "color variation" and described. For example, in image formation by the inkjet method, color variation may occur in the image formed on the recording medium even when the same image is formed due to aging of the piezoelectric element or heater that controls the ink discharge, or changes in temperature or humidity. The same is true in image formation by other methods such as electrophotography. Such color variation, within a range that is not visually problematic, hinders appropriate inspection of the presence or absence of the above-mentioned defects or scratches. The image processing device 100 according to this embodiment generates reference data that follows the color variation. As a result, the image processing device 100 can perform appropriate inspection.

[0045] FIG. 5 is a block diagram showing an example of the internal functional configuration of the printing unit 410 and the updating unit 430 according to the first embodiment. The printing unit 410 has a calibration chart output control unit 502 and a learning chart output control unit 504, and the updating unit 430 has an adjustment update unit 506 and a model update unit 505. Terminals 501 and 507 are input / output terminals. The calibration chart output control unit 502 starts controlling the image forming apparatus 104 using a control signal input via the terminal 501 as a trigger, and causes the image forming apparatus 104 to output a calibration chart. Specifically, the calibration chart output control unit 502 transmits image data (hereinafter referred to as "calibration image data") for causing the image forming apparatus 104 to output the calibration chart to the image forming apparatus 104. As a result, the image forming apparatus 104 outputs the calibration chart. The calibration chart output control unit 502 also transmits a signal (hereinafter referred to as "calibration chart output signal") to the reading unit 406 indicating that the image forming apparatus 104 is being caused to output the calibration chart. Furthermore, the proofreading chart output control unit 502 transmits proofreading halftone image data corresponding to the proofreading image data to the determination unit 420 .

[0046] Here, the control signal received by the calibration chart output control unit 502 via the terminal 501 is a control signal indicating the start of execution of a process for determining whether or not it is necessary to update the learned model or adjustment parameters (hereinafter referred to as a "calibration process"). The calibration process is executed at a predetermined timing, such as when an instruction to execute the calibration process is received from a user, before the start of image formation on a printed matter to be delivered, or during image formation on multiple printed matters.

[0047] The reading unit 406 controls the reading device 105 to read the calibration chart based on the calibration chart output signal received from the calibration chart output control unit 502. The reading unit 406 acquires the read data obtained by the reading as calibration read data. The method of acquiring the calibration read data in the reading unit 406 is the same as the method of acquiring read data corresponding to the printed matter to be delivered.

[0048] The determination unit 420 transmits the calibration halftone image data received from the calibration chart output control unit 502 to the prediction unit 403. The prediction unit 403 inputs the calibration halftone image data received from the determination unit 420 to a learned model, and acquires data of a predicted image (hereinafter referred to as a "calibration predicted image") corresponding to the calibration read data generated by the learned model as calibration reference data. The determination unit 420 determines whether to update the learned model used by the prediction unit 403 or to update the adjustment parameters used by the adjustment unit 404, based on the calibration read data acquired by the reading unit 406 and the calibration reference data acquired by the prediction unit 403.

[0049] Here, the determination of whether to update the adjustment parameters means determining whether to update only the adjustment parameters without updating the trained model. Moreover, the update of the trained model means updating the network parameters in the trained model. In this embodiment, the adjustment parameters used by the adjustment unit 404 are described as parameters indicating one-dimensional gamma curves of each of the three RGB channels.

[0050] When the judgment result of the judgment unit 420 is an update of the learned model, the learning chart output control unit 504 receives a signal indicating that from the judgment unit 420 and controls the image forming device 104 to output the learning chart from the image forming device 104. Specifically, the learning chart output control unit 504 transmits image data (hereinafter referred to as "learning image data") for causing the image forming device 104 to output the learning chart to the image forming device 104. As a result, the image forming device 104 outputs the learning chart. In addition, the learning chart output control unit 504 transmits halftone image data based on the learning image data (hereinafter referred to as "learning halftone image data") to the model update unit 505. Furthermore, the learning chart output control unit 504 transmits a signal indicating that the learning chart has been output to the reading unit 406. Based on the signal, the reading unit 406 controls the reading device 105 to cause the reading device 105 to read the learning chart, and obtains read data corresponding to the learning image data (hereinafter referred to as "learning read data"). The method of obtaining the learning read data in the reading unit 406 is the same as the method of obtaining read data corresponding to the printed matter to be delivered.

[0051] The model update unit 505 updates the learned model based on the learning halftone image data received from the learning chart output control unit 504 and the learning read data acquired by the reading unit 406. When the determination result of the determination unit 420 is that the adjustment parameters should be updated, the adjustment update unit 506 receives a signal indicating this from the determination unit 420 and updates the adjustment parameters. When the model update unit 505 and the adjustment update unit 506 have completed the update, they output a signal indicating this via a terminal 507.

[0052] <Processing flow of image processing device> FIG. 6 is a flowchart showing an example of a processing flow of the calibration process in the image processing device 100 according to the first embodiment. The CPU 101 reads a program for executing the process of the flowchart shown in FIG. 6 from the storage device 103 or the like, and executes the program to realize the functions of the above-mentioned respective units. In the following description, the symbol "S" means a step (process). The process of the flowchart shown in FIG. 6 is started at a predetermined timing such as when the image processing device 100 receives an instruction to execute the calibration process from a user, before the start of image formation on a printed matter to be delivered, or during image formation on a plurality of printed matters. First, in S600, the calibration chart output control unit 502 receives a control signal input via the terminal 501. Next, in S601, the calibration chart output control unit 502 controls the image forming device 104 to output a calibration chart.

[0053] FIG. 7 is a diagram showing an example of the calibration charts 701 and 707 according to the first embodiment. The calibration chart 701 shown in FIG. 7(a) includes a one-dimensional characteristic acquisition area 702 and a color mixing characteristic acquisition area 703. In each of the rows 704 to 706 in the one-dimensional characteristic acquisition area 702, patches of the same hue, such as red (R), are arranged in order from light to dark from left to right in FIG. 7(a). In the following description, it is assumed that the row 704 has a red (R) patch arranged therein, the row 705 has a green (G) patch arranged therein, and the row 706 has a blue (B) patch arranged therein. In the following description, the multiple patches arranged in the row 704 are referred to as one-dimensional characteristic patches of red (R), the multiple patches arranged in the row 705 as one-dimensional characteristic patches of green (G), and the multiple patches arranged in the row 706 as one-dimensional characteristic patches of blue (B).

[0054] Further, a plurality of mixed-color patches are arranged in the mixed-color characteristic acquisition area 703. The mixed-color patches are patches of a different hue from the one-dimensional characteristic patches arranged in the one-dimensional characteristic acquisition area 702. Specifically, the one-dimensional characteristic patches are red, green, or blue, and therefore have a pixel value in which each pixel value of R, G, or B is dominant, whereas the mixed-color patches are patches with a different RGB pixel value ratio from any of them. In other words, the mixed-color patches are patches in which the ratio of each ink used to express the patch is different. Examples of patch colors of the mixed-color patches include orange, dark brown, and purple. The mixed-color characteristic acquisition area 703 does not necessarily need to have a plurality of mixed-color patches covering all mixed-color combinations arranged therein, but it is preferable that a plurality of mixed-color patches that represent each color area, such as hue or brightness, are arranged for each certain range of the color area.

[0055] The calibration chart is not limited to the patch arrangement of the calibration chart 701 shown in Fig. 7(a). For example, the calibration chart may include horizontally elongated patches whose patch width corresponds to the printing width and from which the characteristics of each nozzle of each of the recording heads 201 to 204 can be known, like the calibration chart 707 shown in Fig. 7(b).

[0056] After S601, in S602, the reading unit 406 controls the reading device 105 to obtain calibration read data. Next, in S603, the prediction unit 403 inputs the calibration halftone image data into the learned model to obtain data of the calibration predicted image as calibration reference data. Next, in S604, the determination unit 420 determines the method of calibration processing, i.e., whether it is necessary to update the learned model, or whether it is not necessary to update the learned model and only the adjustment parameters are updated, based on the calibration read data and the calibration reference data.

[0057] <Processing flow of the judgment unit> With reference to FIG. 8, a process flow of the determination process of the determination unit 420 in S604 will be described. FIG. 8 is a flowchart showing an example of a process flow of the determination process in the determination unit 420 according to the first embodiment, and is a flowchart showing an example of the process flow in S604 shown in FIG. 6. First, in S801, the determination unit 420 acquires one-dimensional characteristics in each channel of RGB (hereinafter referred to as "one-dimensional characteristics of the calibration read data") from an area corresponding to a one-dimensional characteristic patch in the calibration read data acquired in S602. A method of acquiring the one-dimensional characteristics will be described in detail with reference to FIG. 7(a). From a calibration read image (RGB 3-channel 8-bit image) indicated by the calibration read data, pixel values ​​of an area corresponding to each patch (hereinafter referred to as "patch area") are acquired, and this is set as the one-dimensional characteristic. The one-dimensional characteristic is acquired as, for example, one vector having each component value of RGB as a component by calculating a statistical value such as an average value, a median value, or a mode value of values ​​of a plurality of pixels included in each patch area. Here, the one-dimensional characteristic means the characteristic of pixel values ​​for different brightness (luminance) in the same hue (channel).

[0058] Next, in S802, the determination unit 420 obtains the one-dimensional characteristic of the calibration reference data obtained in S603, similarly to S801. Next, in S803, the determination unit 420 obtains a one-dimensional adjustment curve (one-dimensional gamma curve) for bringing the calibration reference data closer to the calibration read data. Specifically, the determination unit 420 obtains the one-dimensional adjustment curve by comparing the one-dimensional characteristic of the calibration read data obtained in S801 with the one-dimensional characteristic of the calibration reference data obtained in S802.

[0059] A method for acquiring a one-dimensional adjustment curve will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of one-dimensional characteristics of calibration read data and calibration reference data according to the first embodiment. In FIG. 9, the horizontal axis (X axis) represents pixel values ​​of the calibration reference data, and the vertical axis (Y axis) represents pixel values ​​of the calibration read data. In the following, as an example, FIG. 9 will be described assuming that pixel values ​​of a red (R) channel are shown. In the processes of S801 and S802, pixel values ​​of areas (patch areas) corresponding to each patch included in the one-dimensional characteristic patch of red (R) in row 704 of each of the calibration reference data and calibration read data are obtained. Each of points 902 to 906 shown in FIG. 9 is a plot of pixel values ​​in each patch area. The pixel values ​​acquired from each patch area corresponding to row 704 shown in FIG. 7(a) are plotted.

[0060] A dashed straight line 901 indicates the case where Y=X, that is, the pixel values ​​of the calibration reference data and the calibration read data are equal to each other. The straight line 901 indicates a state in which the trained model generates calibration reference data that correctly simulates the calibration read data, and indicates characteristics in the case where no color variation has occurred since the calibration reference data was previously generated. A dashed-dotted curve 907 is an approximation curve obtained based on points 902 to 906, and is an example of a one-dimensional adjustment curve for the R channel obtained based on the calibration reference data and the calibration read data acquired by the image processing device 100.

[0061] For example, assuming that the X-coordinate of the point 902 is X(902) and the Y-axis coordinate is Y(902), the pixel value X(902) of the calibration reference data can be converted to the pixel value Y(902) of the calibration read data to bring the colors of the two closer together. The parameters used for this conversion are the adjustment parameters of the adjustment unit 404. The adjustment parameters may be obtained by substituting numerical values ​​into a curve 907, which is a one-dimensional adjustment curve approximated by an arbitrary mathematical expression such as a polynomial, during the adjustment process. The adjustment unit 404 may also hold the coordinates of the points 902 to 906, and the adjustment parameters may be obtained by interpolating between the coordinates, such as linear interpolation, during the adjustment process. In S803, the determination unit 420 acquires one-dimensional adjustment curves for each channel of RGB.

[0062] Next, in S804, the determination unit 420 refers to the calibration scan data again and determines whether the pixel values ​​of the areas corresponding to each mixed-color patch included in the color mixing characteristic acquisition area 703 in the calibration scan data are consistent with the one-dimensional adjustment curve obtained in S803. Specifically, for example, the determination unit 420 plots each of the RGB pixel values ​​of the areas corresponding to each mixed-color patch on the diagram of the corresponding RGB one-dimensional adjustment curve shown as an example in FIG. 9. For example, assume that the point where the R pixel value of a certain mixed-color patch is plotted is point 908. At this time, point 908 is in the area on the opposite side of the one-dimensional adjustment curve across Y=X. That is, the directions of deviation between the one-dimensional adjustment curve and the position where the R pixel value of the mixed color is plotted are opposite to each other, and the position where the R pixel value of the mixed color is plotted is far away from the one-dimensional adjustment curve. In such a case, the determination unit 420 determines that the pixel values ​​of the areas corresponding to the mixed-color patches are not consistent with the one-dimensional adjustment curve. Conversely, if the position where the pixel values ​​of the area corresponding to the mixed color patch are plotted is on the same side for all RGB channels and is near the one-dimensional adjustment curve, the determination unit 420 determines that the pixel values ​​of the area corresponding to the mixed color patch are consistent with the one-dimensional adjustment curve.

[0063] A curve 907 shown as an example in FIG. 9 indicates that the pixel value of the R channel should be corrected in the direction of increasing in order to bring the calibration reference data closer to the calibration read data. On the other hand, a point 908 indicates that the pixel value of the R channel should be corrected in the direction of decreasing. Such conflicting characteristics mean that the one-dimensional adjustment curve cannot correct color fluctuations in the color region due to the complex action of color mixing. As a specific determination method, for example, the determination unit 420 determines whether or not the region including the one-dimensional adjustment curve and the region including the position where the pixel value of the mixed color is plotted coincide in two regions sandwiching the straight line Y=X. The determination unit 420 may determine whether or not the distance from the position where the pixel value of the mixed color is plotted to the one-dimensional adjustment curve is within a predetermined threshold value. The determination unit 420 performs the above-mentioned determination in each channel of RGB for the pixel values ​​of the region corresponding to each mixed color patch. If there is even one channel that does not match the one-dimensional adjustment curve for each mixed-color patch, the determining unit 420 determines that the pixel values ​​of the area corresponding to the mixed-color patch do not match the one-dimensional adjustment curve.

[0064] Next, in S805, the determination unit 420 determines whether the pixel values ​​of the areas corresponding to all the mixed-color patches are consistent with the one-dimensional adjustment curve. If it is determined in S804 that the pixel values ​​of the areas corresponding to all the mixed-color patches are consistent with the one-dimensional adjustment curve, in S806, the determination unit 420 determines that the color variation can be calibrated only by updating the adjustment parameters without updating the learned model. If it is determined in S804 that the pixel values ​​of the corresponding areas of even one of the multiple mixed-color patches are not consistent with the one-dimensional adjustment curve, in S807, the determination unit 420 determines that the learned model needs to be updated to calibrate the color variation. The determination unit 420 regards the determination result by the process of S806 or S807 as the calibration process method obtained as the result of the determination in S604. After the process of S806 or S807, the determination unit 420 ends the process of the flowchart shown in FIG. 8.

[0065] The effect of the process of S604 will be described. The trained model has fully learned the characteristics of the image forming device 104. Therefore, if there is no variation (color variation) in the characteristics, the pixel values ​​of the predicted image generated by the trained model and the pixel values ​​of the read image are expected to be very close to each other, that is, on the straight line Y=X shown in FIG. 9. However, as described above, due to color variation in the image forming device 104, the color of the printed matter, that is, the pixel values ​​of the read image, may differ from the pixel values ​​of the predicted image. Since such color variation is caused by physical or environmental factors, it is difficult to suppress the occurrence itself. In addition, since the predicted image generated by the trained model imitates the color characteristics of the image forming device 104 at the time of learning, the trained model cannot imitate the color characteristics of the image forming device 104 for color variations that occur after learning is completed.

[0066] In particular, when color variations occur not locally but over the entire printed matter, they are not visually noticeable unless the variations are excessive, and are therefore often accepted as printed matter that meets a certain level of quality. However, even when such color variations occur to an acceptable degree, the difference in pixel values ​​between the reference data and the read data is taken into account when comparing the data, and therefore the printed matter may be over-detected as defective when the color variations occur to an acceptable degree. In addition, if the inspection threshold is changed to allow the printed matter when the color variations occur in order to prevent such over-detection, defects that should have been detected as defects may be overlooked.

[0067] As a method for suppressing the above-mentioned overdetection, a method of updating the trained model used by the prediction unit 403 to a trained model that matches the current color variation can be considered. Specifically, new or additional learning is performed using scanned data corresponding to a printed matter in a state in which color variation has occurred as correct answer data, and the trained model obtained by this learning (hereinafter referred to as the "updated trained model") is updated as the trained model used by the prediction unit 403. A predicted image generated by the updated trained model mimics the color characteristics of the image forming device 104 in which color variation has occurred, and therefore a predicted image that tracks the color variation can be generated.

[0068] However, updating a trained model involves problems such as risks or disadvantages to users. One of the problems is that the results of training cannot be fully guaranteed when a trained model is newly generated. In general, it is not easy to predict the details of a trained model to be generated before training and to completely control the results of training. As described above, it is possible to provide a trained model that is held in advance by the image processing device 100 and whose quality is guaranteed after sufficient verification by the manufacturer of the image processing device 100. However, the same quality cannot necessarily be guaranteed for a trained model that is arbitrarily generated in the usage environment of an end user. In other words, updating a trained model may cause quality degradation in the form of artifacts in a predicted image, etc. Another problem is that in order to perform new or additional training, time for training and printing, etc. are required to obtain read data as correct answer data for the training data. Such consumption of time and resources is costly for the user.

[0069] Therefore, in this embodiment, the image processing device 100 is provided with an adjustment unit 404 that performs one-dimensional gamma correction on the predicted image for each channel in order to adjust the predicted image in accordance with this color variation and generate reference data. The image processing device 100 generates reference image data (reference data) in which the predicted image is made to follow the color variation through adjustment by the adjustment unit 404. This allows the image processing device 100 to generate reference data corresponding to the color variation of the image processing device 100 without taking the risk of quality deterioration associated with updating a trained model and without paying the cost associated with new or additional learning.

[0070] However, not all color variations can be handled by adjustment by the adjustment unit 404 alone. In particular, in mixed colors in which a color is expressed by mixing inks of multiple colors, the color development characteristics are complex. Therefore, even if the characteristics are corrected independently for each channel, not all of the mixed color developments will necessarily match the one-dimensional adjustment curve (curve 907) shown as an example in FIG. 9. In this way, when color variations of mixed colors that cannot be corrected by adjustment by the adjustment unit 404 alone occur and reference data that takes into account the color variations needs to be generated, the above-mentioned trained model needs to be updated.

[0071] Therefore, in this embodiment, the image processing device 100 is provided with a determination unit 420 that determines whether the application of the one-dimensional adjustment curve can sufficiently deal with the color variation or whether updating of the trained model is necessary. According to such an image processing device 100, the determination unit 420 can determine the degree or characteristics of the color variation of the color mixture, and it becomes possible to determine whether updating of the trained model that may cause risks or disadvantages to the user is necessary.

[0072] After S604, in S605, the determination unit 420 determines whether or not it has been determined in S604 that the learned model needs to be updated. If it has been determined in S604 that the learned model does not need to be updated, that is, that only the adjustment parameters are updated, in S610, the adjustment update unit 506 updates the adjustment parameters used by the adjustment unit 404 during adjustment. Specifically, in this case, the adjustment update unit 506 updates the adjustment parameters by rewriting the adjustment parameters used by the adjustment unit 404 using the one-dimensional adjustment curve acquired in S803. If it has been determined in S604 that the learned model needs to be updated, in S606, the learning chart output control unit 504 controls the image forming device 104 to output a learning chart from the image forming device 104. After S606, in S607, the reading unit 406 controls the reading device 105 to read the learning chart output in S606, and acquires learning read data.

[0073] If the learning chart is the same as the calibration chart, the calibration read data acquired in S602 can be used as the learning read data. In this case, the processes of S606 and S607 can be omitted. However, since the learning data is used for new or additional learning and is an important factor that determines the quality of the learned model, it is preferable that the learning data is not only a patch but also a chart that is less biased than the calibration chart. Conversely, if a chart that is also suitable for the learning chart is always output as the calibration chart by the image forming device 104, the amount of wasted paper increases. From this point of view, it is also preferable that the calibration chart and the learning chart are different charts.

[0074] The aspects of the learning chart will be described. In generating a new trained model including additional learning for an existing trained model, the learning data used for the new or additional learning is important because it affects the quality of the trained model to be generated. In order to widely handle all input images, it is generally desirable to include data of many colors and many types of images in advance in the learning data without being biased toward a specific color or type of image such as a patch image. On the other hand, in this embodiment, a new trained model is generated when there is a specific color that cannot be corrected only by the adjustment process in the adjustment unit 404. Therefore, it is important that the color variation of the specific color is reproduced by generating a new trained model. Therefore, the image processing device 100 may control the output of the learning chart so that the learning data used in generating a new trained model includes more learning data of the specific color.

[0075] More specifically, for example, the image processing device 100 performs the following process. The learning chart output control unit 504 holds learning image data (hereinafter referred to as "basic learning image data") corresponding to a basic learning chart in advance. Here, in the process of determining the consistency between the pixel value and the one-dimensional adjustment curve in S804, the determination unit 420 can obtain information on colors that cannot be corrected by the adjustment process by the adjustment unit 404 alone. The learning chart output control unit 504 receives the information from the determination unit 420, processes the basic learning image data so as to increase the colors that cannot be corrected by the adjustment process alone and the colors similar to the colors, and adds new learning image data or changes the basic learning image data. Furthermore, the learning chart output control unit 504 causes the image forming device 104 to output a learning chart using the learning image data obtained by adding or changing.

[0076] By adding such read data of the learning chart (learning read data) as learning data, more efficient learning can be performed when generating a new trained model. When adding or changing learning data corresponding to a color that cannot be corrected by the adjustment process alone, the user may be allowed to instruct or control which of the multiple colors that cannot be corrected by the adjustment process alone should have learning data added or changed. In this case, the user may be allowed to instruct or control how much learning data corresponding to the color that cannot be corrected by the adjustment process alone should be increased.

[0077] After S607, in S608, the model update unit 505 generates a new trained model by learning using the training halftone image data and the training read data acquired in S607 as training data, and updates the trained model to be used by the prediction unit 403. After S608, in S609, the adjustment update unit 506 updates the adjustment parameters used by the adjustment unit 404 during adjustment. Specifically, the adjustment update unit 506 updates, as the adjustment parameters, the one-dimensional adjustment curve used by the adjustment unit 404 during adjustment.

[0078] Since the predicted image generated by the trained model updated by the process of S608 is a learned color reproduction in a state where color variation occurs, it is preferable to change the one-dimensional adjustment curve, which is an adjustment parameter, so as to be consistent with the trained model after the update. Specifically, the adjustment update unit 506 changes the one-dimensional adjustment curve to a straight line equivalent to a straight line (straight line 901 shown in FIG. 9) that does not change the pixel value, that is, resets it. The adjustment update unit 506 may update the adjustment parameter as follows. Specifically, first, the adjustment update unit 506 instructs the image forming device 104 after updating the trained model to execute the same processes as S802 and S803, and acquires the one-dimensional adjustment curve again. Next, the adjustment update unit 506 calculates new adjustment parameters for adjusting the predicted image generated by the trained model after the update, and updates the adjustment parameters. After S609 or S610, the image processing device 100 ends the process of the flowchart shown in FIG. 6.

[0079] After the flow chart is completed, the reference data generated by the image processing device 100 will follow the current color variation. Therefore, the image processing device 100 can suppress overdetection of defects as described above in the inspection of printed matter. In addition, the image processing device 100 is configured to update only the adjustment parameters if the color variation can be accommodated by updating the adjustment parameters, and to update the trained model only when it is necessary to update the trained model. The image processing device 100 configured in this way can reduce the number of times the trained model is regenerated or updated while responding to color variation. As a result, it is possible to reduce the risk of carelessly changing an existing trained model, the time required for new or additional learning, or the use of a recording medium and ink or toner used to output a learning chart.

[0080] [Embodiment 2] With reference to Figs. 10 and 11, an image processing device 100 according to the second embodiment will be described. The image processing device 100 according to the first embodiment automatically determines the method of calibration processing based on the calibration reference data and the calibration read data. Here, the determination of the calibration processing method means determining whether it is necessary to update the learned model, or whether it is not necessary to update the learned model and only the adjustment parameters are updated. In contrast, the image processing device 100 according to the second embodiment (hereinafter simply referred to as "image processing device 100") presents the result of the determination of the calibration processing method to the user, and leaves it to the user to decide the timing or method of the calibration processing. In particular, the image processing device 100 leaves it to the user to decide the timing or method of updating the learned model. Note that in the second embodiment, the description of the contents common to the first embodiment will be omitted, and the description will be centered on the differences from the first embodiment.

[0081] 10 is a block diagram showing an example of a functional configuration of an image processing device 100 according to the second embodiment. The image processing device 100 has a presentation unit 1001 in addition to the functional configuration of the image processing device 100 according to the first embodiment. The presentation unit 1001 receives information indicating a judgment result of a method of proofreading processing from the judgment unit 420 (hereinafter referred to as "proofreading judgment information"), and performs output control for presenting the proofreading judgment information to a user. In addition, the presentation unit 1001 can receive and use data of a predicted image obtained by the judgment unit 420 from the prediction unit 403 from the judgment unit 420 in order to present the proofreading judgment information.

[0082] Fig. 11 is a flowchart showing an example of a process flow of a calibration process in the image processing device 100 according to the second embodiment. In the description of the process flow, the same processes as those in Fig. 6 are denoted by the same reference numerals and description thereof will be omitted. First, the image processing device 100 executes processes from S601 to S603. After S603, in S1101, the determination unit 420 determines a method of calibration process based on the calibration scan data and the calibration reference data. The process of S1101 is similar to the process of S604, but while the process of S604 determines a method of calibration process to be performed, the process of S1101 merely determines a method of calibration process required to adequately deal with the occurring color variation.

[0083] Next, in S1102, the determination unit 420 determines whether or not it has been determined in S1101 that the learned model needs to be updated. Specifically, the determination unit 420 determines whether or not it has been determined that the learned model needs to be updated to adequately respond to the occurring color variation. If it has been determined in S1101 that the learned model does not need to be updated to respond to the currently occurring color variation, that is, if it has been determined that only the adjustment parameters used by the adjustment unit 404 during arbitration are sufficient to be updated, the image processing device 100 executes the process of S610. If it has been determined in S1101 that the currently occurring color variation cannot be responded to by only updating the adjustment parameters, and the learned model needs to be updated, the image processing device 100 executes the process of S1103. Specifically, in this case, in S1103, the presentation unit 1001 performs control to present to the user that the learned model needs to be updated to adequately respond to the currently occurring color variation.

[0084] For example, the presentation unit 1001 generates a display image indicating this, and controls to output a signal indicating the generated display image to the output device 109. As a result, the display image is displayed on the output device 109, and this can be visually presented to the user. For example, the presentation unit 1001 generates a display image in which a message such as "Density change has occurred in the print result. To perform an inspection that takes this density change into consideration with high accuracy, please regenerate the trained model. You can also continue the inspection with simple adjustment without regeneration" is displayed on the output device 109. The method of presenting to the user is not limited to display, and may be audio or the like. After S1103, the image processing device 100 executes the process of S610. After S610, the image processing device 100 ends the process of the flowchart shown in FIG. 11.

[0085] For example, based on the message presented in S1103, the user can arbitrarily determine the timing to update the trained model and instruct the image processing device 100 to update the trained model. Based on the instruction to update the trained model from the user, the image processing device 100 performs the processes from S606 to S609, updates the trained model, and then regenerates the reference data.

[0086] According to the image processing device 100 configured as above, the user can update the trained model at a desired timing after comprehensively considering the operating status of the image forming device 104, the risks associated with updating the trained model, the costs involved in the update, etc. As a result, the proofreading required for proper inspection of printed matter can be performed in a manner more in line with the user's intentions.

[0087] [Modification 1 of the second embodiment] An image processing device according to Modification 1 of the second embodiment (hereinafter, simply referred to as "Modification 1") will be described with reference to Figs. 12 and 13. The functional configuration of the image processing device according to Modification 1 is the same as that of the image processing device according to the second embodiment shown as an example in Fig. 10. Therefore, hereinafter, the image processing device according to Modification 1 will be referred to as "image processing device 100", and each functional configuration of the image processing device 100 will be described with the reference numerals shown in Fig. 10. The image processing device 100 according to the second embodiment performs control to notify the user that the trained model needs to be updated when it is determined that the trained model needs to be updated. In contrast, the image processing device 100 according to Modification 1 performs control to notify the user that the trained model needs to be updated, as well as related information on color variation obtained based on the read data.

[0088] FIG. 12 is a flowchart showing an example of a process flow of a calibration process in the image processing device 100 according to the first modification. In the description of the process flow, the same processes as those in FIG. 11 are assigned the same reference numerals and the description is omitted. In the flowchart shown in FIG. 12, the process of S1103 is replaced with the process of S1201 compared to the flowchart shown in FIG. 11. That is, in S1102, the determination unit 420 determines whether or not it was determined in S1101 that the learned model needs to be updated, and if it was determined in S1101 that the learned model needs to be updated, the image processing device 100 executes the process of S1201. In S1201, the presentation unit 1001 performs control to present to the user the fact that the learned model needs to be updated and related information regarding color variation.

[0089] FIG. 13 is a diagram showing an example of a display screen 1300 on which the presenting unit 1001 according to the first modification presents related information on color variation to a user via the output device 109. In the display screen 1300, an area 1301 displays an image obtained after the adjustment unit 404 adjusts a predicted image generated by the prediction unit 403 based on data of an input image using adjustment parameters before the update (hereinafter referred to as an "adjusted image"). An area 1302 displays an adjusted image obtained after the adjustment unit 404 adjusts the predicted image using adjustment parameters after the update. An area 1303 displays a read image obtained by the reading device 105 reading a printout formed on a recording medium by the image forming device 104. Note that if data of the read image is not acquired, the display of the read image in the area 1303 is omitted. In order to display the read image, the image processing device 100 may, during processing of S601 and S602, cause the image forming device 104 to print a small number of copies, such as one copy, of the input image and cause the reading device 105 to read the printed copy.

[0090] In the area 1304, information on colors that cannot be handled by updating the adjustment parameters alone (hereinafter referred to as "unsupportable colors") is displayed as related information on color variation. Specifically, for example, the unsupportable colors are displayed as patch-shaped previews in the left column of the area 1304. Also, for example, the pixel values ​​of the unsupportable colors are displayed in the right column of the area 1304. The information displayed in the right column of the area 1304 is not limited to pixel values. For example, the right column of the area 1304 may display information indicating the proportion of the unsupportable colors, or the unsupportable colors and their similar colors in the input image. For example, a scroll bar is displayed at the right end of the area 1304, and the user can check related information on the desired unsupportable colors by changing the position of the scroll bar. Information on the unsupportable colors is obtained by identifying mixed colors whose pixel values ​​do not match the one-dimensional adjustment curve in the process of checking the match between the pixel values ​​of the mixed colors and the one-dimensional adjustment curve in S804.

[0091] The display order of the unsupportable colors is arbitrary, but may be, for example, in descending order of the number of inconsistent channels, or in descending order of the difference between the pixel values ​​of the image after adjustment processing using the adjustment parameters on the predicted image for calibration and the pixel values ​​of the scanned image for calibration. Also, in the image displayed in the area 1302, the image area corresponding to the unsupportable colors may be highlighted. When the trained model is updated, a display of a predicted image generated by the trained model after the update may be added to the display screen 1300, or the predicted image may be displayed in the area 1302. Also, the information displayed in the area 1304 may be updated.

[0092] According to the image processing device 100 configured as above, the user can compare the adjusted image before updating the adjustment parameters with the adjusted image after updating to confirm how the adjusted image has changed due to the update of the adjustment parameters. In addition, the user can confirm to what extent the adjusted image after updating follows the currently occurring color fluctuation by comparing the adjusted image after updating the adjustment parameters with the read image. In addition, the user can confirm the position of the unsupportable color in the adjusted image by comparing the adjusted image before updating the adjustment parameters with the information on the unsupportable color. This allows the user to confirm, for example, whether the unsupportable color is an important color in the printed matter or how important it is to update the trained model. This allows the user to make a more appropriate decision on whether to update the trained model, taking into account the advantages and disadvantages of updating the trained model.

[0093] [Other embodiments] In the above embodiment, an example of the calibration chart has been described, but the calibration chart is not limited to the above. For example, in the above embodiment, patches of each channel (hue) of RGB are used as one-dimensional characteristic patches. This is because the color space in the adjustment process of the adjustment unit 404 is a space of three RGB channels, and therefore patches with good sensitivity in each channel are selected as one-dimensional characteristic patches of the calibration chart.

[0094] Therefore, the one-dimensional characteristic patch of the calibration chart may be of a hue different from RGB. For example, a plurality of patches having different brightness in mixed gray may be used as the one-dimensional characteristic patch, and the R pixel value, the G pixel value, and the B pixel value in the area (patch area) corresponding to each patch in the calibration read image may be acquired as the one-dimensional characteristic of the calibration read data. With such a calibration chart, the number of rows of the one-dimensional characteristic patch can be reduced from three rows of RGB to one row of gray. Furthermore, color variation occurs due to physical or environmental factors of the image forming device 104. Therefore, the calibration chart may be one in which a plurality of patches having different brightness in colors corresponding to the colors of inks (e.g., CMYK) used by the image forming device 104 when forming an image are used as the one-dimensional characteristic patch.

[0095] In addition, when the color of a one-dimensional characteristic patch is changed in the calibration chart, the color of the mixed-color patch may also change because the mixed-color patch in the calibration chart has a different hue and a different ink ratio from the one-dimensional characteristic patch. Whether a patch of a certain color (for example, R) exists as a one-dimensional characteristic patch or a mixed-color patch is not invariable in the patterns that the calibration chart can take.

[0096] In addition, the arrangement of the patches is not limited to the above. In the above, it has been stated that the patches may be long and strip-like. In addition, the calibration chart is not limited to a clearly divided area such as the one-dimensional characteristic acquisition area 702 and the mixed color characteristic acquisition area 703 in the calibration chart 701, and may be arranged randomly. By not clearly dividing the one-dimensional characteristic acquisition area 702 and the mixed color characteristic acquisition area 703, if there is a bias in the characteristics due to the position in the image formation and image reading, the influence of the bias can be reduced. If it is known where the one-dimensional characteristic patch and the mixed color patch are arranged on the calibration chart, the image processing device 100 can perform the same process as the above-mentioned embodiment.

[0097] The image processing device 100 according to the first embodiment determines the method of calibration processing based on the calibration read data, and automatically updates the adjustment parameters and the trained model based on the result of the determination. In addition, the image processing device 100 according to the second embodiment and the first modification, when the method of calibration processing determined based on the calibration read data is updating of the trained model, presents this to the user, and the user can decide the timing of updating the trained model, etc. However, the image processing device may simultaneously realize both of these forms.

[0098] Specifically, for example, the user inputs the conditions for updating the trained model into the image processing device in advance. The conditions include, for example, a case where a color variation occurs in a color specified by the user that cannot be adjusted by the adjustment process alone, a case where the magnitude of the pixel value deviation exceeds a tolerable threshold, or a case where the pixel value deviation continues to occur beyond a specified time. The image processing device holds information indicating the input conditions, and basically operates like the image processing device 100 according to the second embodiment and the first modified example. The image processing device 100 according to the second embodiment and the first modified example notifies the user in the process of S1103 or S1201 that the trained model needs to be updated, and generates a new trained model in response to the user's instruction. In contrast, the image processing device holding the information indicating the above-mentioned conditions generates a new trained model in the same way as when the user's instruction is given, when the conditions are satisfied. According to the image processing device configured in this way, the trained model can be quickly updated to reflect the user's intention, even if the user does not give an instruction on the spot.

[0099] The image processing device 100 according to the first embodiment automatically updates the trained model and the adjustment parameters. The image processing device 100 according to the second embodiment and the first modified example automatically updates the adjustment parameters. However, when updating the trained model or the adjustment parameters, it is possible to wait for a user's instruction to be input and perform the update based on the instruction. In particular, when the calibration process is started based on a user's instruction, it is considered that the user directly operates the image processing device 100, and therefore it is considered that the user is in a state where it is easy to determine whether or not to update the trained model or the adjustment parameters. By updating the trained model or the adjustment parameters based on a user's instruction, it is possible to prevent the image processing device 100 from behaving differently from the user's intention.

[0100] In addition, when the one-dimensional characteristic obtained by the process of S801 or S804 has a deviation larger than a predetermined size beyond the range of a printed matter expected as a good product, the following may be performed. In this case, for example, the image processing device 100 does not execute a calibration process for following the color variation, but presents a prompt to the user to perform output calibration of the image forming device 104. In addition, for example, the image processing device 100 automatically performs output calibration of the image forming device 104. Whether or not the one-dimensional characteristic obtained by the process of S801 or S804 exceeds the range of a printed matter expected as a good product is determined by, for example, comparing the one-dimensional characteristic with a predetermined threshold value. As a result, when a state deviation of the image forming device 104 that does not fall within the allowable color variation occurs, it is possible to make the image forming device 104 output an appropriate printed matter without forcibly following the reference data, and to perform an appropriate inspection on the printed matter.

[0101] The adjustment unit 404 according to the above embodiment performs one-dimensional gamma correction for each of the three RGB channels. However, the adjustment process in the adjustment unit 404 is not limited to one-dimensional gamma correction. For example, the following adjustment process may be used. For example, it is assumed that the final layer (Nth layer) of the trained model used by the prediction unit 403 has a structure that can be expressed by the following formula 3. (Formula 3) X N =tanh(X´ N-1 +b N )

[0102] Where X N is the output from this trained model, and is an RGB3 channel array that corresponds to the predicted image data. X´ N-1 is the feature map received from the previous layer (N-1th layer), X N-1 This represents the result of the convolution operation in this layer (Nth layer) for X' N-1 =ΣW N-1 *X N-1 At this point, the feature map has been converted into a 3-channel RGB format. Nis the bias of this layer (Nth layer), and has information that can be expressed by a three-dimensional vector whose components are the RGB component values. That is, Equation 3 is N-1 This means that the corresponding component in this vector is added to each element in the array. tanh() is the hyperbolic tangent function, and serves as the activation function for this layer (the Nth layer). Note that the output of tanh() ranges from -1 to +1, so the output from this trained model, X N Each element of must be linearly converted to a value from 0 to 255, which corresponds to the pixel value of an 8-bit image. However, this linear conversion is omitted in Equation 3.

[0103] In Equation 3, the bias b is expressed using a three-dimensional vector with one scalar value for each RGB channel as a component. N has the function of adjusting the output value of the entire range for each channel of RGB. That is, the bias b N can adjust the pixel values ​​of each channel of RGB. Compared to one-dimensional gamma correction, the bias b N Since it translates a curve defined by tanh, it is not possible to arbitrarily change each pixel value band, but it can be controlled by a small number of variables, namely, one scalar value for each channel.

[0104] That is, in this system, the following scalability exists. Specifically, there is scalability of variables and controls such as (1) bias that translates a specified curve with one variable, (2) gamma correction that draws an arbitrary adjustment curve for each channel in one dimension, and (3) a learned model that reproduces complex color development with a multi-layer filter and nonlinear calculation. Of these, in the above-mentioned embodiment, gamma correction and a learned model are used separately, but adjustment processing using a bias instead of gamma correction may be performed. In other words, the adjustment unit 404 may perform adjustment processing using this bias instead of one-dimensional gamma correction. Specifically, it is sufficient to select a bias value that is most suitable for the acquired one-dimensional characteristics. Also, based on a user's instruction such as "increase (or decrease) redness", the bias value of the corresponding channel may be increased or decreased.

[0105] As is obvious from the above example, the adjustment unit 404 does not necessarily need to be completely separated from the prediction unit 403, and may be included as part of the trained model, that is, included in the prediction unit 403. The adjustment unit 404 may perform the adjustment process using a trained model based on learning that is different from the trained model used by the prediction unit 403. Since the trained model has a drawback in that it is difficult to predict the results after updating, the adjustment unit 404 that performs the adjustment process using the adjustment parameters is provided as described above.

[0106] However, by limiting the trained model used by the adjustment unit 404 to one that does not attempt to imitate the characteristics of the entire image but is limited to one that follows color variations, the range of small-scale and simple effects can be limited. This makes it possible to reduce the cost of time and resources required to update the trained model used by the adjustment unit 404, as well as the risk of updating the trained model. Even in an image processing device having such an adjustment unit 404, it is possible to generate reference data that takes color variations into consideration while suppressing the risk of carelessly updating the trained model used by the prediction unit 403, the time required for training the trained model, and the output of a training chart.

[0107] The image processing device 100 according to the above-described embodiment updates the adjustment parameters used by the adjustment unit 404 and the learned model used by the prediction unit 403 so as to follow color variations occurring in the image forming device 104. However, the updates of the adjustment parameters and the learned model are not limited to those that follow color variations, and the image processing device 100 may update the adjustment parameters and the learned model in response to state variations occurring in the image forming device 104.

[0108] In the above embodiment, the color space in which the adjustment process is performed has been described as a space based on RGB three channels, but the color space in which the adjustment process is performed is not limited to this. For example, the color space in which the adjustment process is performed may be a space based on CMYK corresponding to the colors of each ink, or a perceptual color space such as Lab.

[0109] In the image processing device 100 according to the above embodiment, the adjustment unit 404 performs an adjustment process on the predicted image generated by the trained model to generate reference data. However, when comparing the reference data with the read data in the inspection process, an adjustment process corresponding to the reverse process of the adjustment process on the predicted image may be performed on the read data. Even with this configuration, it is possible to perform an inspection similar to that of the image processing device 100 according to the above embodiment. Specifically, in FIG. 4, instead of the adjustment unit 404 being disposed downstream of the prediction unit 403, the adjustment unit 404 may be disposed downstream of the reading unit 406.

[0110] However, when performing adjustment processing on a predicted image, the adjustment processing needs to be performed only when generating reference data, whereas when performing adjustment processing on read data, the adjustment processing needs to be performed for each read data, i.e., for each printed matter. Therefore, it is preferable that the adjustment unit 404 is disposed after the prediction unit 403 and performs adjustment processing on the predicted image. Also, from the viewpoint that performing color processing on the read data makes the purpose of inspecting the actual state of the printed matter unclear, it is better to perform adjustment processing on the predicted image. Similarly, the adjustment unit 404 may perform adjustment processing on the data of the input image to be input to the image forming device 104.

[0111] In the above embodiment, the image processing device 100 has been described as inspecting printed matter as an application example. However, the application of the image processing device 100 is not limited to this. For example, the image processing device 100 may generate preview image data used for preview display to confirm a prediction of a print result, instead of reference data. Furthermore, the image processing device 100 may generate control parameters, setting values, or some information for determining them used by the image forming device 104, instead of image data.

[0112] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0113] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0114] [Configuration of this disclosure] The present disclosure includes the following configurations.

[0115] <Configuration 1> A generating means for generating a predicted image which is an image obtained by predicting an output image corresponding to an input image using a trained model; an adjustment means for adjusting pixel values ​​of a target image using an adjustment parameter in order to reduce a difference between the output image and the predicted image; A determination means for determining whether to update the trained model or to update only the adjustment parameters without updating the trained model based on a difference between the output image and the predicted image; Having An image processing device comprising:

[0116] <Configuration 2> the output image is a read image obtained by reading a recording medium on which an image corresponding to the input image is formed, The determination means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model based on the read image as the output image and the predicted image. 2. The image processing device according to claim 1,

[0117] <Configuration 3> The determination means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model in order to reduce a difference between the output image and the predicted image due to a state change in image formation on a recording medium to a predetermined difference or less. 3. The image processing device according to configuration 1 or 2,

[0118] <Configuration 4> The target image is the predicted image. 4. The image processing device according to any one of configurations 1 to 3,

[0119] <Component 5> The adjustment means adjusts the pixel values ​​of the target image using the corresponding adjustment parameters for each channel in a color space of the data of the target image. 5. The image processing device according to any one of configurations 1 to 4,

[0120] <Component 6> The adjustment parameters can be updated without repeated learning. 6. The image processing device according to any one of configurations 1 to 5,

[0121] <Component 7> The output image is a read image obtained by reading a predetermined calibration chart, The determination means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model based on the read image and the predicted image. 7. The image processing device according to any one of configurations 1 to 6,

[0122] <Component 8> the determining means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model by comparing a gradation characteristic of a first patch group in the read image and a gradation characteristic of a second patch group having a hue different from that of the first patch group with a gradation characteristic of the first patch group and a gradation characteristic of the second patch group in the predicted image. 8. The image processing device according to configuration 7,

[0123] <Component 9> an update means for updating at least one of the trained model and the adjustment parameters based on a determination result by the determination means; Further possessing 9. The image processing device according to any one of configurations 1 to 8,

[0124] <Component 10> an output control means for performing output control for presenting a determination result by the determination means to a user; Further possessing 10. The image processing device according to any one of configurations 1 to 9,

[0125] <Component 11> A receiving means for receiving an instruction to update at least one of the trained model and the adjustment parameters from a user; an update means for updating at least one of the trained model and the adjustment parameters based on the instruction; 11. The image processing device according to configuration 10,

[0126] <Component 12> When the determination result is to update only the adjustment parameters without updating the trained model, The update means updates the adjustment parameters; When the determination result is to update the trained model, The output control means performs output control for presenting the determination result to a user, The update means updates the trained model based on the instruction. 12. The image processing device according to claim 11,

[0127] <Component 13> The output control means performs output control to present to a user, in addition to the determination result, color information that cannot be adjusted by only updating the adjustment parameters, which is a difference between the image adjusted by the adjustment means and the output image or the predicted image. 13. The image processing device according to claim 12,

[0128] <Component 14> The output control means performs output control to present to a user, in addition to the determination result, a position in the adjusted image or the output image of a color that cannot be adjusted by only updating the adjustment parameters due to a difference between the image adjusted by the adjustment means and the output image or the predicted image. 14. The image processing device according to configuration 12 or 13,

[0129] <Component 15> The trained model is updated using data of a scanned image obtained by scanning a training chart. 15. The image processing device according to any one of configurations 1 to 14,

[0130] <Component 16> The trained model is updated by using data of the read image obtained by reading the learning chart in which a ratio of colors that cannot be adjusted by only updating the adjustment parameters is increased, to obtain a difference between the image adjusted by the adjustment means and the output image or the predicted image. 16. The image processing device according to configuration 15,

[0131] <Component 17> After the trained model is updated, the tuning parameters are also updated. 17. The image processing device according to any one of configurations 1 to 16,

[0132] <Component 18> an inspection means for inspecting whether or not the output image satisfies a predetermined standard by comparing an image adjusted by the adjustment means with the output image or the predicted image; Further possessing 18. The image processing device according to any one of configurations 1 to 17,

[0133] <Component 19> a preview means for performing output control for presenting to a user the image adjusted by the adjustment means as a preview image of an image formed on a storage medium; Further possessing 18. The image processing device according to any one of configurations 1 to 17,

[0134] <Component 20> A generation step of generating a predicted image, which is an image obtained by predicting an output image corresponding to an input image, using the trained model; an adjustment step of adjusting pixel values ​​of a target image using an adjustment parameter in order to reduce a difference between the output image and the predicted image; a determination step of determining whether to update the trained model or to update only the adjustment parameters without updating the trained model based on a difference between the output image and the predicted image; Contains An image processing method comprising:

[0135] <Component 21> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 19. [Explanation of symbols]

[0136] 100 Image processing device 403 Prediction Department 404 Adjustment section 420 Judgment section

Claims

1. A generation means for generating a predicted image, which is an image obtained by predicting an output image corresponding to an input image, using the trained model; a determination means for determining whether to update the trained model or to update adjustment parameters for adjusting pixel values ​​of a target image without updating the trained model, based on the output image and the predicted image; Having An image processing device characterized by:

2. the output image is a read image obtained by reading a recording medium on which an image corresponding to the input image is formed, The determination means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model based on the read image as the output image and the predicted image.

2. The image processing device according to claim 1, wherein:

3. The determination means determines whether to update the trained model so that a difference between the output image and the predicted image due to a change in a state of image formation on a recording medium becomes equal to or smaller than a predetermined difference, or to update the adjustment parameters without updating the trained model.

2. The image processing device according to claim 1, wherein:

4. The target image is the predicted image.

2. The image processing device according to claim 1, wherein:

5. An adjustment means for adjusting the pixel values ​​of the target image using the corresponding adjustment parameters for each channel in a color space of the data of the target image; Further possessing 4. The image processing device according to claim 3, wherein:

6. The output image is a read image obtained by reading a predetermined calibration chart, The determination means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model, based on the read image and the predicted image.

4. The image processing device according to claim 3, wherein:

7. the determining means determines whether to update the trained model or to update only the adjustment parameters without updating the trained model by comparing gradation characteristics of a first patch group and a second patch group of a hue different from that of the first patch group in the read image with gradation characteristics of the first patch group and a gradation characteristic of the second patch group in the predicted image.

7. The image processing device according to claim 6,

8. an update means for updating at least one of the trained model and the adjustment parameters based on a determination result by the determination means; Further possessing 2. The image processing device according to claim 1, wherein:

9. an output control means for controlling output to present the determination result by the determination means to a user; Further possessing 2. The image processing device according to claim 1, wherein:

10. a receiving means for receiving an instruction from a user to update at least one of the trained model and the adjustment parameters; an update means for updating at least one of the trained model and the adjustment parameters based on the instruction; The image processing device according to claim 9 ,

11. When the determination result is to update only the adjustment parameters without updating the trained model, the updating means updates the adjustment parameters; If the determination result is to update the trained model, the output control means performs output control to present the determination result to a user; The update means updates the trained model based on the instruction. The image processing device according to claim 10,

12. The output control means performs output control to present to the user, in addition to the determination result, color information that cannot be adjusted by simply updating the adjustment parameters, which is a difference between the image adjusted using the adjustment parameters and the output image or the predicted image. The image processing device according to claim 11 ,

13. The output control means performs output control to present to the user, in addition to the determination result, the position in the adjusted image or the output image of a color that cannot be adjusted by only updating the adjustment parameters, which is a difference between the image adjusted using the adjustment parameters and the output image or the predicted image. The image processing device according to claim 11 ,

14. The trained model is updated using data of a scanned image obtained by scanning a training chart.

2. The image processing device according to claim 1, wherein:

15. The trained model is updated using data of the read image obtained by reading the learning chart in which the proportion of colors that cannot be adjusted by updating the adjustment parameters alone is increased, to determine the difference between the image after adjustment using the adjustment parameters and the output image or the predicted image. The image processing device according to claim 14,

16. After the trained model is updated, the adjustment parameters are also updated.

2. The image processing device according to claim 1, wherein:

17. an inspection means for inspecting whether the output image satisfies a predetermined standard by comparing an image adjusted using the adjustment parameters with the output image or the predicted image; Further possessing 2. The image processing device according to claim 1, wherein:

18. a preview unit that performs output control to present the image adjusted using the adjustment parameters to a user as a preview image of an image to be formed on a recording medium; Further possessing 2. The image processing device according to claim 1, wherein:

19. a generation step of generating a predicted image, which is an image obtained by predicting an output image corresponding to an input image, using the trained model; a determination step of determining whether to update the trained model or to update adjustment parameters for adjusting pixel values ​​of a target image without updating the trained model based on a difference between the output image and the predicted image; Including An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

Images