Image forming apparatus and image processing method

Abstract

The present invention aims to provide a method for reducing toner consumption in accordance with the attention level estimated by machine learning of an area in image data. [Solution] The image forming device of the present invention is characterized by having an acquisition means for acquiring an attention level estimated by machine learning for an area in an image based on image data, an execution means for performing image processing on the image data based on the estimated attention level so that the amount of recording material used to draw the area in the image with low attention level is less than the amount of recording material used to draw the area in the image with high attention level, and a printing means for printing on a printing medium based on the image data on which the image processing has been performed.

Description

[Technical Field]

[0001] The present invention relates to an image forming apparatus and an image processing method. [Background technology]

[0002] Patent Document 1 is known as a technique for reducing the amount of recording material used in printing.

[0003] Patent document 1 discloses a technology that analyzes distance information added to photographs taken with multiple cameras using the principles of triangulation, etc., and reduces toner consumption in out-of-focus areas. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-006982 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when printing a photograph of a flower field, the method described in Patent Document 1 may not be appropriate. Assume that the photograph of the flower field depicts not only the sky but also multiple flowers, soil, and the like. Furthermore, assume that the photograph focuses on one flower as the subject, while the other flowers are out of focus, but their blurred colors create a beautiful depiction. In other words, the flower attracts a lot of attention regardless of whether it is in focus or not, while the sky and soil attract relatively little attention.

[0006] The method described in Patent Document 1 reduces toner consumption in out-of-focus areas (objects), and therefore also reduces toner consumption in out-of-focus flowers, which are one of the elements that create a sense of beauty.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a method for reducing toner consumption in accordance with the attention level estimated by machine learning of an area in image data. [Means for solving the problem]

[0008] In order to solve the above problem, the image forming apparatus of the present invention is characterized by having an acquisition means for acquiring an attention level estimated by machine learning in an area in an image based on image data, an execution means for performing image processing on the image data based on the estimated attention level so that the amount of recording material used to draw the area in the image with low attention level is less than the amount of recording material used to draw the area in the image with high attention level, and a printing means for printing on a printing medium based on the image data on which the image processing has been performed. [Effects of the Invention]

[0009] According to the present invention, it is possible to reduce toner consumption in accordance with the attention level estimated by machine learning of an area in image data. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing device 100 according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing an example of a logical configuration of an image processing device 100 according to an embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an example of a toner saving process according to a first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a photograph represented by image data in this embodiment. [Figure 5] FIG. 1 is a diagram showing an example of semantic label information according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram showing an example of saliency information according to the present embodiment; [Figure 7] FIG. 10 is a diagram showing an example of a toner reduction area in the present embodiment. [Figure 8]FIG. 1 is a diagram showing an example of a color conversion LUT according to the present embodiment. [Figure 9] FIG. 10 is a diagram showing an example of density adjustment in this embodiment. [Figure 10] 10 is a flowchart illustrating an example of a toner saving process according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a buffer area according to a second embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a print setting screen according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any manner.

[0012] (First embodiment) <Hardware configuration of image processing device 100> FIG. 1 is a diagram showing the hardware configuration of an image processing device 100 according to this embodiment.

[0013] The image processing device 100 includes a CPU 101, RAM 102, ROM 103, a storage unit 104, a general-purpose I / F unit 105, a video I / F unit 106, a communication I / F unit 107, and a printing unit 111. These units are connected by an internal system bus 108, allowing communication between the modules. Note that CPU stands for Central Processing Unit, RAM stands for Random Access Memory, ROM stands for Read Only Memory, and I / F stands for Interface.

[0014] The CPU 101 reads the main program from the memory unit 104 in accordance with the initial program in the memory unit 104 and stores it in the RAM 102. The RAM 102 is used as a main memory for storing programs and for working. The ROM 103 is used to temporarily store data created during program processing. The memory unit 104 is used to store data such as programs, image data, and look-up tables (hereinafter referred to as "LUTs").

[0015] The general-purpose I / F unit 105 is a serial device interface such as a USB (Universal Serial Bus), and is connected to an input device 109 for inputting user instructions such as a keyboard or mouse. The video I / F unit 106 is connected to a monitor 110. The communication I / F unit 107 is used when communicating via a network. The printing unit 111 is a printer or the like, and is used to print an image based on image data on a printing medium.

[0016] In this embodiment, the input device 109 is an example of a reception unit, and receives various inputs and settings from the user.

[0017] The image processing apparatus is an example of an image forming apparatus, and may be a printing apparatus such as an MFP (Multi Function Peripheral), a multifunction peripheral, or a printer.

[0018] <Logical Configuration of Image Processing Device 100> FIG. 2 is a diagram showing the logical configuration of the image processing device 100 according to this embodiment.

[0019] The image processing device 100 includes an image input unit 210, an image conversion unit 220, and an image formation unit 230. The image input unit 210, the image conversion unit 220, the image formation unit 230, and the image conversion unit 220 are controlled by a CPU 101 and perform their respective functions.

[0020] The image input unit 210 is an image input means for inputting image data. The image data may be input through an application such as a printer driver on a PC connected to the image processing device 100, obtained by scanning using a scanner unit, or received via a network.

[0021] The image conversion unit 220 includes an image information generation unit 221, an area division unit 222, an attention information generation unit 223, a color conversion processing unit 224, a density adjustment unit 225, and a pseudo-halftone processing unit 226, and converts the data sent from the image input unit 210 into an image for printing.

[0022] The image information generation unit 221 performs image processing on the data sent from the image input unit 210 and generates image information that can be distinguished by object, such as graphics, text, photographs, etc. If object information is not added to the data sent from the image input unit 210, for example, edges are detected from the image data of the image and the continuity of the edges is examined to identify the graphic area, text area, photograph area, etc.

[0023] The region division unit 222 performs semantic region division processing (panoptic segmentation) on the photograph region identified by the image information generation unit 221 using a pre-trained classifier, and calculates semantic label information for each pixel.

[0024] The pre-trained classifier can be obtained by, for example, using a learning model such as SegNet or U-Net based on machine learning (particularly convolutional neural network (CNN)) and training it in advance using pairs of images and teacher semantic labels. CNN is an abbreviation for Convolutional Neural Network. Note that while a machine learning-based classifier is used in this embodiment, a rule-based method such as the active contour method (snakes method) or the level set method may also be used as long as semantic region segmentation is possible, or a combination of machine learning and rule-based methods may also be used.

[0025] The attention information generation unit 223 performs saliency inference on the photographic region identified by the image information generation unit 221 using a pre-trained inference device, and calculates saliency for each pixel. In this embodiment, saliency indicates the degree to which a person tends to pay attention to an image when looking at it, and is an example of an attention level where a higher value indicates higher saliency. For example, when a bright object exists on a low-brightness background, the saliency of the bright object is high.

[0026] Furthermore, when a high-luminance object and a low-luminance object exist on a high-luminance background, the salience of the high-luminance object will be relatively low compared to the salience of the low-luminance object. Similarly, when a green object and an orange object exist on a red background, the salience of the green object, which is the complementary color of red, will be relatively high compared to the salience of the orange object. Thus, salience in a still image is determined by the spatial arrangement of visual stimuli such as luminance and color.

[0027] Furthermore, in a moving image, for example, when there are both moving and stationary objects, the salience of the moving object is relatively higher than that of the stationary object. In this way, the salience of a moving image is determined not only by the spatial arrangement of the visual stimuli but also by the temporal changes in the visual stimuli.

[0028] A pre-trained inference machine can be obtained by, for example, using a machine learning model such as SalNet or SalGAN based on machine learning (especially CNN) and training it in advance using a pair of an input image and a saliency map for the input image. Here, the saliency map is created by measuring the eye movement of a subject when the subject freely observes an image displayed on a monitor or the like for several seconds. For example, by using the method described above, it is possible to estimate the attention level of image data by machine learning. Specifically, the attention level is estimated by some or all of these calculations.

[0029] Although this embodiment uses a machine learning-based inference device, a rule-based method may also be used if saliency inference is possible. The rule-based method is a well-known method based on cognitive mechanisms, and saliency inference can be performed using feature integration theory. Feature integration theory states that a person's visual field image is processed in parallel for each feature (brightness, color, and tilt), and these features are ultimately integrated.

[0030] Using feature integration theory, image features such as brightness, color, and gradient are calculated for each image, and the saliency of each image feature is calculated. The saliency of each pixel can be calculated by integrating the calculated saliency of each image feature.

[0031] For example, brightness distribution information I, red-green contrast information RG, and blue-yellow contrast information BY are extracted from the input image, and tilt information of 0°, 45°, 90°, and 135° is extracted from the input image using a Gabor filter. For each piece of extracted information, brightness, color, and tilt feature values ​​are calculated using a Difference of Gaussian filter, and normalization is performed. Saliency can be calculated by taking the linear sum of each calculated feature value.

[0032] Another rule-based saliency inference method is log-spectral analysis, which calculates the log spectrum of an input image, extracts the residual between the calculated log spectrum and a log spectrum obtained by smoothing the calculated log spectrum, and converts the spectral residual into the spatial domain to calculate saliency.

[0033] The color conversion processing unit 224 converts the image data sent from the image input unit 210 into data suitable for the target image processing device. For example, if the input is RGB data and the image processing device is a general color printer using CMYK toner, the conversion is performed using a color conversion table (LUT).

[0034] The color conversion LUT uses a 3D LUT, which is one of the most common color conversion processing methods. This method is a lookup table that shows the correspondence for converting RGB data to CMYK data, and is composed of NxNxN grid points. In principle, if the grid spacing is narrow enough, it is possible to perform color conversion with high accuracy. However, in reality, due to issues such as memory capacity and processing speed, it is extremely rare for the points to be color converted to coincide with grid points, so CMYK is obtained using 3D interpolation processing.

[0035] RGB stands for Red, Green, and Blue, and are known as the three primary colors of light. RGB data is used in display systems that emit light. CMYK stands for Cyan, Magenta, Yellow, and Key plate, and are known as process colors. CMYK data is used for printing on paper.

[0036] The density adjustment unit 225 performs density adjustment on the data processed by the color conversion processing unit 224. For example, if the data processed by the color conversion processing unit 224 is a CMYK signal value, the signal value of each color, C, M, Y, and K, is adjusted individually to adjust the strength of the density. In other words, the density is adjusted by adjusting the single color signal value.

[0037] The pseudo-halftone processing unit 226 performs pseudo-halftone processing on the data processed by the density adjustment unit 225 using a method such as a density pattern method, an ordered dither method, or an error diffusion method.

[0038] The image forming unit 230 receives the image for printing generated by the image conversion unit 220 and forms the image on a printing medium using a recording material such as toner.

[0039] <Toner saving process flow> 3 is a flowchart showing the operation when the image processing device 100 performs toner saving for the photo area of ​​input image data and outputs the image to a recording medium. By performing this process, it becomes possible to efficiently reduce toner consumption while preventing a decrease in print quality.

[0040] 3 is realized by CPU 101 expanding program code stored in ROM 103 into RAM 102, and reading and executing the program code expanded in RAM 102. This process is started, for example, when a user inputs an execution instruction via input device 109 of image processing device 100. Image input unit 210, image conversion unit 220, image formation unit 230, and each unit of image conversion unit 220 are controlled by CPU 101 and perform their respective functions.

[0041] In step S301, the image input unit 210 controlled by the CPU 101 acquires image data. The method of acquiring the image data may be a method in which the image data stored in the storage unit 104 is designated by the user or a method in which the image data is received via the communication I / F unit 107. Furthermore, if the image processing device 100 has a reading unit or a scanner, the image data may be acquired by reading an original document via these.

[0042] In step S302, the image information generation unit 221 controlled by the CPU 101 processes the image data acquired in step S301 and generates image information classified by object. Furthermore, if image data is acquired from the printer driver in step S301, the image data is described in a page description language (PDL), and bitmap image data may be rendered from this PDL. In this case, object information is added to the PDL. PDL is an abbreviation for Page Description Language.

[0043] In step S303, the region dividing unit 222 controlled by the CPU 101 uses the image information generated in step S302 to calculate semantic label information for each pixel of each photographic region in the bitmap image data using a pre-trained classifier. Note that, at this time, the semantic label information is calculated by performing panoptic segmentation on each photographic region.

[0044] FIG. 4 is an example showing a photograph area of ​​the bitmap image data acquired by the image input unit 210 in step S301.

[0045] FIG. 5 shows an example of semantic label information for FIG. 4 generated by the region dividing unit 222 in step 303, where the region dividing unit 222 calculates four pieces of semantic label information 501, 502, 503, and 504.

[0046] In this embodiment, semantic label information is calculated by panoptic segmentation, but semantic label information may be calculated using semantic segmentation, instance segmentation, or the like.

[0047] In step S304, the attention information generation unit 223 controlled by the CPU 101 uses the image information generated in step S302 to perform saliency inference for each photographic area of ​​the bitmap image data using a pre-trained inference device, and calculates saliency information for each pixel.

[0048] 6 is an example of saliency information calculated by the attention information generation unit 223, and indicates that the darker the pixel, the lower the saliency of the pixel. At this time, the saliency information is assumed to be normalized to 0 to 1, etc.

[0049] In step S305, the region dividing unit 222 controlled by the CPU 101 sets a toner reduction region using the semantic label information calculated in step S303 and the saliency information calculated in step S304.

[0050] The region division unit 222 performs a judgment on all pixels and, based on the average saliency of pixels assigned the same semantic label information, sets regions with an average saliency below a predetermined threshold as high toner reduction regions, while regions with an average saliency above the threshold as low toner reduction regions.

[0051] For example, suppose that the average saliency of pixels having semantic label information 501 is 0.2, the average saliency of pixels having semantic label information 502 is 0.4, the average saliency of pixels having semantic label information 503 is 0.9, and the average saliency of pixels having semantic label information 504 is 0.8. Also, suppose that the threshold value set as the high toner reduction region is 0.7.

[0052] 7, pixels having semantic label information 501 and 502 are set as high toner reduction regions 701, and pixels having semantic label information 503 and 504 are set as low toner reduction regions 702. In this embodiment, the region dividing unit 222 sets high toner reduction regions based on the average saliency of pixels assigned the same semantic label information, but high toner reduction regions may also be set based on statistics such as minimum values, maximum values, median values, and percentile values.

[0053] In this embodiment, the region dividing unit 222 sets high toner reduction regions based on the average saliency of pixels assigned the same semantic label information, but it may also set high toner reduction regions by combining average saliency and area information. In this case, for example, it is possible to set regions where (average saliency + number of pixels with the same semantic label information / total number of pixels) / 2 has a value less than a predetermined threshold as high toner reduction regions, and regions where the value is equal to or greater than the threshold as low toner reduction regions.

[0054] In step S306, the color conversion processing unit 224 controlled by the CPU 101 refers to the color conversion LUT and performs color conversion to reduce the amount of toner used to draw the high toner reduction area 701 set in step S305 to less than the amount of toner used to draw the low toner reduction area 702.

[0055] FIG. 8 is a diagram showing an example of a color conversion LUT, and in this embodiment, the color conversion LUT is stored in the storage unit 104.

[0056] Figure 8(a) shows a normal color conversion LUT 801, (b) shows a low-reduction color conversion LUT 802, and (c) shows a high-reduction color conversion LUT 803, which are LUTs such that the toner consumption is normal color conversion LUT 801 > low-reduction color conversion LUT 802 > high-reduction color conversion LUT 803.

[0057] The color conversion processing unit 224 references the low-reduction color conversion LUT 802 and performs color conversion processing to convert the RGB values ​​stored in each pixel of the low toner reduction area 702 set in step S305 into CMYK values. The color conversion processing unit 224 also references the high-reduction color conversion LUT 803 and performs color conversion processing to convert the RGB values ​​stored in each pixel of the high toner reduction area 701 set in step S305 into CMYK values.

[0058] Here, the low-reduction color conversion LUT 802 and the high-reduction color conversion LUT 803 are created by searching for CMYK values ​​that can reduce toner consumption within a certain color difference from the L*a*b* represented by the CMYK values ​​of the normal color conversion LUT 801.

[0059] For example, assume that CMYK=(255,197,0,0) in the normal color conversion LUT 801 represents L*a*b*=(28.7,7.4,-45.7) (not shown). In this case, if a search is made for CMYK values ​​that can reduce toner consumption within a color difference of 3.2, CMYK=(237,181,0,0), L*a*b*=(33.0,7.1,-43.8) is found.

[0060] If multiple candidates are found as a result of the search, for example, by selecting the candidate that best preserves hue from the top 5% of candidates with the greatest reduction in toner consumption, it is possible to reduce toner consumption and select CMYK values ​​that do not impair the impression.The above-mentioned search process also includes UCR (Under Color Removal) processing, which reduces toner consumption by replacing C, M, and Y with K.

[0061] Furthermore, the color conversion processing unit 224 may switch color conversion LUTs even within the same toner reduction region. For example, by performing color conversion by referencing a color conversion LUT with a larger UCR amount than the high-reduction color conversion LUT 803 only for the high-frequency portion of the high-toner reduction region 701, it is possible to utilize the spatial frequency characteristics of the human eye and reduce toner consumption while suppressing degradation of print quality.

[0062] In this embodiment, color conversion is performed on the low toner reduction region 702 using the low-reduction color conversion LUT 802, but color conversion may also be performed using the normal color conversion LUT 801. The method is not limited to these, as long as it is possible to reduce the amount of toner used to print (draw) the high toner reduction region 701 compared to the amount of toner used to print (draw) the low toner reduction region 702.

[0063] In step S307, the density adjustment unit 225 controlled by the CPU 101 adjusts the CMYK signal values ​​obtained in step S306, and performs density adjustment to reduce the amount of toner used to draw the high toner reduction area 701 to less than the amount of toner used to draw the low toner reduction area 702.

[0064] 9 shows an example of density adjustment. Of density adjustments 901 to 903, density adjustment 901 is a state in which no density adjustment has been performed. The toner reduction amount increases in the order density adjustment 903 > density adjustment 902 > density adjustment 901, and the amount of toner consumed for printing decreases.

[0065] The density adjustment unit 225 controlled by the CPU 101 adjusts the CMYK signal values ​​for the low toner reduction region 702 so that the input / output signal value relationship indicated by density adjustment 902 is met. Also, the CMYK signal values ​​for the high toner reduction region 701 are adjusted so that the input / output signal value relationship indicated by density adjustment 903 is met.

[0066] In this embodiment, the CMYK signal values ​​for the low toner reduction region 702 are adjusted to the input signal values ​​indicated by the density adjustment 902, but density adjustment 901 may also be used. Any method may be used as long as it is possible to reduce the amount of toner used to draw the high toner reduction region 701 compared to the amount of toner used to draw the low toner reduction region 702.

[0067] In step S308, the pseudo-halftone processing unit 226 controlled by the CPU 101 uses the CMYK signal values ​​obtained in step S307 to perform pseudo-halftone processing that reduces the amount of toner used in the high toner reduction area 701 to less than the amount of toner used in the low toner reduction area 702.

[0068] The pseudo-halftone processor 226 performs high-line screen processing on the high toner reduction region 701, and performs low-line screen processing on the low toner reduction region 702 with a lower line count than the high toner reduction region 701. Alternatively, after performing the screen processing, toner consumption may be reduced by performing a high-thinning-out amount of thinning-out processing on the high toner reduction region 701 and a low-thinning-out amount of thinning-out processing on the low toner reduction region 702. As long as it is possible to reduce the amount of toner used to draw the high toner reduction region 701 compared to the amount of toner used to draw the low toner reduction region 702, the method for doing so is not limited.

[0069] In step S309, the image forming unit 230 controlled by the CPU 101 forms (prints) an image on the print medium using a recording material such as toner, based on the adjustments made in the steps up to this point. That is, the amount of toner used to print the high toner reduction area 701 is less than the amount of toner used to print the low toner reduction area 702. When printing is complete, step S309 ends, and the processing of this flowchart also ends.

[0070] In this way, we have explained the processing of the present invention, which reduces the amount of toner used to render high toner reduction regions compared to the amount of toner used to render low toner reduction regions. Note that high toner reduction regions and low toner reduction regions were set by performing panoptic segmentation and saliency inference on photographic regions with high toner consumption. This is because, when given an image in which degradation has occurred in highly salient areas and an image in which the same degree of degradation has occurred in less salient areas, humans perceive the former as having a stronger degradation impression.

[0071] Also, as a method for reducing toner consumption, we have explained that the hue is maintained from the color used during normal printing (S306), gradation is maintained (S307), and color conversion is performed using the spatial frequency characteristics of the human eye (S306), etc. As a result, it is possible to efficiently reduce toner consumption while preventing a decrease in print quality.

[0072] In this embodiment, an example has been shown in which different color conversion processes, density adjustment processes, and pseudo-halftoning processes are executed as image processing suited to the high toner reduction region and the low toner reduction region, but all processes do not necessarily have to be performed separately. For example, the color conversion process may be common, with only the density adjustment process and pseudo-halftoning process being performed using different methods, as in steps S307 and S308, or the pseudo-halftoning process and density adjustment process may be common, with only the color conversion process being performed using a different method, as in step S306.

[0073] In this embodiment, an example is shown in which the area is divided into a high toner reduction area and a low toner reduction area, and different color conversion processes, density adjustment processes, and pseudo-halftoning processes are performed on the area. However, it is also possible to perform color conversion processes, density adjustments, and pseudo-halftoning processes according to the average saliency of pixels with the same semantic label information.

[0074] In this case, in step S305, the region division unit 222 does not divide the image into two regions, a high toner reduction region and a low toner reduction region, but sets pixels assigned the same semantic label information as multiple toner reduction regions and calculates the average saliency of each toner reduction region.

[0075] In step S306, the color conversion processing unit 224 controlled by the CPU 101 creates a color conversion LUT as follows according to the average saliency of each toner reduction region, and performs color conversion on each toner reduction region. New color conversion LUT= Low reduction color conversion LUT802 × average saliency + high reduction color conversion LUT803 × (1 - average saliency)

[0076] In step S307, the density adjustment unit 225 controlled by the CPU 101 adjusts the CMYK signal values ​​according to the average saliency of each toner reduction region so that the following input / output signal value relationship is achieved. New Density Adjustment = Density Adjustment 902 × Average Saliency + Density Adjustment 903 × (1 – Average Saliency)

[0077] In step S308, the pseudo-halftone processing unit 226 controlled by the CPU 101 performs pseudo-halftone processing with the following thinning amount from among a predetermined low thinning amount and a high thinning amount according to the average saliency of each toner reduction region. New Thinning Amount = Low Thinning Amount × Average Salience + High Thinning Amount × (1 - Average Salience)

[0078] Furthermore, in this embodiment, saliency inference in step S304 is performed for each photographic region of the input image data, but processing may also be performed simultaneously for multiple photographic regions of the input image data. This is because saliency is relative, and a highly salient region of a certain image 1 may appear relatively less salient when viewed simultaneously with a certain image 2 that has a more salient region. In this way, more salient regions of the input image data are set as low toner reduction regions, and less salient regions are set as low toner reduction regions, thereby making it possible to efficiently reduce toner consumption.

[0079] In this embodiment, the toner saving process flow for photographic regions has been described, but similar processes may be performed for graphic regions, or known toner saving processes may be performed. Furthermore, known toner saving processes may be performed for text regions, or steps S303 to S305 may be skipped and toner consumption reduction color conversion processes or pseudo-halftone processes that reduce degradation of print quality may be performed. Here, the color conversion process can reduce toner consumption while reducing degradation of print quality by, for example, increasing the density of the edges of text and decreasing the density inside the text.

[0080] Pseudo-halftoning, for example, involves performing a screen process followed by a thinning process using blue noise, which is highly dispersed, irregular, and uniform, making it possible to reduce toner consumption while utilizing optical dot gain and without causing large chips or jagged edges.

[0081] In addition, in this embodiment, the amount of toner consumption for input image data is not set, but a screen showing print settings may be displayed on the monitor 110, and the user may set the amount of toner consumption via the input device 109.

[0082] 12 is a diagram showing an example of a screen on which a user sets print settings used when the image processing device 100 forms an image. The setting screens shown in (a) and (b) of FIG. 12 may be displayed on the monitor 110 of the image processing device 100, or may be displayed on a PC screen communicably connected to the image processing device. The CPU 101 accepts setting values ​​set by the user on a main print setting screen 1200 or a toner save advanced setting screen 1220.

[0083] 12(a) shows a main screen 1200 for print settings. The main screen 1200 includes a preview area 1201, a document size setting area 1202, an output size setting area 1203, a number of copies setting area 1204, and print orientation setting areas 1205 and 1206, all of which are used for print settings. It also includes an OK button 1211 for confirming settings and a Cancel button 1212 for canceling settings, which are common to other screens.

[0084] The preview area 1201 is an area where a preview of the data to be printed is displayed, allowing the user to check the finished printout before starting printing. The document size setting area 1202 is an area where the document size is displayed, and the user selects a document size that matches the document size from the document sizes displayed in a pull-down menu. Note that if the image processing device 100 has acquired document size information, that information is displayed in the document size setting area 1202 first.

[0085] The output size setting area 1203 is an area for setting the size of the printed matter to be output, and the user selects the desired size from the output paper sizes displayed in a pull-down menu. The number of copies setting area 1204 is an area for the user to set the number of copies to be output, and the user can set the number of copies by pressing the arrow button or by inputting a number. The print orientation setting area 1205 is an area for specifying the paper orientation when printing the printed matter, and the user can select either portrait or landscape.

[0086] The toner saving checkbox 1207 is used to print objects in the print image after applying toner saving, and is set to enabled when checked and disabled when unchecked. If printing is performed with toner saving enabled (checked), the image will be printed with toner saving applied, and if printing is performed with toner saving disabled (unchecked), the image will be printed without toner saving applied.

[0087] An automatic button 1208 and a manual button 1209 are buttons for specifying whether to set detailed toner saving settings automatically or manually by the user, and when a toner saving check box 1207 is checked, either one can be selected.

[0088] When the automatic button 1208 is selected, a toner save processing flow is executed using a predetermined threshold value set as a high toner reduction area, a color conversion LUT, etc. When the manual button 1209 is selected, the image conversion conditions are changed according to the toner consumption amount set in the advanced settings 1210. On the other hand, when the automatic button 1208 is selected, printing is performed according to the above method.

[0089] When the advanced settings button 1210 is pressed, the screen transitions to a toner save advanced settings screen 1220.

[0090] 12(b) shows a toner save advanced setting screen 1220. The toner save advanced setting screen 1220 includes toner consumption setting areas 1221, 1222, and 1223. In the toner consumption setting areas 1221, 1222, and 1223, the user can set the toner consumption for each object within a range of 0% to 100%. When the toner consumption is set by the user, in steps S305 to S309, some or all of the region dividing unit 222, color conversion processing unit 224, density adjustment unit 225, and pseudo-halftone processing unit 226 change the image conversion conditions according to the set toner consumption.

[0091] Specifically, in step S305, the region dividing unit 222 adjusts the threshold value set as a high toner reduction region in accordance with the amount of toner consumed. In step S306, the color conversion processing unit 224 creates the new color conversion LUT described above and performs color conversion on pixels having the same semantic label information in the low toner reduction region in descending order of average salience. In step S307, the density adjustment unit 225 performs the new density adjustment described above on pixels having the same semantic label information in the low toner reduction region in descending order of average salience. The pseudo-halftone processing unit 226 performs pseudo-halftone processing using the new thinning amount described above on pixels having the same semantic label information in the low toner reduction region in descending order of average salience.

[0092] In this way, by changing the image conversion conditions according to the amount of toner consumed, it becomes possible to print with the amount of toner consumption intended by the user. Also, in this embodiment, the image processing device 100 performs image conversion and forms an image using CMYK toner, but it may also use special color toner, or ink or the like instead of toner.

[0093] Note that some or all of the above-described processes may be performed by machine learning. Note that the above description is an example of the processes performed by machine learning, and each process may be performed by a method other than the above.

[0094] (Second embodiment) <Toner saving process flow> In the first embodiment, a method was described in which high toner reduction areas and low toner reduction areas are set according to the saliency (attention level) of photographic areas, and the amount of toner used to draw the high toner reduction areas is reduced compared to the amount used to draw the low toner reduction areas. This makes it possible to reduce toner consumption while preventing a decrease in print quality, for example. Next, in this embodiment, a method is described with reference to FIG. 10 in which a buffering process is performed on the boundary between the high toner reduction areas and the low toner reduction areas to prevent the appearance of a step due to the difference in toner consumption.

[0095] The following describes the present embodiment in terms of differences from the first embodiment. Note that parts that are not described in detail are the same as the first embodiment, and therefore descriptions thereof will be omitted.

[0096] Steps S1001 to S1005 are the same as steps S301 to S305, and therefore a description thereof will be omitted.

[0097] 11 in the area near the boundary between the high toner reduction area 701 and the low toner reduction area 702 set in step S1005. The area division unit 222 sets the buffer area 1101, for example, based on the difference between the low toner reduction area 702 and an area obtained by extending the low toner reduction area 702 by 10 pixels from the boundary of the low toner reduction area 702 in the outward direction. If there is an area where the low toner reduction area 702 and the buffer area 1101 overlap, the overlapping area is set as the low toner reduction area 702.

[0098] In step S1007, the color conversion processing unit 224 controlled by the CPU 101 performs color conversion so that the amount of toner used for rendering the buffer area 1101 set in step S1006 is an amount between the amounts of toner used for rendering the high toner reduction area 701 and the low toner reduction area 702. During this color conversion, the color conversion processing unit 224 performs the color conversion by referring to the color conversion LUT stored in the storage unit 104. The color conversion method is the same as in step S306, so a description thereof will be omitted.

[0099] In step S1008, the density adjustment unit 225 controlled by the CPU 101 performs density adjustment so that the amount of toner used for rendering the buffer area 1101 is between the amounts of toner used for rendering the high toner reduction area 701 and the low toner reduction area 702. During this density adjustment, the density adjustment unit 225 adjusts the CMYK signal values ​​obtained in step S1006 to perform the density adjustment. The density adjustment method is the same as in step S307, so a description thereof will be omitted.

[0100] In step S1009, the pseudo-halftone processing unit 226 controlled by the CPU 101 performs pseudo-halftone processing to set the amount of toner used for rendering the buffer area 1101 to an amount between the amounts of toner used for rendering the high toner reduction area 701 and the low toner reduction area 702. During this pseudo-halftone processing, the pseudo-halftone processing unit 226 performs pseudo-halftone processing using the CMYK signal values ​​obtained by the density adjustment unit 225 in step S1007. The pseudo-halftone processing method is the same as in step S308, so a description thereof will be omitted.

[0101] Step S1010 is similar to step S308, and therefore a description thereof will be omitted.

[0102] In this way, by performing buffer processing on the boundary between the high toner reduction region and the low toner reduction region, it is possible to prevent the difference in toner consumption from becoming apparent. Note that in this embodiment, an example has been shown in which different color conversion processing, density adjustment, and pseudo-halftoning processing are performed as image processing appropriate for each of the high toner reduction region, low toner reduction region, and buffer region, but it is not necessarily necessary to perform all processing separately for each region.

[0103] For example, the color conversion process for the high toner reduction region and the buffer region may be common, with only the density adjustment and pseudo-halftoning process being performed by different methods, as in steps S1008 and S1009. Alternatively, the density adjustment and pseudo-halftoning process for the low toner reduction region and the buffer region may be common, with only the color conversion process being performed by a different method, as in step S1007. The method is not limited as long as the amount of toner used to draw the buffer region of the converted print image by the image conversion unit 220 is an amount between the amounts of toner used to draw the high toner reduction region and the low toner reduction region.

[0104] In addition, in this embodiment, the processing is switched by dividing the area into three regions: a high toner reduction region, a low toner reduction region, and a buffer region, but it is also possible to divide the area into more regions to prevent the emergence of steps due to differences in toner consumption.

[0105] Note that some or all of the above-described processes may be performed by machine learning. Note that the above description is an example of the processes performed by machine learning, and each process may be performed by a method other than the above.

[0106] As explained above, a process has been described in which high toner reduction areas and low toner reduction areas are set according to the content of the photographic area, and the amount of toner used to draw the high toner reduction areas is reduced compared to the amount of toner used to draw the low toner reduction areas. According to the present invention, it is possible to efficiently reduce toner consumption while preventing a decline in print quality. Furthermore, by performing a buffering process on the boundary between the high toner reduction area and the low toner reduction area, it is also possible to prevent the noticeable step caused by the difference in toner consumption. [Explanation of symbols]

[0107] 701 High Toner Reduction Area 702 Low Toner Reduction Area 801 Normal Color Conversion LUT 802 Low Reduction Color Conversion LUT 803 High Reduction Color Conversion LUT 1101 Buffer area 1200 Main Screen 1220 Toner save advanced settings screen

Claims

1. An acquisition means for acquiring an attention level estimated by machine learning in a region in an image based on image data; an execution means for executing image processing on the image data based on the estimated degree of attention so that the amount of recording material used to render the area of ​​the image with low attention is less than the amount of recording material used to render the area of ​​the image with high attention; a printing means for printing on a printing medium based on the image data that has been subjected to the image processing; An image forming apparatus comprising:

2. Calculation of saliency information and semantic label information is performed for each pixel of the image data by the machine learning; The attention level is calculated according to the calculated saliency information and the saliency information.

2. The image forming apparatus according to claim 1, wherein the image forming apparatus is a recording medium.

3. The image forming apparatus according to claim 2, characterized in that the execution means performs image processing on the image data so that the amount of recording material used to draw the area with low saliency information is less than the amount of recording material used to draw the area with high saliency information.

4. The image forming apparatus according to claim 2, characterized in that the execution means performs image processing on the image data so that the amount of recording material used to draw the nearby area included in the area with low saliency information, among the areas near the boundary between the area with low saliency information and the area with high saliency information, is less than the amount of recording material used to draw the area with high saliency information.

5. The image forming apparatus according to claim 4, characterized in that the execution means performs image processing on the image data so that the amount of recording material used to draw the area with low saliency information is less than the amount of recording material used to draw the nearby area.

6. 2. The image forming apparatus according to claim 1, wherein the execution unit executes image processing on the image data using a color conversion table.

7. 2. The image forming apparatus according to claim 1, wherein the execution means includes a process for adjusting density by adjusting a single color signal value.

8. Further comprising a reception unit, 2. The image forming apparatus according to claim 1, wherein the reception unit receives a setting for the amount of recording material to be used when the printing unit prints.

9. 2. The image forming apparatus according to claim 1, wherein the image data includes image data of a photograph.

10. 10. The image forming apparatus according to claim 1, wherein the execution unit executes the process by machine learning.

11. an acquisition step of acquiring an attention level estimated by machine learning for a region in an image based on image data; an execution step of executing image processing on the image data based on the estimated attention level so that an amount of recording material used to render the area of ​​the image with low attention level is less than an amount of recording material used to render the area of ​​the image with high attention level; a printing step of printing on a printing medium based on the image data that has been subjected to the image processing; An image processing method comprising:

12. Calculation of saliency information and semantic label information is performed for each pixel of the image data by the machine learning; The attention level is calculated according to the calculated saliency information and the saliency information.

12. The image processing method according to claim 11.

13. The image processing method according to claim 12, characterized in that the execution step performs image processing on the image data so that the amount of recording material used to render the area with low saliency information is less than the amount of recording material used to render the area with high saliency information.

14. The image processing method according to claim 12, characterized in that the execution step performs image processing on the image data so that the amount of recording material used to draw the nearby area included in the area with low saliency information, among the areas near the boundary between the area with low saliency information and the area with high saliency information, is less than the amount of recording material used to draw the area with high saliency information.

15. The image processing method according to claim 14, characterized in that the execution step performs image processing on the image data so that the amount of recording material used to render the area with low saliency information is less than the amount of recording material used to render the nearby area.

16. 12. The image processing method according to claim 11, wherein the executing step executes image processing on the image data using a color conversion table.

17. 12. The image processing method according to claim 11, wherein the executing step includes a process of adjusting monochrome signal values ​​to adjust density.

18. Further, the method includes a reception process.

12. The image processing method according to claim 11, wherein the receiving step receives a setting for an amount of recording material to be used when the printing step prints.

19. 12. The image processing method according to claim 11, wherein the image data includes image data of a photograph.

20. 20. The image processing method according to claim 11, wherein the executing step is performed by machine learning.

Images