Image processing device, image processing method, and program
The image processing apparatus uses machine learning models to identify and process fluorescent color regions, addressing the challenge of inconsistent reproduction by combining fluorescent and non-fluorescent data, ensuring accurate and consistent output of highly saturated colors.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- KONICA MINOLTA INC
- Filing Date
- 2024-12-17
- Publication Date
- 2026-06-29
Smart Images
Figure 2026106119000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.
Background Art
[0002] A multifunction peripheral (MFP) or the like equipped with a scanner function and a printer function can read various originals and reproduce and output them. Such a multifunction peripheral irradiates light on a set original and reads the reflected light from the reading surface of the original with an image sensor such as a charge-coupled device (CCD). The image sensor detects the density level corresponding to the amount of light of each RGB color component included in the reflected light and generates an image signal. The multifunction peripheral performs A / D conversion on the image signal generated by the image sensor to generate image data.
[0003] Among the various originals read by such a multifunction peripheral, there are originals containing fluorescent colors. Such fluorescent colors contain fluorescent components. Fluorescent components exhibit the characteristic of emitting visible light (fluorescing) the energy obtained by absorbing short-wavelength and non-visible light such as ultraviolet light. The color of a normal color that does not contain a fluorescent component is determined only by the reflected light of the irradiated light. On the other hand, the color of a fluorescent color containing a fluorescent component is determined by the light obtained by adding the reflected light of the irradiated light and the fluorescence emitted by itself. Fluorescent colors containing fluorescent components have a very high value of reflectance represented by the ratio of the intensity of the light irradiating the original to the intensity of the reflected light from the original in a certain wavelength range. When the reflectance of standard white light is set to 100%, the reflectance of a non-fluorescent color that does not contain a fluorescent component is always a value of 100% or less. On the other hand, the reflectance of a fluorescent color containing a fluorescent component may be a value of 100% or more in a certain wavelength range. Since a normal toner that does not contain a fluorescent component has a reflectance of 100% or less in any wavelength range, it is difficult to faithfully reproduce the fluorescent colors contained in the original.
[0004] One reason for the reduced reproducibility of fluorescent colors is that the ratio of each RGB color component value in the fluorescent color changes due to the brightness adjustment performed on the scanned image data. Brightness adjustment is performed by multiplying each RGB color component value in the image data by a predetermined correction coefficient. Fluorescent colors emit fluorescence, and depending on the wavelength of the fluorescence, there are color components among the RGB color components whose values become extremely large. Therefore, among the RGB color component values before brightness adjustment, there are values that are close to the maximum processing value (for example, 255 for 8-bit) or that reach the maximum processing value. For example, suppose the RGB color component values of a certain fluorescent color are (255, 200, 100), and the predetermined correction coefficient is 1.2. The R value has reached the maximum processing value, so it remains 255 even after brightness adjustment. On the other hand, the G value changes from 200 to 240 and the B value changes from 100 to 120 after brightness adjustment. In other words, while the R value remains unchanged after brightness adjustment, the G and B values change to 1.2 times their original values. Therefore, the ratio of each RGB color component value before brightness adjustment and the ratio of each RGB color component value after brightness adjustment change significantly. When the ratio of each RGB color component value changes significantly due to brightness adjustment, there is a problem in that the reproducibility of fluorescent colors decreases.
[0005] Therefore, Patent Document 1 discloses a configuration in which, when adjusting brightness, the correction coefficient used in the fluorescent color image region is a correction coefficient that reduces the rate of brightness increase compared to the non-fluorescent color image region. According to the configuration described in Patent Document 1, it is possible to suppress large changes in the ratio of the values of each RGB color component in the image data after brightness adjustment. Thus, the reproducibility of fluorescent colors can be improved. [Prior art documents] [Patent Documents]
[0006] [Patent Document 1] Japanese Patent Publication No. 2020-28088 [Overview of the project] [Problems that the invention aims to solve]
[0007] Generally, the color of highlighter ink used on a document varies depending on the manufacturer and paper type. Furthermore, it is difficult to accurately reproduce highly saturated / bright colors (yellow, yellow-green) used with highlighter ink on a document, making accurate acquisition challenging. As a result, when outputting (copying) a particular color, the output color will differ each time, posing a problem where the output may not match the color the user desires.
[0008] The configuration described in Patent Document 1 can improve the reproducibility of fluorescent colors by adjusting the correction coefficient. However, while the configuration described in Patent Document 1 is effective for colors that can be expressed by adjusting the correction coefficient, it cannot accurately reproduce colors that cannot be expressed by adjusting the correction coefficient. Colors that cannot be expressed by adjusting the correction coefficient include, for example, highly saturated yellow fluorescent pens and yellow-green fluorescent pens. Furthermore, there is a problem that the output color may differ even for the same color if the manufacturer of the fluorescent pen or the paper used for printing is different, but the configuration described in Patent Document 1 cannot solve this problem.
[0009] The present invention aims to provide an image processing apparatus, an image processing method, and a program that can reproduce fluorescent colors that are not possible to express by adjusting correction coefficients (high-saturation fluorescent colors), and that are independent of the manufacturer and paper type. [Means for solving the problem]
[0010] The invention described in claim 1 was made to achieve the above objective, In an image processing device, Image acquisition unit that acquires image data, A fluorescent color region identification unit identifies the fluorescent color region in the image data acquired by the image acquisition unit, A composite data creation unit creates composite data of fluorescent color image data created for an image of a fluorescent color region identified by the fluorescent color region identification unit and non-fluorescent color image data created for an image of a region different from the fluorescent color region. Equipped with, The aforementioned composite data creation unit, For the image of the fluorescent color region, a first machine learning model is used to create fluorescent color image data that outputs fluorescent pen color image data that matches the image, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, This method is characterized by creating composite data.
[0011] The invention described in claim 2 is an image processing apparatus described in claim 1, The fluorescent color region identification unit is characterized by identifying the fluorescent color region using image data acquired by the image acquisition unit at a first light intensity and image data acquired by the image acquisition unit at a second light intensity which is lower than the first light intensity.
[0012] The invention described in claim 3 is an image processing apparatus described in claim 1, The fluorescent color region identification unit is characterized by identifying the fluorescent color region using a second machine learning model.
[0013] The invention described in claim 4 is an image processing apparatus described in claim 3, The system is characterized by comprising a first update unit that, when the fluorescent color region identified by the fluorescent color region identification unit is deemed inappropriate, further trains the second machine learning model using the input image data acquired by the image acquisition unit and the output image data to which the corrected region attributes have been assigned, thereby updating the second machine learning model.
[0014] The invention described in claim 5 is an image processing apparatus described in claim 1, Regarding the colors in the fluorescent color range, even if there are differences in the corresponding highlighter colors depending on the manufacturer or paper type, they are defined as a single class and defined as input data for the first machine learning model in the first definition unit, A second definition unit defines the image data of the fluorescent pen color that matches the image data acquired by the image acquisition unit as a single color, and defines it as the output data of the first machine learning model, A learning data creation unit that creates, as learning data, the input data defined by the first definition unit and the output data defined by the second definition unit characterized by comprising the same.
[0015] The invention according to claim 6 is the image processing apparatus according to claim 5, wherein the first definition unit defines, as the input data, at least one piece of information among the manufacturer of the fluorescent pen, the color of the fluorescent pen, the manufacturer of the paper, and the paper type, in addition to the color information of the image data acquired by the image acquisition unit.
[0016] The invention according to claim 7 is the image processing apparatus according to claim 5, when the fluorescent color image data created by the composite data creation unit is determined to be inappropriate, the first machine learning model is additionally trained using the input image data acquired by the image acquisition unit and the corrected output image data, and a second update unit for updating the first machine learning model is provided.
[0017] The invention according to claim 8 is the image processing apparatus according to claim 4 or 7, characterized by comprising a collection unit that collects additional learning data from data on the cloud.
[0018] The invention according to claim 9 is the image processing apparatus according to claim 8, wherein the additional learning data is data to which at least one piece of information among the manufacturer of the fluorescent pen, the color of the fluorescent pen, the manufacturer of the paper, and the paper type is attached.
[0019] The invention according to claim 10 is the image processing apparatus according to claim 1, wherein the first machine learning model is provided on the cloud or inside an edge device.
[0020] The invention according to claim 11 is the image processing apparatus according to claim 3, The second machine learning model is characterized by being installed on the cloud or inside an edge device.
[0021] The invention described in claim 12 is an image processing apparatus described in claim 1, The system includes a storage unit for temporarily storing image data acquired by the image acquisition unit, The synthesized data creation unit is characterized by discarding the image data stored in the storage unit when it determines that no correction is needed to the fluorescent color image data created using the first machine learning model.
[0022] The invention described in claim 13 is an image processing apparatus described in claim 1, The image data is updated by modifying the attributes of the fluorescent color region, and is characterized by having a third update unit that replaces data only in the region whose attributes have been modified.
[0023] The invention described in claim 14 is, An image processing method for an image processing apparatus equipped with an image acquisition unit that acquires image data, A fluorescent color region identification step for identifying the fluorescent color region in the image data acquired by the image acquisition unit, A composite data creation step, which creates composite data of fluorescent color image data created for an image of a fluorescent color region identified in the fluorescent color region identification step and non-fluorescent color image data created for an image of a region different from the fluorescent color region, Includes, The aforementioned synthesis data creation process is as follows: For the image of the fluorescent color region, a first machine learning model is used to create fluorescent color image data that outputs fluorescent pen color image data that matches the image, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, This method is characterized by creating composite data.
[0024] The invention described in claim 15 is, A computer for an image processing device equipped with an image acquisition unit that acquires image data, A fluorescent color region identification unit identifies the fluorescent color region in the image data acquired by the image acquisition unit. A composite data creation unit creates composite data of fluorescent color image data created for an image of a fluorescent color region identified by the fluorescent color region identification unit and non-fluorescent color image data created for an image of a region different from the fluorescent color region. To make it function as, The aforementioned composite data creation unit, For the image of the fluorescent color region, a first machine learning model is used to create fluorescent color image data that outputs fluorescent pen color image data that matches the image, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, This program is characterized by its ability to create composite data. [Effects of the Invention]
[0025] According to the present invention, even colors that cannot be expressed by adjusting the correction coefficient (highly saturated fluorescent colors) can be reproduced in a way that is independent of the manufacturer and type of paper. [Brief explanation of the drawing]
[0026] [Figure 1] This figure shows the overall configuration of an image forming apparatus equipped with an image processing apparatus according to this embodiment. [Figure 2] This figure shows the configuration of the image processing apparatus according to Example 1. [Figure 3] This figure shows an example of a method for identifying a fluorescent color region using a fluorescent color region identification unit. [Figure 4] This figure shows an example of a method for creating a fluorescent color region identification model in advance. [Figure 5] This figure shows another example of a method for identifying fluorescent color regions using a fluorescent color region identification unit. [Figure 6] This figure shows the configuration of the image processing apparatus according to Example 2. [Figure 7]This figure shows an example of how the results of identifying fluorescent regions using a fluorescent region identification model are displayed as a preview on the control panel. [Figure 8] This figure shows an example of a method for creating a fluorescence color estimation model in advance. [Figure 9] This figure shows the configuration of the image processing apparatus according to Example 6. [Figure 10] This diagram illustrates the handling of image data in an image processing device. [Figure 11] This diagram illustrates how an image processing device temporarily stores image data when creating a highly compressed PDF. [Modes for carrying out the invention]
[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0028] The image processing apparatus 1 according to this embodiment can be incorporated into the image forming apparatus 10. Figure 1 shows the overall configuration of an image forming apparatus 10 that includes an image processing device 1.
[0029] The image forming apparatus 10 is a Multi-functional Peripheral (MFP) with numerous functions, including copying and facsimile communication. The image forming apparatus 10 comprises a main controller 11, an ADF 12, an image scanner 13, an operation panel 14, a communication interface 15, an engine controller 16, a printer engine 17, storage 18, and a machine learning model 19.
[0030] The main controller 11 oversees the control of the entire image forming apparatus 10. The main controller 11 performs predetermined controls in response to user instructions via the operation panel 14 and operation requests from external devices (not shown) communicating via the communication interface 15. In copy operations, the main controller 11 controls the ADF 12 and the image scanner 13. The image scanner 13 functions as the image acquisition unit of the present invention, acquiring image data. In various printing operations, including copying, the main controller 11 gives instructions to the engine controller 16, which is responsible for controlling the printer engine 17. The main controller 11 accesses the hard disk drive (HDD), which serves as the built-in storage 18, as needed. The main controller 11 has a CPU, RAM, ROM, and GPU that function as the image processing apparatus 1 of the present invention. The computer program for realizing the image processing apparatus 1 is stored in the ROM or storage 18 of the main controller 11.
[0031] The machine learning model 19 is configured within the image forming apparatus 10 and is used for image processing using the machine learning model 19. The machine learning model 19 includes a fluorescence color estimation model (first machine learning model) 191 and a fluorescence color region identification model (second machine learning model) 192. Note that the machine learning model 19 is not limited to being configured within the image forming apparatus 10, but may also be configured within the image processing apparatus 1. Alternatively, the machine learning model 19 may be configured in an external device outside the image forming apparatus 10, and the image forming apparatus 10 (image processing apparatus 1) and the external device (machine learning model 19) may be connected by a network.
[0032] In the copying operation, the image processing device 1 receives image data input from the image scanner 13 that reads the image. The image processing device 1 identifies the fluorescent color regions in the input image data. Based on the above region identification, the image processing device 1 defines region attributes for each pixel. The region attribute information (data indicating the region attributes of pixels) generated by the image processing device 1 is passed to a functional element that generates rasterized data for printing. The image processing device 1 performs image processing on the fluorescent color regions and non-fluorescent color regions indicated by the region attribute information to optimize the image quality according to the region attributes defined for each pixel. This generates a raster image. The generated raster image is printed by the printer engine 17 using electrophotography.
[0033] (Example 1) Figure 2 shows the configuration of the image processing device 1 according to Example 1. The image processing apparatus 1 according to Embodiment 1 comprises a fluorescent color area identification unit 110 and a composite data creation unit 120. The fluorescent color area identification unit 110 and the composite data creation unit 120 are functional elements realized by a CPU, which is a computer, executing an image processing program.
[0034] First, the fluorescent color region identification unit 110 receives input image data 40 from the image scanner 13 that has scanned the original document. The fluorescent color region identification unit 110 identifies the fluorescent color region 40a in the input image data 40 acquired by the image scanner 13. This allows the fluorescent color region 40a and the non-fluorescent color region 40b in the input image data 40 to be identified. The composite data creation unit 120 creates composite data 50 from the fluorescent color image data 41 created from the image of the fluorescent color region 40a identified by the fluorescent color region identification unit 110 and the non-fluorescent color image data 42 created from the image of a region different from the fluorescent color region 40a (non-fluorescent color region 40b). Specifically, the composite data creation unit 120 creates the fluorescent color image data 41 from the image of the fluorescent color region 40a using the fluorescent color estimation model 191. More specifically, first, the composite data creation unit 120 compares the image of the fluorescent color region 40a with the fluorescent color estimation model 191 (see reference numeral 101). The fluorescence color estimation model 191 is a machine learning model that outputs fluorescent pen color image data (fluorescent color image data 41) that matches the image of the fluorescent color region 40a. Next, the composite data creation unit 120 creates the fluorescent color image data 41 according to the fluorescence color estimation model 191 (see reference numeral 102). The composite data creation unit 120 also creates non-fluorescent color image data 42 using a color reproduction method for an image of a region different from the fluorescent color region 40a (non-fluorescent color region 40b) (see reference numeral 103). Then, the composite data creation unit 120 creates composite data 50 of the created fluorescent color image data 41 and the created non-fluorescent color image data 42 (see reference numeral 104).
[0035] The image processing device 1 according to Example 1 can estimate the color of a fluorescent pen (fluorescent color) by using a fluorescent color estimation model 191. This makes it possible to reproduce the color information of the fluorescent color region even when copying a document written with a highly saturated fluorescent pen (yellow, yellow-green). It is preferable to improve the discrimination accuracy of the fluorescent color estimation model 191 by performing machine learning on various highly saturated fluorescent colors.
[0036] (Example 2) Figure 3 shows an example of a method for identifying the fluorescent color region 40a using the fluorescent color region identification unit 110. First, the image processing device 1 scans the original document with the image scanner 13 at normal light intensity (first light intensity) (reference numeral 105). Normal light intensity is the initial light intensity and is the light intensity used during normal scanning. The fluorescent color area identification unit 110 receives input image data 40 acquired by the image scanner 13 when scanning the original document at normal light intensity. Input image data 40 is image data acquired by the image scanner 13 at first light intensity (normal light intensity). When an original document written with a highly saturated fluorescent pen is scanned at normal light intensity, one of the RGB values reaches its maximum value (255), resulting in the loss of color information. Therefore, the image processing device 1 first extracts the area (area E1) in the input image data 40 where one of the RGB values reaches its maximum value (255) (reference numeral 106). Separately from the above, the image scanner 13 scans the original document with a light intensity lower than normal (in this case, half the light intensity) (reference numeral 107). This acquires input image data 43 with half the light intensity. The input image data 43 is image data acquired by the image scanner 13 at a second light intensity (half the light intensity in this case), which is lower than the first light intensity. Then, a region (region E2) in the input image data 43 in which the RGB values are greater than half the RGB values in the input image data 40 is extracted (indicated by 108). Next, the region that overlaps between region E1 and region E2 is identified as the fluorescent color region 40a (indicated by 109). This makes it possible to identify the fluorescent color region 40a and the non-fluorescent color region 40b in the input image data 40. As described above, the fluorescent color region identification unit 110 identifies the fluorescent color region 40a using the input image data 40 acquired with the first light intensity and the input image data 43 acquired with the second light intensity.
[0037] (Example 3) Referring to Figures 4 and 5, another example of a method for identifying the fluorescent color region 40a using the fluorescent color region identification unit 110 will be described. Figure 4 is a diagram showing an example of a method for creating a fluorescent color region identification model 192 in advance. Figure 5 is a diagram showing another example of a method for identifying the fluorescent color region 40a using the fluorescent color region identification unit 110.
[0038] The image processing device 1 prepares a fluorescence region identification model 192 in advance. The fluorescence region identification model 192 is a machine learning model that identifies fluorescence regions 40a in the input image data 40. Specifically, as shown in Figure 4, first, a document written with a fluorescent pen is scanned by the image scanner 13 with a light intensity lower than normal (indicated by 111). Next, training data (training data) is created by adding two attributes, "fluorescence region 40a" and "non-fluorescent region (non-fluorescent region 40b)," to the scanned image data 60 (indicated by 112). Then, this training data is accumulated to create the fluorescence region identification model 192 (indicated by 113).
[0039] The image processing device 1 identifies the fluorescent color regions 40a in the input image data 40 using the fluorescent color region identification model 192. Specifically, as shown in Figure 5, first, the original document is scanned by the image scanner 13 with a light intensity lower than normal (reference numeral 114). The input image data 40 acquired by the image scanner 13 after scanning the original document is input to the fluorescent color region identification unit 110. The fluorescent color region identification unit 110 identifies the fluorescent color regions 40a in the input image data 40 acquired by the image scanner 13. Specifically, the fluorescent color region identification unit 110 identifies the fluorescent color regions 40a in the input image data 40 using the fluorescent color region identification model 192. More specifically, first, the fluorescent color region identification unit 110 inputs the input image data 40 to the fluorescent color region identification model 192 for comparison (see reference numeral 115). Next, the fluorescence color region identification unit 110 identifies the fluorescence color region 40a in the input image data 40 by referring to the training data accumulated in the fluorescence color estimation model 191 (see reference numeral 116). This makes it possible to identify the fluorescence color region 40a and the non-fluorescent color region 40b in the input image data 40. As described above, the fluorescent color region identification unit 110 identifies the fluorescent color region 40a using the fluorescent color region identification model 192.
[0040] (Example 4) Figure 6 shows the configuration of the image processing device 1 according to Example 2. First, the fluorescent color region identification unit 110 receives input image data 40 from the image scanner 13 that has scanned the original document. The fluorescent color region identification unit 110 uses the fluorescent color region identification model 192 to identify the fluorescent color region 40a in the input image data 40. After identifying the fluorescent color region 40a using the fluorescent color region identification model 192, it checks whether correction is necessary (see reference numeral 117). The result of identifying the fluorescent color region 40a using the fluorescent color region identification model 192 is previewed on the operation panel 14, as shown in Figure 7. If the user determines that correction of the identification result is necessary, they correct the fluorescent color region 40a on the operation panel 14. This corrects the fluorescent color region 40a in the input image data 40, and the region attribute of the input image data 40 is corrected (see reference numeral 118). When the user presses the learning button B1 on the operation panel 14, the main controller 11 feeds back the correction result (output image data with the corrected region attribute) to the fluorescent color region identification model 192 (see reference numeral 119). In other words, if the main controller 11 determines that the fluorescent region 40a identified by the fluorescent region identification unit 110 is inappropriate, it uses the input image data 40 and the output image data to which the corrected region attributes have been added to further train the fluorescent region identification model 192. This updates the fluorescent region identification model 192. In other words, the main controller 11 functions as the first update unit of the present invention. Subsequently, the fluorescent region identification model 192, after feedback, is used to identify the fluorescent region 40a in the input image data 40 again. This makes it possible to identify the fluorescent region 40a and the non-fluorescent region 40b in the input image data 40. After that, the synthesized data 50 is created in the same manner as in Example 1.
[0041] (Example 5) Figure 8 shows an example of a method for creating a fluorescence color estimation model 191 in advance.
[0042] The image processing device 1 prepares a fluorescence color estimation model 191 in advance. The fluorescence color estimation model 191 is a machine learning model that outputs fluorescent pen color image data (fluorescence color image data 41) that matches the image of the fluorescence color region 40a. Specifically, as shown in Figure 8, first, the original document written with a fluorescent pen is scanned by the image scanner 13 with the light intensity reduced from normal (indicated by 121). Next, the first definition unit 130 defines the color of the fluorescence color region 40a in the scanned image data 60 as a single class, even if there are differences in the corresponding fluorescent pen colors depending on the manufacturer and paper type, and defines it as input data for the fluorescence color estimation model 191. Next, the second definition unit 140 defines the fluorescent pen color image data that matches the image data 60 acquired by the image scanner 13 as a single color, and defines it as output data for the fluorescence color estimation model 191. In other words, the fluorescent pen color image data corresponding to the input fluorescence color can be pre-associated as output data. Next, the learning data creation unit 150 creates learning data (training data) from the input data defined in the first definition unit 130 and the output data defined in the second definition unit 140. Then, it accumulates this learning data to create a fluorescence color estimation model 191 (indicated by 122).
[0043] For example, the first definition unit 130 defines the "blue" highlighter colors of manufacturers "aaa", "bbb", "cccc", etc. as a single class (blue). The second definition unit 140 defines the "blue" image data that matches the image data 60 as "blue". In this case, the fluorescence color estimation model 191 outputs image data of the highlighter color (blue) that matches the "blue" image of the fluorescence color region 40a in the input image data 40. This makes it possible to output the same image data corresponding to the highlighter color even if there are differences in highlighter colors depending on the manufacturer.
[0044] The image processing device 1 creates fluorescent color image data 41 from an image of the fluorescent color region 40a using a fluorescent color estimation model 191. Specifically, first, the original document is scanned by the image scanner 13 with a light intensity lower than normal. The image of the fluorescent color region 40a identified in the input image data 40 obtained in this way is input to the fluorescent color estimation model 191. Then, the image data of the fluorescent pen color that matches the image of the fluorescent color region 40a (fluorescent color image data 41) is estimated and output (created).
[0045] (Input data in the training data) Furthermore, the first definition unit 130 may define, in addition to the color information (RGB values) of the image data acquired by the image scanner 13, at least one piece of information from among the highlighter manufacturer, highlighter color, paper manufacturer, and paper type as input data. That is, at least one piece of information from among the highlighter manufacturer, highlighter color, paper manufacturer, and paper type may be added as input data to the training data. The highlighter color is the color indicated on the product (highlighter) by the manufacturer, for example, pink, yellow, yellow-green, light blue, purple, orange, etc. The paper type is for example, fine paper, medium-quality paper, art paper, matte paper, recycled paper, etc. The above additional information is quantified and used as a feature.
[0046] (Example 6) Figure 9 shows the configuration of the image processing apparatus 1 according to Example 6. First, the fluorescent color region identification unit 110 receives input image data 40 from the image scanner 13 that has scanned the original document. The fluorescent color region identification unit 110 uses the fluorescent color region identification model 192 to identify the fluorescent color region 40a in the input image data 40. This allows the fluorescent color region 40a and the non-fluorescent color region 40b in the input image data 40 to be identified. The composite data creation unit 120 compares the image of the fluorescent color region 40a with the fluorescent color estimation model 191 to create fluorescent color image data 41 (see reference numerals 101 and 102). After creating the fluorescent color image data 41 using the fluorescent color estimation model 191, it is checked whether correction is necessary (see reference numeral 123). The fluorescent color image data 41 created using the fluorescent color estimation model 191 is previewed on the operation panel 14. If the user determines that correction is necessary for the fluorescent color image data 41, they correct the highlighter color in the fluorescent color image data 41 on the operation panel 14. This corrects the highlighter color in the fluorescent color image data 41 (see reference numeral 124). When the user presses the learning button B1 on the operation panel 14, the main controller 11 feeds back the correction result (corrected output image data) to the fluorescence color estimation model 191 (see reference numeral 125). That is, if the main controller 11 determines that the fluorescence color image data 41 created by the composite data creation unit 120 is inappropriate, it uses the input image data 40 and the corrected output image data to further train the fluorescence color estimation model 191. This updates the fluorescence color estimation model 191. In other words, the main controller 11 functions as the second update unit of the present invention. Subsequently, the fluorescence color estimation model 191 after the feedback is used to create the fluorescence color image data 41 again. After that, composite data 50 is created in the same manner as in Example 1.
[0047] (Collection of additional training data) If the accuracy of the fluorescence color estimation model 191 or the fluorescence color region identification model 192 is insufficient, more training data can be collected and additional training can be performed. For example, the main controller 11 collects additional training data (data for additional training of each machine learning model 19) from data on the cloud. In this case, the main controller 11 functions as the data collection unit of the present invention.
[0048] (Selection of additional training data) Furthermore, it is preferable to train the model with additional training data that is significant (highly reliable) as training data. Therefore, it is preferable that the additional training data includes information such as the manufacturer of the highlighter, the color of the highlighter, the manufacturer of the paper, and the type of paper.
[0049] (Deployment of machine learning models) Generally, each machine learning model 19 (fluorescence color estimation model 191, fluorescence color region identification model 192) is created on the cloud side where high-performance GPU specifications are available. Therefore, generally, each machine learning model 19 is hosted on the cloud. In particular, if the machine learning model 19 is large in size or if the CPU / GPU specifications of the edge device are insufficient, computation is performed on the cloud. However, in cases where it is not possible to connect the cloud and the edge device (each image forming apparatus 10) for security reasons, each machine learning model 19 may be created using the GPU installed in the edge device. In this case, as in this embodiment, the created machine learning model 19 should be installed inside the edge device (image forming apparatus 10) (see Figure 1). This makes it possible to use each machine learning model 19 even when the edge device cannot be connected to the cloud.
[0050] (Handling of image data) Figure 10 is a diagram illustrating the handling of image data in the image processing device 1. Normally, in image processing for image formation, images scanned by the image scanner 13 are discarded when the image data is compressed. This is done to avoid unnecessarily increasing the use of memory by discarding the original image data. However, if corrections are made after identifying the fluorescent color region or after estimating the fluorescent color, irreversible compression (high-compression PDF) is performed, requiring rescanning. Therefore, in this embodiment, first, the scan data (input image data 40) read by the image scanner 13 is temporarily stored in the storage unit (storage 18) (see reference numeral 126). The scan data is then retained until it is determined that no correction is needed to the fluorescent color image data 41 created using the fluorescent color estimation model 191. Then, the composite data creation unit 120 discards the scan data when it determines that no correction is needed to the created fluorescent color image data 41 (see reference numeral 127).
[0051] (Method for temporarily saving image data when using highly compressed PDFs) Figure 11 illustrates how the image processing device 1 temporarily stores image data when creating a highly compressed PDF. In PDF and highly compressed PDF, each image data is stored as multiple objects. As shown in Figure 11, areas other than the fluorescent color area 40a (non-fluorescent color area 40b) are stored as one object. On the other hand, if there are multiple fluorescent color areas 40a on one page, each fluorescent color area 40a is defined as one object, and multiple objects are stored. When corrections are made after fluorescent color area identification or fluorescence color estimation while the scanned data (input image data 40) is stored, only the objects whose attributes are subject to correction are changed. At this time, by replacing the data only for the changed objects, unnecessary objects are not retained. That is, when the main controller 11 updates the image data by correcting the attributes of the fluorescent color area 40a, it replaces the data only for the areas whose attributes have been corrected. In other words, the main controller 11 functions as the third update unit of the present invention. Here, updating the image data by correcting the attributes of the fluorescent color area 40a means updating the attributes of each area of the image data by identifying (correcting) the fluorescent color area 40a using the fluorescent color area identification model 192 that reflects the corrections made after fluorescent color area identification. Alternatively, the attributes of each region of the image data can be updated by estimating (correcting) the fluorescence color in the fluorescence region 40a using the fluorescence color estimation model 191, which incorporates corrections made after fluorescence color estimation. This eliminates the need for a new scan, thereby reducing memory usage and processing time.
[0052] As described above, the image processing apparatus 1 according to this embodiment comprises an image acquisition unit (image scanner 13), a fluorescent color area identification unit 110, and a composite data creation unit 120. The image acquisition unit acquires image data. The fluorescent color area identification unit 110 identifies the fluorescent color areas in the image data acquired by the image acquisition unit. The composite data creation unit 120 creates composite data 50 consisting of fluorescent color image data 41 created for the image of the fluorescent color area 40a identified by the fluorescent color area identification unit 110, and non-fluorescent color image data 42 created for the image of a region different from the fluorescent color area 40a. The fluorescent color image data 41 is created using a first machine learning model (fluorescent color estimation model 191) that outputs image data of a fluorescent pen color that matches the image of the fluorescent color area 40a. The non-fluorescent color image data 42 is created using a color reproduction method for the image of a region different from the fluorescent color area 40a. Therefore, according to the image processing device 1 of this embodiment, high-saturation / brightness fluorescent pen colors that could not be expressed before can be reproduced using a pre-created machine learning model (fluorescent color estimation model 191). Furthermore, by training the fluorescent color estimation model 191 with images of the same color but with different shades as training data, even if the same color is output differently depending on the manufacturer of the fluorescent pen or the type of paper, it is possible to output them all as the same color. Thus, even colors that cannot be expressed by adjusting the correction coefficient (high-saturation fluorescent colors) can be reproduced without depending on the manufacturer or type of paper.
[0053] Furthermore, the fluorescent color region identification unit 110 identifies the fluorescent color region 40a using image data acquired by the image acquisition unit at a first light intensity and image data acquired by the image acquisition unit at a second light intensity which is lower than the first light intensity. Therefore, the fluorescent color region 40a can be accurately identified. Thus, the fluorescent color of the fluorescent color region 40a can be accurately reproduced.
[0054] Furthermore, the fluorescence region identification unit 110 identifies the fluorescence region 40a using a second machine learning model (fluorescence region identification model 192). Therefore, the fluorescent color region 40a can be accurately identified. Thus, the fluorescent color of the fluorescent color region 40a can be accurately reproduced.
[0055] Furthermore, if the fluorescent color region 40a identified by the fluorescent color region identification unit 110 is deemed inappropriate, the system includes a first update unit (main controller 11) that uses the input image data and output image data to further train the second machine learning model and update the second machine learning model. The input image data is acquired by the image acquisition unit. The output image data has the corrected region attributes assigned to it. Therefore, the accuracy of identifying the fluorescent color region 40a by the second machine learning model can be improved. As a result, the fluorescent color of the fluorescent color region 40a can be reproduced with high accuracy.
[0056] The system also includes a first definition unit 130, a second definition unit 140, and a training data creation unit 150. The first definition unit 130 defines the colors of the fluorescent color region 40a as a single class, even if there are differences in the corresponding fluorescent pen colors depending on the manufacturer or paper type, and defines them as input data for the first machine learning model. The second definition unit 140 defines the image data of fluorescent pen colors that match the image data acquired by the image acquisition unit as a single color, and defines it as output data for the first machine learning model. The training data creation unit 150 creates training data from the input data defined in the first definition unit 130 and the output data defined in the second definition unit 140. Therefore, it is possible to output image data of the appropriate fluorescent pen color for the color of the fluorescent color region 40a in the input image data 40. Thus, the fluorescent color of the fluorescent color region 40a can be reproduced with high accuracy.
[0057] Furthermore, the first definition unit 130 defines, in addition to the color information of the image data acquired by the image acquisition unit, at least one of the following pieces of information as input data: the manufacturer of the highlighter, the color of the highlighter, the manufacturer of the paper, and the type of paper. Therefore, when identifying the fluorescent color region 40a in the input image data 40, a lot of information can be referenced to accurately identify the fluorescent color region 40a. Thus, the fluorescent color of the fluorescent color region 40a can be accurately reproduced.
[0058] Furthermore, if the fluorescent image data 41 is deemed inappropriate, a second update unit (main controller 11) is included that uses the input image data and output image data to further train the first machine learning model and update the first machine learning model. The fluorescent image data 41 is created by the composite data creation unit 120. The input image data is acquired by the image acquisition unit. The output image data is the corrected version. Therefore, the accuracy of the fluorescence color estimation by the first machine learning model can be improved. Thus, the fluorescence color of the fluorescence region 40a can be reproduced with high accuracy.
[0059] It also includes a data collection unit (main controller 11) that collects additional training data from data on the cloud. Therefore, the accuracy of the fluorescence color estimation by the first machine learning model can be improved. Furthermore, the accuracy of the identification of the fluorescence region 40a by the second machine learning model can be improved. Thus, the fluorescence color of the fluorescence region 40a can be reproduced with high accuracy.
[0060] Furthermore, the additional training data includes data that has at least one piece of information attached to it, such as the manufacturer of the highlighter, the color of the highlighter, the manufacturer of the paper, and the type of paper. Therefore, the accuracy of the fluorescence color estimation by the first machine learning model can be further improved. Furthermore, the accuracy of the identification of the fluorescence region 40a by the second machine learning model can be further improved. Thus, the fluorescence color of the fluorescence region 40a can be reproduced with greater accuracy.
[0061] Furthermore, the first machine learning model is located in the cloud or inside an edge device. Similarly, the second machine learning model is also located in the cloud or inside an edge device. Therefore, the deployment of each machine learning model can be adjusted according to the situation, such as security and the CPU / GPU specifications of edge devices. This allows for optimal operation of each machine learning model.
[0062] The system also includes a storage unit (storage 18) for temporarily storing image data acquired by the image acquisition unit. The composite data creation unit 120 discards the image data stored in the storage unit when it determines that no correction is needed to the fluorescent color image data created using the first machine learning model. Therefore, it is possible to avoid rescanning the image. Thus, the user's burden in estimating the fluorescence color and identifying the fluorescence color region 40a can be reduced.
[0063] Furthermore, when updating image data by correcting the attributes of the fluorescent color region 40a, the system includes a third update unit (main controller 11) that replaces the data only in the region whose attributes have been corrected. Therefore, it is possible to reduce memory usage and shorten processing time. Consequently, the load required for estimating fluorescence color and identifying the fluorescence color region 40a can be reduced.
[0064] Although the present invention has been specifically described above based on embodiments, the present invention is not limited to the above embodiments and can be modified without departing from its spirit.
[0065] For example, in the above embodiment, a method for identifying the fluorescent color region 40a is provided, which involves using input image data 40 and 43 acquired at different light intensities to identify the fluorescent color region 40a, but the method is not limited thereto. The method for identifying the fluorescent color region 40a is not particularly limited, and for example, the means described in Japanese Patent Application Publication No. 2-170673 or the means described in Japanese Patent Application Publication No. 7-23210 may be used.
[0066] Furthermore, when creating training data by adding attributes for fluorescent color regions 40a and non-fluorescent color regions 40b to the scanned image data 60, the user may be allowed to manually label the data. Furthermore, when associating the image data of the fluorescent pen colors that match the image data 60 acquired by the image scanner 13 with each fluorescent pen color in the image data 60, the user may be allowed to manually label them.
[0067] Furthermore, the detailed configuration and detailed operation of each device constituting the image forming apparatus can also be modified as appropriate without departing from the spirit of the present invention. [Explanation of symbols]
[0068] 1 Image Processing Device 10 Image forming apparatus 11. Main Controller (1st Update Unit, 2nd Update Unit, Collection Unit, 3rd Update Unit) 12 ADF 13. Image scanner (image acquisition unit) 14. Control Panel 15 Communication Interface 16 Engine Controller 17 Printer Engine 18. Storage (memory unit) 19 Machine Learning Models 191 Fluorescence Color Estimation Model (First Machine Learning Model) 192 Fluorescence Color Region Identification Model (Second Machine Learning Model) 110 Fluorescent color area identification section 120 Composite Data Creation Section 130 1st definition part 140 Second definition part 150 Training Data Creation Section 40 Input image data 40a Fluorescent color area 40b Non-fluorescent color region 41 Fluorescent color image data 42 Non-fluorescent image data 50 Composite Data
Claims
1. Image acquisition unit that acquires image data, A fluorescent color region identification unit identifies the fluorescent color region in the image data acquired by the image acquisition unit, A composite data creation unit creates composite data of fluorescent color image data created for an image of a fluorescent color region identified by the fluorescent color region identification unit and non-fluorescent color image data created for an image of a region different from the fluorescent color region. Equipped with, The aforementioned composite data creation unit, A fluorescent color image data created using a first machine learning model that outputs fluorescent pen color image data that matches the image of the fluorescent color region, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, An image processing apparatus characterized by creating composite data.
2. The image processing apparatus according to claim 1, characterized in that the fluorescent color region identification unit identifies the fluorescent color region using image data acquired by the image acquisition unit at a first light intensity and image data acquired by the image acquisition unit at a second light intensity which is lower than the first light intensity.
3. The image processing apparatus according to claim 1, characterized in that the fluorescent color region identification unit identifies the fluorescent color region using a second machine learning model.
4. The image processing apparatus according to claim 3, further comprising a first update unit that, when the fluorescent color region identified by the fluorescent color region identification unit is deemed inappropriate, uses the input image data acquired by the image acquisition unit and the output image data to which the corrected region attributes have been assigned to further train the second machine learning model and updates the second machine learning model.
5. Regarding the colors in the fluorescent color range, even if there are differences in the corresponding highlighter colors depending on the manufacturer or paper type, they are defined as a single class and defined as input data for the first machine learning model in the first definition unit, A second definition unit defines the image data of the fluorescent pen color that matches the image data acquired by the image acquisition unit as a single color, and defines it as the output data of the first machine learning model, A learning data creation unit that creates learning data from the input data defined in the first definition unit and the output data defined in the second definition unit, The image processing apparatus according to claim 1, characterized by comprising:
6. The image processing apparatus according to claim 5, characterized in that the first definition unit defines, in addition to the color information of the image data acquired by the image acquisition unit, at least one of the following pieces of information as input data: the manufacturer of the highlighter, the color of the highlighter, the manufacturer of the paper, and the type of paper.
7. The image processing apparatus according to claim 5, further comprising a second update unit that, when the fluorescent color image data created by the composite data creation unit is deemed inappropriate, uses the input image data acquired by the image acquisition unit and the corrected output image data to further train the first machine learning model and update the first machine learning model.
8. The image processing apparatus according to claim 4 or 7, characterized by comprising a collection unit for collecting additional training data from data on the cloud.
9. The image processing apparatus according to claim 8, characterized in that the additional training data is data to which at least one piece of information is attached, selected from the manufacturer of the highlighter, the color of the highlighter, the manufacturer of the paper, and the type of paper.
10. The image processing apparatus according to claim 1, characterized in that the first machine learning model is provided on the cloud or inside an edge device.
11. The image processing apparatus according to claim 3, characterized in that the second machine learning model is provided on the cloud or inside an edge device.
12. The system includes a storage unit for temporarily storing image data acquired by the image acquisition unit, The image processing apparatus according to claim 1, characterized in that the composite data creation unit discards the image data stored in the storage unit at the timing when it determines that no correction is needed to the fluorescent color image data created using the first machine learning model.
13. The image processing apparatus according to claim 1, further comprising a third update unit that replaces data only in the areas whose attributes have been modified when updating the image data by modifying the attributes of the fluorescent color area.
14. An image processing method for an image processing apparatus equipped with an image acquisition unit that acquires image data, A fluorescent color region identification step for identifying the fluorescent color region in the image data acquired by the image acquisition unit, A composite data creation step, which creates composite data of fluorescent color image data created for an image of a fluorescent color region identified in the fluorescent color region identification step and non-fluorescent color image data created for an image of a region different from the fluorescent color region, Includes, The aforementioned synthesis data creation process is as follows: A fluorescent color image data created using a first machine learning model that outputs fluorescent pen color image data that matches the image of the fluorescent color region, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, An image processing method characterized by creating composite data.
15. A computer for an image processing device equipped with an image acquisition unit that acquires image data, A fluorescent color region identification unit identifies the fluorescent color region in the image data acquired by the image acquisition unit. A composite data creation unit creates composite data of fluorescent color image data created for an image of a fluorescent color region identified by the fluorescent color region identification unit and non-fluorescent color image data created for an image of a region different from the fluorescent color region. To make it function as, The aforementioned composite data creation unit, A fluorescent color image data created using a first machine learning model that outputs fluorescent pen color image data that matches the image of the fluorescent color region, Non-fluorescent color image data created using a color reproduction method for an image in a region different from the aforementioned fluorescent color region, A program characterized by creating composite data.