Image processing system, reading device, and method for controlling image processing system

Abstract

To solve the problem in which conventionally, read image data of a document output from a scanner is, in many cases, image data on which hue adjustment or the like are performed according to settings made by a user or the lie, and the read image data may thus not be appropriate to be used as input data for machine learning, and may affect the accuracy of machine learning.SOLUTION: An image processing system acquires, with a linear RGB image acquisition unit 360, image data (linear RGB image) read by a scanner device 100 and before being processed by a second image processing unit 380 and outputs the image data to a PC 200. The PC 200 performs machine learning by using the image data (linear RGB image) acquired from the scanner device 100 as data for machine learning.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image processing system, a reading device, and a control method for an image processing system. [Background technology]

[0002] Image recognition (classification, etc.) using machine learning models is becoming more widely used as its performance improves. In Patent Document 1, learning is performed using read image information as input data, page information of the document is confirmed, and the presence or absence of missing pages is determined based on the results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-57710 Summary of the Invention [Problem to be solved by the invention]

[0004] Scanned image data of a document output from an image reading device is often image data that has been image-processed according to image processing settings (including initial settings) made by a user or the like. For example, the image data is often image data in which color has been adjusted, such as by changing the contrast or performing color dropout processing. Even if a user does not intentionally configure image processing settings, in many cases, the initial settings are set to, for example, make the image appear natural to the human eye, or image processing is performed to reduce data size. Therefore, while such image data may be an image desired by the user, it may not be suitable for use as input data (learning data) for machine learning, which could affect the accuracy of machine learning.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide a mechanism that can improve the accuracy of machine learning performed using image data read from a document or the like as input data. [Means for solving the problem]

[0006] The present invention is characterized by comprising a reading device having a generation means for reading a document or subject with a reading device and generating image data for each color component that constitutes a color image, a modification processing means for performing processing to modify the image data in accordance with a setting value that can be set by the user, an acquisition means for acquiring image data before it is processed by the modification processing means, and an output means for outputting at least one of the image data processed by the modification processing means and the image data acquired by the acquisition means, and an information processing device having a learning means for performing machine learning using the image data acquired and output by the acquisition means as learning data. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the accuracy of machine learning that uses image data read from a document or the like as input data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of a sheet-fed scanner according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing an example of a control configuration of a scanner. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a PC. [Figure 5] Block diagram showing an example of the configuration of machine learning running on a PC. [Figure 6] FIG. 1 is a diagram showing an example of the configuration of an application and a scanner driver that run on a PC. [Figure 7]FIG. 2 is a block diagram showing an example of the configuration of an image processing unit of a scanner. [Figure 8] FIG. 2 is a diagram showing an example of the detailed configuration of an image processing unit of the scanner. [Figure 9] FIG. 10 is a diagram showing an example of a user gamma setting screen. [Figure 10] A conceptual diagram showing an example of the input / output configuration when using a learning model and a trained model. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the invention will be described with reference to the drawings. An image processing system including an image reading device and an information processing device connectable to the image reading device will be described below.

[0010] [First embodiment] FIG. 1 is a diagram showing an example of an image processing system according to an embodiment of the present invention. In FIG. 1, scanner 100 is an image reading device that reads image data from a document. Scanner 100 is communicably connected to a personal computer (PC) 200. Scanner 100 and PC 200 may be connected via a communication interface such as USB (Universal Serial Bus), or may be connected via a network. The communication interface may be wired or wireless, such as wireless USB, Bluetooth (registered trademark), or NFC (Near Field Communication). The network may be either wired or wireless. The PC 200 is an information processing device that controls the scanner 100. The PC 200 is not limited to a personal computer, but may also be a tablet terminal, a smartphone, or the like.

[0011] The configuration and operation of the scanner 100 will be described below with reference to FIG. 2 is a diagram showing an example of the configuration of a sheet-fed scanner as an example of the scanner 100. In this embodiment, a sheet-fed scanner that sequentially scans stacked documents (hereinafter referred to as "sheets") will be used for explanation, but the present invention is not limited to this.

[0012] As shown in FIG. 2, the scanner 100 includes an automatic document feeder (ADF) 101. A plurality of sheets are stacked on a sheet stacking table 1 of the ADF 101, and the sheet stacking table 1 is configured to be movable up and down. A sheet stacking table driving motor (not shown) moves the sheet stacking table 1 up and down.

[0013] The sheet detection sensor 3 detects that the sheets stacked on the sheet stacking tray 1 are at the sheet intake position. The sheet stacking detection sensor 12 detects that the sheets are stacked on the sheet stacking surface 1a of the sheet stacking tray 1.

[0014] Further, the document bounce detection sensor 35 includes a plurality of sensors arranged in a direction perpendicular to the sheet stacking surface 1a, and detects any bounce of sheets stacked on the sheet stacking tray 1. For example, the document bounce detection sensor 35 can detect any bounce of a document that occurs when stapled documents are stacked on the sheet stacking tray 1 and fed. This makes it possible to perform control such as stopping the feeding of stapled documents.

[0015] A pickup roller 4, which is an example of a sheet pickup section, feeds a sheet on the sheet stacking tray 1 from the sheet stacking tray 1. A pickup roller drive motor (not shown) rotates the pickup roller 4 in the direction to pick up the sheet. The pickup roller 4 can be moved to a sheet pick-up position and a retracted position above the sheet pick-up position. The pickup roller 4 is moved to the sheet pick-up position when picking up a sheet, and is moved to the retracted position after pick-up is complete. The pickup roller 4 plays an auxiliary role in ensuring that separation and feeding by a pair of separation rollers 42, which will be described later, is performed reliably. If the pickup roller 4 sends the sheet on the sheet stacking tray 1 into the nip portion of the pair of separation rollers 42, separation and feeding by the pair of separation rollers 42 can be performed reliably.

[0016] The separation roller pair 42 is composed of a feeding roller 6 and a separation roller 7 . In the separation roller pair 42, the feed roller 6 is driven by a feed motor (not shown) to rotate in a direction (feed direction) to feed the sheet downstream in the conveying direction. The separation roller 7 constantly receives a rotational force from the separation motor (not shown) via a torque limiter (slip clutch) (not shown) to rotate in a direction to push the sheet back upstream in the conveying direction.

[0017] When one sheet is present between the feed roller 6 and the separation roller 7, the frictional force between the sheet sent downstream by the feed roller 6 and the separation roller 7 causes the rotational force in the direction in which the sheet is fed downstream to exceed the upper limit of the rotational force in the direction in which the separation roller 7 pushes the sheet back upstream, which is transmitted by the torque limiter. Therefore, the separation roller 7 rotates following the feed roller 6.

[0018] On the other hand, when there are multiple sheets between the feeding roller 6 and the separation roller 7, the separation roller 7 receives rotation from the roller shaft in a direction that pushes the sheets back upstream, so that only the topmost sheet is transported downstream.

[0019] In this way, overlapping sheets are separated and fed by the action of the feed roller 6 to feed the sheets downstream and the action of the separation roller 7 to prevent the sheets from being transported downstream. Even if overlapping sheets are fed into the nip portion (contact portion between the feed roller 6 and the separation roller 7) formed between the feed roller 6 and the separation roller 7, only the topmost sheet is fed downstream, and the other sheets are not transported downstream.

[0020] The feed roller 6 and the separation roller 7 constitute a pair of separation rollers 42 (sheet separation unit). In this embodiment, the separation roller pair 42 is used, but instead of the separation roller pair 42, a separation belt roller pair in which either the separation roller or the feed roller is a belt may be used. Also, the separation roller may be replaced with a separation pad that abuts against the sheet to prevent multiple sheets from being transported downstream. Also, the separation roller 7 may be used so that it abuts against the sheet like a separation pad without rotating.

[0021] The sheet pickup unit, which is configured as described above and is composed of the pickup roller 4, the feeding roller 6, the separation roller 7, etc., separates the sheets stacked on the sheet stacking tray 1 one by one and takes them into the scanner 100.

[0022] Furthermore, by providing the double feed detection sensor 30 at a position where the separated sheets pass (that is, downstream of the separation roller pair 42), it is possible to detect whether the sheets have been separated one by one by the sheet separation unit.

[0023] A conveying motor (not shown) drives other rollers (sheet conveying section) to convey the separated sheet to an image reading position where the image on the sheet is read by image reading sensors 14 and 15, and further to a discharge position. The conveying motor also drives each roller so that the sheet conveying speed can be changed according to settings such as the optimum speed for reading the sheet and the sheet resolution.

[0024] A nip gap adjustment motor (not shown) adjusts the gap between the feed roller 6 and the separation roller 7, or the pressure contact force (nip pressure) with which the feed roller 6 presses against the separation roller 7 via the sheet. This adjusts the gap or pressure contact force to suit the thickness of the sheet, allowing the sheet to be separated.

[0025] A resist clutch (not shown) transmits or blocks the rotational driving force of the conveying motor to the resist rollers 18 (sheet conveying section). By stopping the rotation of the first resist roller pair consisting of the resist rollers 17 and 18, the leading edge of the fed sheet may be abutted against the nip portion of the resist roller pair to correct skew of the sheet.

[0026] A second registration roller pair consisting of registration rollers 20 and 21, a conveying roller pair consisting of conveying rollers 22 and 23, a conveying roller pair consisting of conveying rollers 24 and 25, and a paper discharge roller pair consisting of paper discharge rollers 26 and 27 convey the sheet, and finally convey it to a discharge stacking section 44. A paper discharge sensor 16 detects the passage of the conveyed sheet. The two guide plates, the upper guide plate 40 and the lower guide plate 41, guide the sheets conveyed by the separation roller pair, the registration roller pair, each conveying roller pair, and the paper discharge roller pair.

[0027] The pre-registration sensor 32 is disposed upstream of the first pair of registration rollers 17, 18 and detects the sheet being fed. The post-registration sensor 34 is disposed downstream of the second pair of registration rollers 20, 21 and detects the sheet being conveyed. Furthermore, the mid-registration sensor 33 is disposed downstream of the first pair of registration rollers 17, 18 and upstream of the second pair of registration rollers 20, 21 and detects the sheet being conveyed.

[0028] When the post-registration sensor 34 detects the sheet, a control unit (CPU 306, described later) issues an image reading instruction to the image reading sensors 14 and 15, and the image on the conveyed sheet is read. The image reading sensor 14 reads the front side of the sheet, and the image reading sensor 15 reads the back side of the sheet simultaneously. The image reading sensors 14 and 15 are provided with platen rollers 14a and 15a, respectively, so that the sheet can be read while being brought into close contact with the image reading sensors 14 and 15. The sheet images read by the image reading sensors 14 and 15 are transmitted to an external device such as an information processing device (PC 200 in the example of FIG. 1) via an interface unit (not shown).

[0029] FIG. 3 is a block diagram showing an example of the control configuration of the scanner 100. As shown in FIG. A / D converters (ADC) 301a and 301b perform analog processing such as amplification and black level clamping on the output signals of the line image sensors (image reading sensors 14 and 15 in FIG. 2), and then convert them into digital data (image data). The image processing unit 302 controls the line image sensors 14 and 15 and the ADCs 301a and 301b, and performs various image processing operations on the image data output from the ADCs 301a and 301b. The image processing unit 302 has an internal image processing buffer that stores image data.

[0030] The I / F 304 is an interface for communicating with an external host device (such as the PC 200). The I / F 304 is connected to the external host device (such as the PC 200 in the example of FIG. 1) via a signal cable 305. For example, the I / F 304 is a USB interface. The I / F 304 may be a wired interface such as a USB interface or a wired LAN interface, or a wireless interface such as a wireless LAN, wireless USB, Bluetooth (registered trademark), or NFC.

[0031] The CPU 306 is a control unit that controls the scanner 100. The image processing unit 302 and the CPU 306 are connected via a bus 307. The CPU 306 accesses the RAM 303 and the ROM 310 via the bus 307. The ROM 310 stores programs and various data. The RAM 303 functions as a work memory for the CPU 306.

[0032] The drive unit 309 is a motor or the like for driving various loads such as the pickup roller 4, the feeding roller 6, and the separation roller 7. The motor driver 308 is a control circuit that controls the drive unit 309 based on instructions from the CPU 306.

[0033] FIG. 4 is a block diagram showing an example of the configuration of a PC 200, which is an example of an external host device connected to the scanner 100. As shown in FIG. The CPU 406, based on a computer program, comprehensively controls each unit of the PC 200. The CPU 406 controls the scanner 100 in accordance with an application 501 and a scanner driver 502 shown in FIG.

[0034] The ROM 401 is a non-volatile storage unit that stores control programs such as firmware. The RAM 402 is a volatile storage unit that functions as a work area. The hard disk drive (HDD) 403 is a large-capacity storage unit. Note that the configuration may include other storage devices such as an SSD (Solid State Drive) or an eMMC (Embedded MultiMediaCard) instead of or in addition to the HDD (Hard Disk Drive).

[0035] The display device 405 is a display unit for displaying various information to the user. The operation unit 407 is an input unit such as a pointing device or keyboard. The communication I / F 404 is a communication unit such as a USB. The CPU 406 communicates with the scanner 100 via the communication I / F 404. For example, the communication I / F 404 is a USB interface. The communication I / F 404 may be a wired interface such as a USB interface or a wired LAN interface, or a wireless interface such as a wireless LAN, wireless USB, Bluetooth (registered trademark), or NFC.

[0036] The following describes where the learning model is provided in the image processing system of this embodiment and the processing using the learned model. FIG. 5 is a block diagram showing an example of the configuration of machine learning that runs on the PC 200. The HDD 403 of the PC 200 stores a training data generation unit 250, a learning unit 251, a training model 252, a data collection and provision unit 253, an inference unit 254, a trained model 255, etc. in a computer-readable manner. Of these, the training data generation unit 250, the learning unit 251, the data collection and provision unit 253, and the inference unit 254 are programs that the CPU 406 loads into the RAM 303 and executes to realize various functions on the PC 200. The HDD 403 also stores various other programs and data in a computer-readable manner, including an application 501, a scanner driver 502, an operating system (OS), etc., which will be described later and are shown in FIG. 6 .

[0037] The learning data generation unit 250 is a module that generates learning data for the learning model 252 from data received from outside. The learning data is teacher data T, which is a set of input data X and a label LBL that indicates the correct answer of the learning result for the input data X. The learning data generation unit 250 supplies the generated learning data to the learning unit 251.

[0038] The learning unit 251 is a program module that executes learning of the learning model 252 using the learning data generated by the learning data generation unit 250. The learning model 252 accumulates the results of learning performed by the learning unit 251. Here, the learning model 252 is implemented using a neural network. Learning of the learning model 252 is executed by optimizing the weighting parameters between each node of the neural network using the learning data by a known method. The learning model 252 (i.e., the trained model) for which parameter optimization (learning) has been completed is held as the trained model 255.

[0039] The data collection and provision unit 253 is a module that stores data received from the scanner 100 in the HDD 403 as data for generating learning data. The inference unit 254 is a program module that provides input data based on data received from the scanner 100 to the trained model 255, executes inference processing, and outputs the results of the inference processing.

[0040] In this embodiment, the location of the learning model in the HDD 403 and the location of the processing using the trained model are shown. However, the location of the learning model and the location of the processing using the trained model may be changed. For example, the learning model and the inference processing using the learning model may be performed on a cloud server (not shown). For example, it can be determined whether to deploy the learning model on the PC 200 based on the relationship between the processing speed and power consumption required for calculations related to the learning model and the hardware resources of the PC 200. If the learning model cannot be deployed on the PC 200 or deployment is undesirable, the learning model is deployed on an external device.

[0041] FIG. 6 is a diagram showing an example of the configuration of an application and a scanner driver that run on the PC 200. As shown in FIG. The application 501 and the scanner driver 502 are program modules installed in the PC 200. That is, the application 501 and the scanner driver 502 are realized by the CPU 406 of the PC 200 loading the programs stored in the HDD 403 into the RAM 402 as needed and executing them.

[0042] The application 501 is a program capable of inputting and outputting information to and from the scanner driver 502, and specific examples thereof include an image editing program, an album program, a text editing program, etc. Note that the application 501 is not limited to these, and any application that receives image data from the scanner driver 502 may be used.

[0043] The application 501 controls the scanner 100 via the scanner driver 502 and performs image processing on image data received from the scanner driver 502. The image processing of the images read by the image reading sensors 14 and 15 of the scanner 100 may be performed on the scanner 100, on the PC 200, or may be performed by both.

[0044] Communication is performed between the application 501 and the scanner driver 502, and between the scanner driver 502 and the scanner 100, using a respective predetermined protocol. For example, the application 501 and the scanner driver 502 communicate using a protocol determined by the TWAIN standard. This protocol determined by the TWAIN standard is a driver interface for communication with the scanner driver 502, and is also an example of an application interface for communication with an application program.

[0045] Communication between the scanner driver 502 and the scanner 100 is performed using a protocol defined by, for example, the USB standard. The protocol defined by the USB standard is used as an interface for communicating with an image reading device. Note that the communication standards and protocols are not limited to these, and other protocols may also be used. Note that control commands and image data are sent and received between the scanner 100 and the scanner driver 502 via these protocols.

[0046] <Image processing unit> The image processing unit 302 of the scanner 100 is described in detail below. FIG. 7 is a diagram showing an example of the configuration of the image processing unit of the scanner 100. As shown in FIG. As shown in Fig. 7, the image processing unit 302 processes the front and back sides of the sheet separately. The image of the front side of the sheet is output from the ADC 301a to the image processing unit 302a. Similarly, the image of the back side of the sheet is output from the ADC 301b to the image processing unit 302b. The image processing units 302a and 302b perform the same processing except that the input images are front and back. When the processing is completed in the image processing units 302a and 302b, the processed image data is transferred to the RAM 303 or the like via the bus 307.

[0047] In this embodiment, the image processing units 302a and 302b are configured to perform the same processing on the front and back sides independently, but they do not necessarily have to be independent. There may be cases where an image of one side is required for processing the other side, and the processing of the front and back sides may mutually use each other's images.

[0048] 8 is a diagram showing an example of the configuration of the image processing unit 302a. The image processing unit 302b has a similar configuration. In FIG. 8, the flow of image data is indicated by solid arrows, and the flow of settings to each processing unit and notifications of determination results is indicated by dotted arrows. The line image sensor 14 can read the three colors R (red), G (green), and B (blue), and the ADC 301a converts the read data into digital image data for each of the R, G, and B components. Images for the three colors R, G, and B are also input to the image processing unit 302a. Note that the image processing unit 302b is similar to the image processing unit 302a, so a description thereof will be omitted.

[0049] In this embodiment, image data output from each processing unit in the image processing unit 302a (such as the shading correction processing unit 350, device gamma correction unit 351, etc.) is configured by pipeline processing so that it is input to another processing unit. However, this configuration may be different. For example, some or all of the processing units may be connected to the RAM 303 via the bus 307. In this case, image data is input to and output from each processing unit via the RAM 303.

[0050] Among the processing units in the image processing unit 302a, the processing units from the shading correction processing unit 350 to the noise removal processing unit 353 are processing units for performing image processing according to the characteristics of reading devices such as the image reading sensors 14 and 15 (for example, image processing that should be performed in common regardless of the user's preference or purpose). The processing units in this range are referred to as a first image processing unit 370. Furthermore, the processing units from the color dropout processing unit 354 to the JPEG conversion processing unit 356 are processing units that perform processing (change processing) including color adjustment processing according to user settings. This range of processing units is referred to as the second image processing unit 380. The processing of the second image processing unit 380 can be changed by user settings according to, for example, the user's preferences or purposes. On the other hand, the processing of the first image processing unit 370 may be changeable by user settings in terms of the degree of noise removal, but color, etc. cannot be changed by user settings.

[0051] Image data for each component separated into R, G, and B is input to the image processing unit 302a. The image data for each component input to the image processing unit 302a is subjected to shading correction for each component by the shading correction processing unit 350. The line image sensor 14 cannot perform uniform image reading as it is due to uneven light intensity caused by the light source and uneven sensitivity of the optical sensor elements. To enable uniform image reading, shading correction is performed to correct the image data obtained from the output value when the sheet is read using the output value when the color reference member is read. The image data after shading correction is output for each R, G, and B component, just like the input. The image data output from the shading correction processing unit 350 is input to a device gamma correction unit 351.

[0052] The device gamma correction unit 351 performs gamma correction for each of the R, G, and B components. To improve the gradation reproducibility of the scanned image, the line image sensor 14 must adjust the device's specific response characteristic (γ: gamma) to approach 1. Therefore, a correction value (gamma curve) for correcting the output value is calculated in advance based on the output value and expected density value information when the sheet is scanned, using a sheet printed with colors that gradually change from white to black, whose density values ​​have been measured in advance. The device gamma correction unit 351 uses this correction value to correct the output value of the shading correction processing unit 350. The image data after device gamma correction is output for each of the R, G, and B components, just like the input. The image data output from the device gamma correction unit 351 is input to a three-dimensional gamma correction processing unit 352.

[0053] The 3D gamma correction processor 352 performs gamma correction for each combination of R, G, and B. To improve the saturation and color reproducibility of the scanned image, the line image sensor 14 uses a sheet printed with multiple colors, each of which has its R, G, and B density values ​​measured in advance, to correct output values ​​based on the output values ​​at the time of sheet scanning and expected density value information. For example, the 3D gamma correction processor 352 corrects the color of the image by determining output values ​​based on the positional relationship between input values ​​and the grid points, based on color correction data (data indicating the correspondence between color separation signals and color correction signals at each grid point of a three-dimensional grid that divides a specified color space). The image data after 3D gamma correction is output for each R, G, and B component, just like the input. The image data output from the 3D gamma correction processor 352 is input to the noise reduction processor 353.

[0054] The noise removal processing unit 353 performs a filter process on the input image data to remove noise components. The image data after noise removal is output for each R, G, and B component, just like the input. The processing in the first image processing unit 370 is not limited to a combination of shading correction processing, device gamma correction processing, three-dimensional gamma correction processing, and noise removal processing, and may include none of the processes or other processes.

[0055] As described above, the processing from the shading correction processing unit 350 to the noise removal processing unit 353 is performed by the first image processing unit 370, which performs image processing according to the characteristics of the reading devices such as the image reading sensors 14 and 15, and the image data that is output is data that has not undergone color adjustment according to user settings (hereinafter referred to as "linear RGB image data"). The image data output from the noise removal processing unit 353 is input to the color dropout processing unit 354.

[0056] The image processing unit of this embodiment is provided with a linear RGB image acquisition unit 360. In this embodiment, the image data output by the noise removal processing unit 353 is acquired by the linear RGB image acquisition unit 360 and used as learning data. The linear RGB image data acquired by the linear RGB image acquisition unit 360 is stored in the RAM 303 via the bus 307, and then transmitted to the PC 200 via the I / F 304. The scanner driver 502 of the PC 200 receives this linear RGB image data from the scanner 100 and saves it in a preset storage location (for example, the HDD 403).

[0057] The processing after the color dropout processing unit 354 will be described below. The color dropout processing unit 354 outputs image data using only the color components specified by the user. For example, if the user specifies "red dropout," image data is output from which color components in the specified range (specified by hue, for example) have been removed (for example, converted to the same value as the background color). If there is no particular specification from the user, this processing moves on to the next processing unit without performing any processing. The image data output from the color dropout processing unit 354 is input to the user gamma correction unit 355.

[0058] The user gamma correction unit 355 performs gamma correction for each of the R, G, and B components. This correction data may be preset or may be freely changed by the user. The same correction data may be set for each of the R, G, and B components, or different settings may be allowed. These setting screens will be described later with reference to FIG. 9. Gamma correction processing is performed using the set data, and image data is output. The image data output from the user gamma correction unit 355 is input to the JPEG conversion processing unit 356.

[0059] The JPEG conversion processor 356 compresses the image data into JPEG format. JPEG compression is often performed in YCbCr format, achieving a high image compression rate while minimizing visual artifacts by thinning the color difference components Cb and Cr to 1 / 2 or 1 / 4. The CPU 306 also sets the compression method in the JPEG conversion processor 356 depending on whether the output image is a color image or a grayscale image. When a color image is output, JPEG compression is performed using the Y, Cb, and Cr components. When a grayscale image is output, compression is performed using only the Y component. The compressed image data is stored in RAM 303 via bus 307 and then transmitted to PC 200 via I / F 304. A scanner driver 502 in the PC 200 receives this image data as scanned image data and passes it to application 501.

[0060] The scanner 100 can output both scanned image data processed by the second image processing unit 380 and linear RGB image data not processed by the second image processing unit 380. Depending on the scan settings from the scanner driver 502, the scanner 100 can selectively output only scanned image data or only linear RGB image data, or can output both scanned image data and linear RGB image data. These modes can be switched by setting the scan mode in the scanner driver 502. For example, in the first mode, only image data processed by the second image processing unit 380 is output. In the second mode, only linear RGB image data is output. In the third mode, both image data processed by the second image processing unit 380 and linear RGB image data are output. These scan modes are included in the scan settings. The scan settings can be set to execute an inference process (described later). The second or third mode can be set when the inference process is executed, and the first mode can be set when the inference process is not executed. The scan settings can be set to execute a learning process. The second or third mode can be set when the learning process is executed, and the first mode can be set when the learning process is not executed. When performing the learning process, a label LBL indicating the correct answer of the learning result may be set in the reading settings, and the application 501 or the scanner driver 502 may assign that label LBL to the entire image output in the second mode.

[0061] Furthermore, each of the image processing units 302a and 302b of the scanner 100 may be provided with another first image processing unit (referred to as the "first image processing unit (2)"), and the processing in the first image processing unit 370 may be different from the processing in the first image processing unit (2). For example, the first image processing unit 370 and the first image processing unit (2) may use different setting values ​​for processing. In this case, the image data input to the second image processing unit 380 may be the output of the first image processing unit 370, and the image data input to the linear RGB image acquisition unit 380 may be the output of the first image processing unit (2). This makes it possible to output linear RGB image data that has been subjected to image processing by the first image processing unit suitable for learning, while outputting an output image according to the read settings.

[0062] (Settings screen) The setting screen used by the second image processing unit 380 will be described below. The scanner driver 502 has setting screens for making settings used by the color dropout processing unit 354, user gamma correction unit 355, JPEG conversion processing unit 356, etc. (i.e., settings used by the second image processing unit 380), and the user can make each setting from these setting screens. Here, the setting screen (user gamma setting screen) for correction data used by the user gamma correction unit 355 will be described as a representative with reference to FIG. 9.

[0063] FIG. 9 is a diagram showing an example of a user gamma setting screen, which is displayed on the display device 405 under the control of the scanner driver 502. On the user gamma setting screen, the user can specify in a data specification section 901 whether to set gamma correction for gray, red, green, or blue on the front and back of the document. Furthermore, in the input method selection section 902, the user can select "parameter" or "freehand" as the correction input method. If "parameter" is selected, the user can input a parameter (gamma correction value) using a slider 902a. Note that a gamma curve 903a in an area 903 changes in response to the parameter input. On the other hand, if "freehand" is selected, the user can set gamma correction by drawing the gamma curve 903a freehand in the area 903. When the OK button 904 is pressed, the gamma correction settings made on the user gamma setting screen are confirmed and saved in the HDD 403, for example.

[0064] As mentioned above, although not shown, setting screens are also provided for the settings used by the color dropout processing unit 354 and the JPEG conversion processing unit 356, and settings can be made from these setting screens. These setting screens are well known, so a description thereof will be omitted. These settings are transmitted to the scanner 100 when the scanner driver 502 instructs the scanner 100 to perform a scan. The scanner 100 performs processing in the second image processing unit 380 based on these settings.

[0065] The processing in the second image processing unit 380 is not limited to a combination of color dropout processing, gamma correction processing, and JPEG conversion processing, and may include other processing without including any of the processing.

[0066] (Learning model) FIG. 10 is a diagram schematically illustrating the learning process of the learning model 252 and the inference process using the trained model 255. FIG. 10(a) is a diagram schematically showing input / output data of the learning model 252 in the learning process executed by the learning unit 251, and the learning method. The input data X801 is supplied to the input layer of the learning model 252. In this embodiment, the input data X801 is image data of the training data T that corresponds to the linear RGB image data that is output from the first image processing unit 370, acquired by the linear RGB image acquisition unit 360, and transmitted to the PC 200. For example, the user can input the linear RGB image data as the input data X801 by using the operation unit 407 to specify a storage destination for the linear RGB image data as a data collection destination for the learning data from a specification screen (not shown) that the data collection and provision unit 253 or the like displays on the display device 405.

[0067] Then, output data Y803 is output as a result of processing the input data X801 with the machine learning model 252. The output data Y803 in this embodiment is label information of the document type discrimination result. During learning, correct answer data 802 of the training data T is provided as correct answer data for processing the input data X801. The correct answer data 802 of the training data T in this embodiment is label information of document type information such as slips, photographs, and text documents. For example, the data collection and provision unit 253 or the training data generation unit 250 or the like may display on the display device 405 a specification screen (not shown) from which the user specifies label information of document type information such as slips, photographs, and text documents (which may be a form type defined by the user or the like), and the specified label information can be set as the correct answer data 802 of the training data T.

[0068] As described above, by providing the output data Y803 and the correct answer data 802 of the training data T to the loss function 804, the deviation L805 of the processing result relative to the correct answer (the correct answer data of the training data T) is obtained. Then, for a large amount of training data, the connection weighting coefficients between the nodes of the neural network constituting the training model 252 are updated so that the deviation L805 approaches 0. The connection weighting coefficients between the nodes of the neural network that have been updated by training in this way become the trained model 255.

[0069] For example, backpropagation is an example of a technique for optimizing the connection weighting coefficients between nodes of a neural network so as to reduce the deviation L805. Specific examples of machine learning algorithms include nearest neighbor algorithms, naive Bayes algorithms, decision trees, and support vector machines. Deep learning and deep metric learning are also known, which use neural networks to generate features and connection weighting coefficients for learning. In this embodiment, any of these well-known algorithms can be used as appropriate, taking into account the intended use of machine learning. The implementation method of the learning model 252 is not particularly limited. The learning model 252 can be implemented using, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), or the like.

[0070] Note that the output destination of the linear RGB image data from the scanner driver 502 may be set to the data collection and provision unit 253, so that the linear RGB image data can be input to the data collection and provision unit 253 as input data X801. In this case, the data collection and provision unit 253, the learning data generation unit 250, and the learning unit 251 may operate and perform learning in conjunction with the output of the linear RGB image data from the scanner driver 502 to the data collection and provision unit 253. In this case, the document type set in the read settings or the document type that can be specified by the user for the image data output from the second image processing unit 380 and passed to the scanner driver 502 and application 501 may be used as the correct answer data for the training data T.

[0071] FIG. 10( b ) is a diagram showing input and output data of the trained model 255 in the inference process executed by the inference unit 254 . The input data X811 is supplied to the input layer of the trained model 255. For example, the user can use the operation unit 407 to specify a storage destination for the linear RGB image data as the data input destination from a specification screen (not shown) that the inference unit 254 displays on the display device 405, thereby making it possible to input the linear RGB image data as the input data X811. Note that the input data X811 has the same format as the input data X801 used during training, but there is no corresponding correct answer data (label information).

[0072] Output data Y813 is output as a result of processing input data X811 by the trained model 255. In the inference process, the output data Y813 is displayed as a processing result, for example, on the display device 405. Alternatively, it is output as data in a format used by a subsequent system. The trained model 255 may be implemented by a neural network with the same configuration as the training model 252, or may include only the part of the training model 252 necessary for the inference process. By making the trained model 255 have a smaller configuration than the training model 252, it is possible to reduce the amount of data in the trained model 255 and shorten the calculation time during the inference process.

[0073] Note that the output destination of the linear RGB image data from the scanner driver 502 may be set to the inference unit 254, so that the linear RGB image data can be input to the inference unit 254 as input data X811. In this case, the inference unit 254 and the like may operate to perform inference in conjunction with the output of the linear RGB image data from the scanner driver 502 to the inference unit 254.

[0074] When processing with the trained model 255, for example, if a document (e.g., a slip) is read using the scanner 100 and inference processing is performed using the linear RGB image data output by the first image processing unit of the image processing unit as input data, it is possible to determine that the read document is a "slip" and output the result.

[0075] Now, let us consider the case where the image data output by the second image processing unit is used as input data. For example, if a user performs "color dropout processing," image data with a different color tone from the original document is output. If this image data is used as input data for inference processing, it becomes difficult to determine that the scanned document is a "slip."

[0076] As described above, according to the first embodiment, in image data generated from a document read by an image reading device, by using the linear RGB image data output from the first image processing unit 370 as learning data, it is possible to learn using information that is closer to (more faithful to) the document than by using image data output from the second image processing unit 380 that includes color adjustment processing, thereby improving the accuracy of image recognition. In other words, by learning and inferring using image data that has not undergone color adjustment processing set by the user, it is possible to improve the accuracy of image recognition regardless of user settings.

[0077] In addition, in this embodiment, an example is shown in which the document type is determined by image recognition, but any process that performs image recognition may be used. For example, the blank page skip process may be determined by inference processing. In this case, the document type determination and the determination content will change, so the label of the training data can be changed.

[0078] 5 are provided on a cloud server, the scanner 100 and the PC 200 transmit the correct answer data of the input data X and the training data T to the cloud server. In this case, learning and inference may be performed in the cloud server in conjunction with the input of data. The results of inference and the like may be stored in the cloud server or sent back to the PC 200.

[0079] Second Embodiment In the second embodiment, a configuration will be described in which document type setting information set by a user when scanning a document is used as the correct answer data attached to the document to generate a trained model. Only the differences from the first embodiment will be described below.

[0080] In the second embodiment, when an application 501 installed on the PC 200 issues a command to perform image scanning processing on the scanner 100, the application 501 is configured to allow the type of document to be scanned to be set via the operation unit 407 of the PC 200. When the user sets the document type via the operation unit 407, the application 501 sets the set document type as label information in the supervised answer data 802 of the training data T. The application 501 also sets the linear RGB image data output from the first image processing unit 370 as input data X 801. The learning data can be updated by performing the machine learning processing described above using these data.

[0081] Furthermore, if the user does not set the type of document via the operation unit 407, there is no label information to set, and therefore the application 501 does not set anything in the supervised answer data 802 of the training data T, nor does it update the learning data. In this case, the application 501 sets the linear RGB image data output from the first image processing unit 370 to the input data X801, and performs a process (inference process) of determining the type of document using the trained model 255.

[0082] As described above, according to the second embodiment, it becomes possible for the user to add training data to a trained model, thereby enabling image recognition processing with even higher accuracy.

[0083] 5 are provided on a cloud server, the linear RGB image data output from the linear RGB image acquisition unit 360 can be sent directly from the scanner 100 (without going through the PC 200) to the cloud server. In this case, the address of the cloud server to which the linear RGB image data is to be sent, etc., is set in advance from the operation unit (not shown) of the scanner 100. The type of document is also set from the operation unit (not shown) of the scanner 100. However, the type of document set by the user in the application 501 may be acquired and used.

[0084] The scanner 100 as a reading device may also be a camera. That is, the camera may have the following functions: read an object using a reading device to generate image data for each color component constituting a color image; perform image processing on the generated image data according to the characteristics of the reading device; modify the processed image data (linear RGB image data) according to user-configurable settings; acquire the processed image data before the modification; and output at least one of the modified image data and the acquired image data. In this case, the result inferred by the inference process is preferably an image recognition result for distinguishing objects or detecting features of the objects.

[0085] As described above, according to each of the above embodiments, image data that is not affected by the user's image processing settings can be input as learning data for image recognition using a machine learning model, thereby improving the accuracy of machine learning that uses image data read from a document or the like as input data.

[0086] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and that the data may be configured in various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device. Furthermore, the present invention also includes any combination of the above embodiments.

[0087] Furthermore, in each of the above embodiments, an example has been described in which image processing unit 302 is provided inside scanner 100, and both image data processed by the change processing means (second image processing unit 380) and image data output from first image processing unit 370 and acquired by linear RGB image acquisition unit 360 are transferred from scanner 100 to PC 200, but this is not limiting. For example, the output of ADC 301a may be transferred directly to scanner driver 502 of PC 100, and the image processing described as being performed by the first image processing unit and second image processing unit may be performed by scanner driver 502 or application 501. In this case, linear RGB image acquisition unit 360 may be provided anywhere on PC 200.

[0088] Furthermore, even in the case of image processing described in the above embodiment as image processing by the second image processing unit 380, if the processing is parameterized so that the color tone is not substantially adjusted, it may be configured to be executed by the first image processing unit 370.

[0089] Furthermore, the image processing described as image processing by the first image processing unit 370 in the above embodiment may be configured to be executed after the image is acquired by the linear RGB image acquisition unit 360. In other words, the image data in the middle of the pipeline processing of the first image processing unit 370 may be acquired by the linear RGB image acquisition unit 360.

[0090] Other Embodiments The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the present invention may be applied to a system made up of multiple devices, or to an apparatus made up of a single device. The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In other words, all configurations that combine the above-described embodiments and their modifications are included in the present invention. [Explanation of symbols]

[0091] 1 seat loading platform 14,15 Image reading sensor (line image sensor) 100 scanners 200 Personal Computers (PC) 301a, 301b A / D conversion section (ADC) 302, 302a, 302b Image processing unit 303 RAM 304 Interface 310 ROM 350 Shading correction processing section 351 Device gamma correction section 352 3D gamma correction unit 353 Noise removal processing unit 354 Color dropout processing unit 355 User gamma correction section 360 Linear RGB Image Acquisition Unit 370 First Image Processing Unit 380 Second Image Processing Unit 401 ROM 402 RAM 403 Hard Disk Drive (HDD) 404 Communication I / F 405 Display device 406 CPU 407 Operation section 501 Application 502 Scanner Driver

Claims

1. a generating means for generating image data for each color component constituting a color image by reading an original or an object with a reading device; a change processing means for performing a process of changing the image data in accordance with a setting value that can be set by a user; an acquisition means for acquiring image data before being processed by the modification processing means; a reading device having an output means for outputting at least one of the image data processed by the change processing means and the image data acquired by the acquisition means; an information processing device having a learning means for performing machine learning using the image data acquired and output by the acquisition means as learning data; An image processing system comprising:

2. the reading device has an image processing means for executing image processing on the image data generated by the generating means in accordance with the characteristics of the reading device; the modification processing means performs processing on the image data processed by the image processing means, 2. The image processing system according to claim 1, wherein said acquisition means acquires image data that has been processed by said image processing means but has not yet been processed by said modification processing means.

3. The reading device The image processing means includes: a first processing means for executing processing on the image data generated by the generating means in accordance with the characteristics of the reading device; a second processing means for processing the image data generated by the generating means in accordance with the characteristics of the reading device different from those of the first processing means, the modification processing means performs processing on the first image data processed by the first processing means, 2. The image processing system according to claim 1, wherein the acquisition means acquires second image data processed by the second processing means.

4. the information processing device has an inference means for performing inference using the trained model trained by the training means, 4. The image processing system according to claim 1, wherein the inference means uses the image data acquired and output by the acquisition means as input data.

5. 5. The image processing system according to claim 4, wherein the learning data is training data including image data acquired by the acquisition means and labels that are the results of the inference for the image data.

6. The information processing device includes:

6. The image processing system according to claim 5, further comprising a designation unit that designates the label for the learning unit.

7. The information processing device includes: a setting means for allowing a user to set reading setting information including information on the type of document or subject to be read by the reading device; a transmitting means for transmitting information on the reading setting set by the setting means to the reading device, 6. The image processing system according to claim 5, wherein the label is data corresponding to information on the type of the document or subject included in the information on the reading settings.

8. 4. The image processing system according to claim 2, wherein the processing performed by said image processing means includes noise removal processing.

9. a generating means for generating image data for each color component constituting a color image by reading an original or an object with a reading device; an image processing unit that processes the image data generated by the generating unit in accordance with the characteristics of the reading device; a modification processing means for performing modification processing on the image data processed by the image processing means in accordance with a setting value that can be set by a user; an output means for selectively outputting either the image data processed by the modification processing means or the image data processed by the image processing means but before being processed by the modification processing means, or outputting both; A reading device comprising:

10. A control method for an image processing system having a reading device including a generating means for generating image data for each color component that constitutes a color image by reading an original or an object with a reading device, and a modifying means for performing a process to modify the image data in accordance with a setting value that can be set by a user, and an information processing device, an acquisition step executed by the reading device to acquire image data before being processed by the modification processing means; a learning step executed by the information processing device, in which machine learning is performed using the image data acquired and output by the acquisition step as learning data; 1. A method for controlling an image processing system, comprising:

Images