Image reading system

Abstract

This provides an image reading system that accurately determines the state of the reading unit used for image reading. [Solution] The image reading device 100 includes an image reading unit 70 for reading an image on a medium, and a control unit 200 that performs shading correction to correct unevenness in the optical system using the result of reading a color reference member by the image reading unit 70. The machine learning device 600 includes a storage medium 630 that stores a trained model 630a that has been trained using a dataset including the correction coefficients of the control unit 200, and a control unit 620 that uses the trained model 630a to determine if there is an abnormality in the image reading unit 70 from the correction coefficients calculated by the control unit 200.

Description

Technical Field

[0001] The present invention relates to an image reading system using an image reading device and a machine learning device.

Background Art

[0002] A failure prediction method using a machine learning device has been proposed to predict the failure of a device.

[0003] For example, in Patent Document 1, machine learning is performed using teacher data in which image data generated by an image reading device reading a document is associated with device abnormality information regarding an abnormality of the image reading device. When the image reading device reads a document, an image processing device that predicts the life time of a roller based on the generated image data and the learned model is described.

[0004] In such an image reading device, when paper dust of the document itself or toner peeled off from the document adheres to the image reading position of the reading glass as dust, or when staples are mixed during conveyance, the glass may be damaged, or sensors and light sources for reading images may deteriorate due to their lifespan, and image defects may occur due to various factors.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, even if Patent Document 1 can predict the life time of a roller, it is difficult for a serviceman or an operator to accurately respond to an abnormality regarding an image defect in which various events occur in a complex correlation.

[0007] Therefore, the object of the present invention is to provide an image reading device that can more accurately determine the state of the device (reading unit). [Means for solving the problem]

[0008] The image reading system according to the present invention is An image reading means for reading images from a medium, A shading correction unit performs shading correction to correct unevenness in the optical system using the result of reading the color reference member by the image reading means, A storage unit that stores a trained model obtained by machine learning using a dataset that includes the correction coefficient of the shading correction unit, The system is characterized by comprising an image reading anomaly determination unit that performs an anomaly determination of the image reading means based on the correction coefficient calculated by the shading correction unit using the aforementioned trained model. [Effects of the Invention]

[0009] According to the present invention, the status of the device (reading unit) can be accurately notified to service personnel and operators. Furthermore, by storing a learning model and updating that information, the status of the image reading device can be understood more accurately. [Brief explanation of the drawing]

[0010] [Figure 1] This is a cross-sectional view of the internal components of the image reading device, seen from the side. [Figure 2] Cross-sectional view of the image reading unit inside the image reading device. [Figure 3] A block diagram showing the electrical connection between the image reading device and the machine learning device. [Figure 4] Flowchart for the process of reading the transported media. [Figure 5] Flowchart of the training data generation process. [Figure 6] A flowchart for machine learning processing. [Figure 7] A diagram showing the input and output of a machine learning model. [Figure 8]Figure showing the shading correction coefficient of each pixel of the image sensor [Figure 9] Another figure showing the shading correction coefficient of each pixel of the image sensor

Embodiments for Carrying out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in the following order. Note that the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. (1) First Embodiment (1 - 1) Configuration of the image reading device and the machine learning device: (1 - 2) Generation of training data: (1 - 3) Machine learning: (1 - 4) Inference of abnormal states:

[0012] (1) First Embodiment (1 - 1) Configuration of the image reading device and the machine learning device: <Explanation of the internal configuration of the image reading device> FIG. 1 is a cross-sectional view of the internal configuration of an image reading device according to the first embodiment of the present invention as viewed from the side.

[0013] The image reading device 100 conveys one by one the one or more transport media S stacked on the mounting table 1 into the device through the path RT and discharges them to the discharge tray 2.

[0014] Here, the transport medium S is, for example, a sheet such as OA paper, checks, bills, business cards, cards, etc., and may be a thick sheet or a thin sheet. Examples of cards include insurance cards, driver's licenses, credit cards, etc. Also included are booklets such as passports.

[0015] When targeting a booklet, the booklet in a folded state is housed in a transparent holder and placed on the mounting table 1, so that the booklet is conveyed together with the holder.

[0016] In addition to the above, the image reading device 100 reads images of the transport medium S within the path RT and performs image processing on those images.

[0017] <Paper feed> A first transport unit 10 is provided as a transport mechanism for supplying the transport medium S along the route RT. The first transport unit 10 comprises a feed roller 11 and a separation roller 12 positioned opposite the feed roller 11, and sequentially transports the transport medium S on the mounting table 1 one by one in the transport direction D1.

[0018] The feed roller 11 receives driving force from the paper feed drive unit 3 via the transmission unit 5 and is rotated in the direction of the arrow in the figure (the positive direction that transports the transport medium S along the path RT).

[0019] <Drive Unit> The transmission unit 5, which connects the paper feed drive unit 3 and the feed roller 11, is normally configured to transmit driving force, but cuts off the driving force when the transport medium S is reversed or stopped. When the transmission of driving force to the feed roller 11 is cut off by the transmission unit 5, the feed roller 11 becomes capable of free rotation. Note that the transmission unit 5 does not need to be provided if the feed roller 11 is driven in only one direction.

[0020] <Separated structure> The separation roller 12, positioned opposite the feed roller 11, is a roller for separating the conveyed medium S one sheet at a time, and is in constant pressure contact with the feed roller 11. To ensure this pressure contact, the separation roller 12 is provided to be swingable and is configured to be biased toward the feed roller 11.

[0021] The separation roller 12 receives driving force from the paper feed drive unit 3 via the torque limiter 12a and is rotated in the direction of the solid arrow (opposite to the forward direction of the feed roller 11).

[0022] When the separation roller 12 is in contact with the feed roller 11, the torque limiter 12a restricts the transmission of driving force, causing it to rotate in the direction of rotation with the feed roller 11 (in the direction of the dashed arrow). As a result, when multiple conveying media S are conveyed to the contact area between the feed roller 11 and the separation roller 12, two or more conveying media S are prevented from being conveyed downstream, leaving one behind.

[0023] It should be noted that such a separation mechanism is not necessarily required; any feeding mechanism that sequentially feeds the transport medium S one by one along the path RT will suffice. Alternatively, instead of a configuration like the separation roller 12, a separation pad that applies frictional force to the transport medium S may be pressed against the feed roller 11 to achieve a similar separation effect.

[0024] <Conveying Structure> The second conveying unit 20 includes a drive roller 21 and a driven roller 22 that moves in accordance with the drive roller 21. It is located downstream of the first conveying unit 10 in the conveying direction and conveys the conveying medium S that has been conveyed from the first conveying unit 10 to the downstream side.

[0025] The drive roller 21 receives driving force from a conveying drive unit 4 such as a motor and is rotated in the direction of the arrow in the figure. The driven roller 22 presses against the drive roller 21 with constant pressure and rotates along with the drive roller 21. The driven roller 22 may be configured to be biased against the drive roller 21 by a biasing unit (not shown) such as a spring.

[0026] The third conveying unit 30 includes a drive roller 31 and a driven roller 32 that moves in accordance with the drive roller 31. It is located downstream of the second conveying unit 20 in the conveying direction and conveys the conveyed medium S that has been conveyed from the second conveying unit 20 to the discharge tray 2. In other words, this third conveying unit 30 functions as a discharge mechanism.

[0027] The drive roller 31 receives driving force from a conveying drive unit 4 such as a motor and is rotated in the direction of the arrow in the figure. The driven roller 32 presses against the drive roller 31 with constant pressure and rotates along with the drive roller 31. The driven roller 32 may be configured to be biased against the drive roller 31 by a biasing unit (not shown) such as a spring.

[0028] <Image reading position detection structure> The image reading position detection sensor 50 detects the position of the transport medium S being transported by the first transport unit 10, specifically whether the end of the transport medium S has reached or passed the detection position of the image reading position detection sensor 50.

[0029] Various types of image reading position detection sensors 50 can be used, but in this embodiment, it is an optical sensor comprising a light-emitting unit 51 and a light-receiving unit 52, and detects the transport medium S based on the principle that the light-receiving intensity (amount of light received) changes when the transport medium S arrives or passes through.

[0030] Furthermore, the image reading position detection sensor 50 is not limited to the optical sensor described above. For example, a sensor capable of detecting the end of the transport medium S (such as an image sensor) may be used, or a lever-type sensor protruding from the path RT may also be used.

[0031] In this embodiment, the image reading position detection sensor 50 is placed on the upstream side in the transport direction of the second transport unit 20, but the image reading position detection sensor 60 may be placed on the downstream side in the transport direction. In addition, both image reading position detection sensors 50 and 60 may be placed.

[0032] <Image reading structure> Figure 2 is a cross-sectional view of the image reading unit 70 of an image reading device 100 according to one embodiment of the present invention.

[0033] The image reading unit 70 is positioned opposite the second transport unit 20 and the third transport unit 30. In the following description, these will be referred to as image reading units 70a and 70b, respectively. Image reading units 70a and 70b are sensor units having the same structure, positioned opposite each other with the path RT in between.

[0034] The image reading unit 70a optically scans and converts the image into an electrical signal to read it as image data. It contains an optical unit 74a, an image reading light source 75a, and an image sensor 76a. A white reference plate 72a is also provided for shading correction of the opposing reading unit.

[0035] Furthermore, the image reading unit 70a is sealed to prevent contamination of the transport medium S with paper dust or other materials. Therefore, the transport medium S passes through the path RT, which is sandwiched between the glass surfaces 77a and 77b.

[0036] Since the image reading unit 70b has the same structure as the image reading unit 70a, its explanation will be omitted.

[0037] Although embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications are possible within the scope of the present invention.

[0038] <Block diagram of image reading device and machine learning device> Figure 3 is a block diagram showing the electrical connection between an image reading device and a machine learning device according to an embodiment of the present invention.

[0039] The image reading device 100 includes a control unit 200, a ROM 302, a RAM 303, a UI (User Interface) unit 290, an image reading unit 270, and a media transport unit 210.

[0040] The control unit 200 consists of a CPU, microcontroller, etc., and is responsible for the control, calculations, and information transfer of the entire device. The control performed by the control unit 200 includes the control of the media transport unit 210 (paper feed drive unit 3, transport drive unit 4, transmission unit 5), the control of the image reading unit 270 (image reading position detection sensors 50 and 60, image sensor 76, image reading light source 75, A / D conversion unit 301), and the control of the UI unit 290 (display unit 93, input unit 94). The control unit 200 also includes a communication unit 202 and an image processing unit 207.

[0041] The communications unit 202 communicates with the machine learning device 600 (described later) and other external devices (not shown). Communication standards include wired connections such as USB, LAN, and SCSI, and wireless connections such as wireless LAN and Bluetooth®.

[0042] The image processing unit 207 processes the digital image data output from the AD conversion unit 301 and outputs the processed image data to the RAM 303.

[0043] ROM302 is a non-volatile storage device for storing data such as program data for the control unit 200, correction data for the pixel processing unit 207, and the learned model 630a described later.

[0044] RAM303 is a high-speed access storage device for temporarily storing data such as some of the program data from the control unit 200 and image data acquired from the image sensor 76.

[0045] The UI unit 290 includes an input unit 93 such as a touch panel, various keys and switches, and a display unit 94 such as a display or LEDs.

[0046] The control unit 200 can acquire user operations via the input unit 93. The control unit 200 can also display various information on the display unit 94 to inform the user.

[0047] The image reading unit 270 includes an image sensor 76, an image reading light source 75, an optical unit 74, a white reference plate 72, an A / D conversion unit 301, and image reading position detection sensors 50 and 60.

[0048] The image sensor 76 is a line sensor extending in one direction, with multiple photoelectric conversion elements arranged in that direction. Each photoelectric conversion element outputs a signal corresponding to the intensity of the received light. In this embodiment, the direction in which the line sensor extends (main scanning direction) is positioned perpendicular to the transport direction of the transport medium (sub-scanning direction).

[0049] The image reading light source 75 is equipped with an LED, fluorescent lamp, or the like to irradiate the transport medium with light. The light reflected from the transport medium or white reference plate 72 located at the irradiation position is received by the image sensor 76 of the image reading unit 70, and the image sensor 76 generates a signal corresponding to the amount of light received by each photoelectric conversion element.

[0050] The optical unit 74 includes an optical element that forms an optical path that guides the light from the transport medium, generated when the light from the image reading light source 75 irradiates the transport medium, to the line sensor. The optical path may be provided by various structures, and the optical element can also be made up of various materials.

[0051] The A / D (analog-to-digital) conversion unit 301 is a converter that amplifies and offsets minute analog signals before converting them into digital image data. In this embodiment, it is configured separately from the image reading unit 70.

[0052] The media transport unit 210 includes a paper feed drive unit 3, a transport drive unit 4, and a transmission unit 5. The paper feed drive unit 3 is, for example, a motor, and the transmission unit 5 is, for example, an electromagnetic clutch.

[0053] The machine learning device 600 is a computer that generates and stores training data based on information acquired from the image reading device 100, performs machine learning based on the stored training data, and outputs the resulting trained model 630a to the image reading device 100.

[0054] The machine learning device 600 may, for example, generate training data based on information acquired from multiple image reading devices of the same type, and distribute the trained model 630a generated as a result of machine learning using the training data to multiple image reading devices of the same type. Alternatively, it may be configured as an integrated device or system with any of the image reading devices.

[0055] The machine learning device 600 includes a control unit 620 equipped with a CPU, RAM, ROM, etc., a storage medium 630, and a communication unit 650.

[0056] The control unit 620 can perform machine learning-related functions by executing a machine learning program (not shown) recorded on the storage medium 630.

[0057] In this embodiment, the control unit 200 is configured to transmit various information, including data generated as a result of reading by the image reading device 100, to the machine learning device 600 via the communication unit 202.

[0058] Furthermore, the control unit 200 stores the trained model 630a generated as a result of machine learning in the machine learning device 600 in the ROM 302 when it is acquired from the machine learning device 600 via the communication unit 202. Thereafter, when reading a document, the trained model 630a is used to infer the state of the image reading unit 270 in the image reading device 100. Alternatively, data may be sent to the machine learning device 600 when inferring the state of the image reading unit 270, and the inference results may be acquired from the machine learning device 600.

[0059] In this embodiment, a machine learning model for inferring the state of the image reading unit 270 within the image reading device 100 is generated in the machine learning device 600. For the generation of the machine learning model, machine learning data is transmitted from the image reading device 100 to the machine learning device 600.

[0060] The data transmitted includes the light intensity adjustment value of the image reading light source used for this image reading, the amplification and offset adjustment values ​​of the A / D conversion unit 301, the correction data for shading correction, the comparison data of the light intensity adjustment value, amplification, offset adjustment value, and shading correction data between this time and the previous time, the same data comparing this time with the time of shipment, the image data obtained from this image reading (for machine learning), the usage history including transport errors and maintenance performed, the image reading parameters, the device ID, and the reading date and time, and is called the machine learning dataset.

[0061] Furthermore, since the correction data for shading correction would be enormous if it included the entire line data, it is acceptable to use features that highlight the correction data, such as the maximum, minimum, and average values ​​of the correction data.

[0062] Furthermore, the machine learning device 600 generates training data based on a machine learning dataset and stores it in the storage medium 630 (training data 630b).

[0063] (1-2) Generation of training data: In this embodiment, the state of the image reading unit 270 within the image reading device 100 is inferred using a machine learning model. In this embodiment, the system is configured to learn the characteristics of the machine learning dataset for cases where the reading unit is normal, cleaning of the reading unit is necessary, replacement of the reading unit is necessary, and recalibration is necessary.

[0064] Figure 4 is a flowchart showing the document reading process performed by the control unit 200 of the image reading device 100 when generating training data.

[0065] In the image reading process shown in Figure 4, the user sets the transport medium S on the platform 1, specifies the reading conditions (reading resolution, color / monochrome, automatic / fixed size (including orientation specification), type of medium, single-sided / double-sided reading, etc.) via the UI unit 290 or an external device (not shown) via the communication unit 202, and then inputs a read start command to begin the image reading process.

[0066] When the image reading process shown in Figure 4 is started, the control unit 200 generates shading correction data to correct for unevenness in brightness and aberrations (step S010).

[0067] In shading correction, the control unit 200 generates image data from the image sensor 76 with the transport medium S stopped and the image reading light source 75 turned off. From the generated image data, the A / D conversion unit 301 adjusts the offset adjustment value and generates shading correction data (black correction data) (black reference adjustment).

[0068] After black reference adjustment, the control unit 200 stops the transport medium S, turns on the image reading light source 75, reads the white reference plate 72 with the image sensor 76, and generates image data. From the generated image data, the control unit 200 adjusts the light intensity value of the image reading light source 75, the gain adjustment value of the A / D conversion unit 301, and generates shading correction data (white correction data) (white reference adjustment).

[0069] Next, the control unit 200 determines whether an abnormality has occurred in the image reading unit 70 (step S105).

[0070] Conditions that result in an abnormality include when the light intensity adjustment value of the image reading light source or the amplification / offset adjustment value of AFE301 exceeds a predetermined value, when the output of the image sensor 76 is abnormal, or when there is a large difference between the current and previous setting values. As mentioned above, inference is made using the trained model 630a acquired from the machine learning device 600, and the image reading unit 70 is in a state where cleaning, replacement, and recalibration are necessary.

[0071] In step S105, if an abnormal condition is detected, the control unit 200 notifies the user of the abnormal condition. The information provided may include that there is dirt on the image reading unit 70, that there are scratches on the glass 77, that the light source 74 is degraded, or that the output of the image sensor 76 is uneven.

[0072] Furthermore, the information provided may include instructions to resolve the abnormal condition. For example, this could include instructions to clean the device, replace the reading unit 70, or recalibrate it.

[0073] The user inputs whether the displayed abnormality notification is correct using the UI unit 290 or an external device (not shown). If it is incorrect, the user selects the corrected abnormality notification. After selecting the abnormality notification, the image reading process ends (step S140).

[0074] For example, the main control unit 200 displays an abnormal status on the UI unit 290's display. Furthermore, if a transport error occurs or maintenance is performed, the processor 20 updates the corresponding data in the usage history.

[0075] If it is determined in step S105 that there is no abnormal condition, the control unit 200 starts driving the first transport unit 10, the second transport unit 20, and the third transport unit 30. The transport media S loaded on the mounting table 1 are transported one by one, starting from the transport media S located at the bottom.

[0076] When the leading edge of the transport medium S reaches the image reading position detection sensor 50 or 60, the image reading position detection sensor 50 or 60 outputs a detection signal, and the control unit 200 measures the timing for reading and starts reading the image with the image sensor 76 (step S110).

[0077] After executing step S110 or step S140, the control unit 200 transmits the machine learning dataset to the machine learning device 600 via the communication unit 202 and terminates the image reading process (step S150). It is preferable that if the user checks the image read in step S110 and determines that some kind of abnormality has occurred, the abnormality details will be provided as feedback. In that case, the machine learning dataset will be transmitted in step S150 along with the abnormality details specified by the user.

[0078] Figure 5 shows the training data generation process performed by the control unit 620 of the machine learning device 600, and is the process on the machine learning device 600 side that corresponds to the image reading process in Figure 4.

[0079] The training data generation process is executed when a machine learning dataset is received from the image reading device 100. When the training data generation process starts, the control unit 620 determines whether or not the machine learning dataset is data from when an anomaly was detected (step S200).

[0080] In step S200, if the received machine learning dataset is not the dataset used when an anomaly was detected, the control unit 620 treats the received machine learning dataset as the dataset used when a normal condition is met, terminates the training data generation process, and stores the machine learning dataset in the storage medium 630 (step S210).

[0081] In step S200, if a machine learning dataset for an anomaly detection is received, the control unit 620 configures training data by associating the machine learning dataset with the anomaly state as the dataset for an anomaly (step S215).

[0082] (1-3) Machine Learning: Figure 6 is a flowchart showing the machine learning process.

[0083] Once a predetermined amount of training data has been accumulated, the control unit 620 executes machine learning processing using the training data. The machine learning processing may be executed at any time after a predetermined amount of training data 630b has been accumulated.

[0084] When machine learning processing is started, the control unit 620 acquires the training model 630d (step S300).

[0085] Here, the model is information that shows the equation that derives the correspondence between the data to be estimated and the estimated data.

[0086] Training indicates that something is being studied. In other words, the training model 630d takes a machine learning dataset as input and outputs the predicted abnormal state of the image reading unit 70 of the image reading device 100. However, the correspondence between the machine learning dataset and the predicted abnormal state of the image reading unit 70 is not accurate initially.

[0087] In other words, in the training model 630d, the number of layers and nodes that make up a node are determined, but the parameters that define the relationship between input and output (weights, biases, etc.) are not optimized. These parameters are optimized (i.e., trained) during the machine learning process.

[0088] The training model 630d may be predetermined, or it may be obtained by an operator operating the machine learning device 600 by operating a UI unit (not shown) connected to the machine learning device 600.

[0089] In any case, the control unit 620 acquires the parameters of a neural network that outputs the abnormal state of the image reading unit 70 of the image reading device 100, based on the machine learning dataset acquired by the image reading device 100, as the training model 630d, as shown in the example in Figure 7 described later.

[0090] Next, the control unit 620 acquires training data (step S305). In this embodiment, the training data is configured as described in the image reading process in Figure 4 and the training data generation process in Figure 5, and is stored in the storage medium 630. The control unit 620 acquires the training data 630b by referring to the storage medium 630.

[0091] Next, the control unit 620 acquires test data 630c (step S310). In this embodiment, a portion of the training data 630b is extracted to become the test data. The test data is not used for training.

[0092] Next, the control unit 620 determines the initial values ​​(step S315). That is, the control unit 620 assigns initial values ​​to the variable parameters of the training model 630d acquired in step S300.

[0093] Initial values ​​can be determined using various methods. For example, random values ​​or 0 can be used as initial values, and the initial values ​​may be determined using different approaches for weights and biases. Of course, the initial values ​​may be adjusted so that the parameters are optimized during the learning process.

[0094] Next, the control unit 620 performs learning (step S320). Specifically, the control unit 620 inputs the machine learning dataset (light intensity adjustment value of the reading light source 75, amplification and offset adjustment value of the A / D conversion unit 301, correction data for shading correction, comparison value between the current and previous measurements, usage history, etc.) from the training model 630d acquired in step S300 into the control unit 630b, and calculates the output value of the output layer Lo.

[0095] Furthermore, the control unit 620 identifies the error using a loss function that shows the error between the output parameters and the parameters indicated by the training data 630b. The control unit 620 then repeats the process of updating the parameters a predetermined number of times based on the derivative of the loss function with respect to the parameters.

[0096] Of course, various functions can be used as the loss function, such as the cross-entropy error. The process of calculating the loss function described above is performed on all or some of the images included in the training data 630b, and the loss function for one training run is expressed by the average or sum of these results. Once the loss function for one training run is obtained, the control unit 620 updates the parameters using a predetermined optimization algorithm, such as stochastic gradient descent.

[0097] Once the parameters have been updated a predetermined number of times as described above, the control unit 120 determines whether or not the generalization of the training model 630d has been completed (step S325).

[0098] In other words, the control unit 620 inputs the image data shown by the test data acquired in step S310 to the training model 630d and obtains an output indicating an abnormal state.

[0099] The control unit 620 then obtains the degree to which the output matches the test data. In this embodiment, the control unit 620 determines that generalization is complete when the degree of matching is equal to or greater than a threshold.

[0100] In addition to evaluating generalization performance, the validity of hyperparameters may also be verified. That is, in a configuration in which hyperparameters other than weights and biases, such as the number of nodes, are tuned, the control unit 620 may verify the validity of the hyperparameters based on verification data.

[0101] Validation data may be extracted from training data using the same process as in step S310. Of course, like test data, validation data is not used for training.

[0102] If it is determined in step S325 that the generalization of the training model 630d is not complete, the control unit 620 repeats step S320, that is, it further updates the weights and biases.

[0103] On the other hand, if it is determined in step S325 that the generalization of the training model 630d is complete, the control unit 620 records it as the trained model 630a (step S330). In other words, the control unit 620 records the training model 630d as the learned model 630a in the storage medium 630.

[0104] Figure 7 is a schematic diagram showing an example of a model used in this embodiment.

[0105] In this embodiment, we will explain using a model as an example in which the input data consists of the light intensity adjustment value of the reading light source 75, the amplification and offset adjustment value of the A / D conversion unit 301, the correction data for shading correction, the comparison value of the current and previous shading corrections, the comparison value of the current and factory shading corrections, and the usage history, and the output data consists of the estimated result of the abnormal state of the image reading unit 70 in the image reading device 100.

[0106] In this embodiment, the model includes the following as nodes in the input layer corresponding to the input data: the light intensity adjustment value of the reading light source 75, the amplification and offset adjustment value of the A / D conversion unit 301, the correction data for shading correction, the comparison value of the current and previous shading corrections, the comparison value of the current and factory shading corrections, and the usage history.

[0107] In this embodiment, each of the above data and each of the data constituting the usage history are used as input data to each node of the intermediate layer 1, and intermediate output data is output from the intermediate layer 3.

[0108] Each node in the output layer corresponds to an abnormal state of the inferred image reading unit 70 (normal, requires cleaning of the reading unit, requires replacement of the reading unit (image sensor 76 or glass 77), or requires recalibration). The sum of the output values ​​of each node N1 to N4 in the output layer is normalized to 1.

[0109] With the above configuration, a model (trained model 630a) for inferring abnormal states of the image reading unit 70 of the image reading device 100 can be produced.

[0110] The control unit 620 transmits the learned model 630a to the image reading device 100 via the communication unit 650. When the control unit 200 of the image reading device 100 receives the learned model 630a via the communication unit 202, it stores it in the ROM 302.

[0111] (1-4) Inference of abnormal conditions: The control unit 200 of the image reading device 100 stores the trained model 630a in the ROM 302, and when it subsequently reads a sheet transport material S, it inputs a machine learning dataset into the trained model 630a. The control unit 200 acquires the output values ​​of each node in the output layer of the trained model 630a and can infer the abnormal state of the image reading unit 70 based on the output values ​​of each node.

[0112] The inference of abnormal conditions is performed in step 105 of the image reading process in Figure 4, during the abnormality detection step.

[0113] In step S105, the control unit 200 inputs the training dataset to the trained model 630a and obtains an output.

[0114] For example, if the output (N1, N2, N3, N4) from the trained model 630a is (0.07, 0.9, 0.02, 0.01), the control unit 200 selects N2, which has the largest output value. N2 is associated with cleaning the reading unit. Therefore, the control unit 200 infers that this is an abnormal condition requiring cleaning of the reading unit.

[0115] Here, we will explain the abnormal condition. Figures 8 and 9 show the shading correction data. The image sensor 76 has multiple photoelectric conversion elements arranged in that one direction, and there is variation in the output of the multiple photoelectric conversion elements. Therefore, a shading correction coefficient is calculated to make the output uniform (see the explanation of S010 above).

[0116] Figure 8(a) shows an example where the shading correction data calculated for each pixel position is all correct.

[0117] In this embodiment, since the light intensity is adjusted so that each photoelectric conversion element does not become saturated, the shading correction coefficient to be multiplied will be a value of 1.0 or greater. Furthermore, since the shading correction coefficient to be multiplied will not exceed the sum of the worst lower limits of variations in light sources, sensors, etc., it is preferable to set an upper limit for the correction coefficient.

[0118] In this embodiment, if the shading correction coefficient is less than 0.8 or greater than 1.8, it is determined to be an abnormality in the shading correction coefficient.

[0119] Figure 8(b) shows an example of a result where the shading correction coefficient is abnormal. In this example, one pixel of the photoelectric conversion element exceeds the upper limit of the shading correction coefficient (1.8), while the other pixel falls below the lower limit of the shading correction coefficient (0.8).

[0120] Figure 8(b) shows a situation where an anomaly has occurred in a single pixel, indicating that the anomaly is occurring in a very minute area. This corresponds to cases where tiny dust particles are attached or where a single sensor pixel is destroyed (failed). However, this does not necessarily mean that these conditions are definitively met.

[0121] If the shading correction coefficient exceeds the upper limit (1.8), it indicates insufficient sensor output. This can occur if the attached debris is black, if a damaged sensor output is fixed as a black output, or if the glass is scratched and does not transmit light, resulting in insufficient sensor output.

[0122] Furthermore, if the shading correction coefficient is below the lower limit (0.8), it indicates that the sensor output is excessive. This can occur if the attached debris is white, or if the damaged sensor output is fixed as a white output.

[0123] Figure 9(a) also shows an example of a result where the shading correction coefficient is abnormal. In this example, multiple consecutive photoelectric conversion elements exceed the upper limit of the shading correction coefficient (1.8).

[0124] Figure 9(a) shows abnormalities occurring across multiple pixels over a wide area. This is often due to large black dust particles or scratches on the glass 77. However, this is not definitively confirmed.

[0125] Figure 9(b) also shows an example of a result where the shading correction coefficient is abnormal. In this example, multiple consecutive photoelectric conversion elements exceed the upper limit (1.8) of the shading correction coefficient and fall below the lower limit (0.8).

[0126] Figure 9(b) shows that the range of multiple pixels exceeding the upper limit or falling below the lower limit of the shading correction coefficient has the same pixel width. The image sensor 76, which consists of multiple photoelectric conversion elements arranged in one direction, is formed into blocks of a certain number, and abnormalities are occurring continuously within the width of these blocks. This often indicates that the image sensor 76 is destroyed (or malfunctioning) at the block level. However, this is not definitively the case.

[0127] Furthermore, although this embodiment uses a single threshold for abnormality detection, since the correction coefficient in the event of an abnormality tends to vary depending on the cause of the failure, two thresholds may be provided for determining the cause. This configuration makes it possible to distinguish between a case where the attached dust is not completely black and slightly exceeds the upper limit of the correction coefficient, and a case where the faulty sensor output is fixed as a black output and the shading correction coefficient far exceeds the upper limit.

[0128] Figures 8 and 9 illustrate the effect of abnormal conditions on the shading correction coefficient. Before and after calculating this shading correction coefficient, the light intensity adjustment value of the image reading light source 75 and the amplification / offset adjustment value of the A / D conversion unit 301 are adjusted. According to this embodiment, the status of the equipment is determined based on these adjustment values, as well as the comparison value of the shading correction coefficient with the previous value, the comparison value with the value at the time of shipment, and the usage history including maintenance.

[0129] As described above, according to this embodiment, by storing a learning model and updating that information, the status of the image reading device can be grasped more accurately, and the status of the device (reading unit) can be accurately notified to service personnel and operators.

[0130] In the above embodiment, a method for determining the state of the device was described using a learning model trained by supervised learning using a dataset containing data corresponding to abnormal states, but the invention is not limited to this. That is, the state may also be determined using a learning model trained by semi-supervised learning using training data labeled as normal states. Alternatively, the state may be determined using a learning model in which each dataset has been clustered by unsupervised learning. [Explanation of Symbols]

[0131] 70 Image reading unit 74 Optics Department 75 Image reading light source 76 Image Sensors 77 Glass 93 Input section 94 Display section 100 Image reading device 200 Control Unit 600 Machine Learning Devices 620 Control Unit 630 Storage medium 630a Pre-trained Model 630b Training data 630c Test Data 630d training model

Claims

1. An image reading means for reading images from a medium, A shading correction unit performs shading correction to correct unevenness in the optical system using the result of reading the color reference member by the image reading means, A storage unit that stores a trained model obtained by machine learning using a dataset that includes the correction coefficient of the shading correction unit, An image reading system comprising an image reading anomaly determination unit that uses the trained model to determine anomalies in the image reading means based on correction coefficients calculated by the shading correction unit.

2. The image reading system according to claim 1, characterized in that the dataset includes light intensity adjustment values ​​of the light source of the image reading means.

3. The image reading system according to claim 1 or 2, characterized in that the trained model outputs state information including that the image reading means needs to be replaced as a result of the abnormality determination.

4. The image reading system according to claim 3, characterized in that the dataset includes usage history information, including that the image reading means has been replaced.

Images