Image processing apparatus, image processing method, and control program

Abstract

To provide an image processing apparatus, an image processing method, and a control program capable of determining whether absence of a corner area of a medium has occurred in an image, simply with high accuracy.SOLUTION: An image processing apparatus includes: an acquisition unit which acquires an input image obtained by imaging a medium; a detection unit which detects a plurality of corner areas each estimated to include corners of the medium in the input image; a determination unit which determines whether absence of image has occurred in each of the corner areas by comparing the corner areas detected by the detection unit; and an output unit which outputs information on a result determined by the determination unit.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a control program. [Background technology]

[0002] Conventionally, image reading devices such as scanners that capture a medium and generate an image have been used. In such image reading devices, there are cases where the medium is placed outside the imaging range, or where the corner of the medium to be imaged is folded by the user. In such cases, an image generated by the image reading device is missing the corner of the medium, resulting in an image loss.

[0003] An image forming device that forms an original image on a recording sheet has been disclosed (see Patent Document 1). This image forming device detects each edge of the original image, and detects the presence or absence of an inclined side at the corner of the effective area where image formation is performed in the original image. When the image forming device detects an inclined side, it determines whether the inclined side is present within the effective area, and if it is determined that the inclined side is present within the effective area, it prohibits the saving of the original image or the image formation of the original image.

[0004] An image reading device is disclosed that detects edge pixels from an input image, detects straight lines from the edge pixels, and determines whether edge pixels that are not within a predetermined distance from the straight lines represent the edge of a document (see Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6561952 [Patent Document 2] JP 2014-147046 A Summary of the Invention [Problem to be solved by the invention]

[0006] There is a demand for image processing devices that can easily and accurately determine whether or not image loss has occurred in a corner area of ​​a medium in an image.

[0007] An object of the present invention is to provide an image processing device, an image processing method, and a control program that can easily and accurately determine whether or not an image is missing in a corner area of ​​a medium in an image. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present invention has an acquisition unit that acquires an input image of a medium, a detection unit that detects multiple corner areas within the input image each of which is estimated to contain a corner of the medium, a judgment unit that determines whether or not an image is missing in each of the multiple corner areas by comparing the multiple corner areas detected by the detection unit, and an output unit that outputs information regarding the judgment result by the judgment unit.

[0009] An image processing device according to one aspect of the present invention has an acquisition unit that acquires multiple input images of a medium, a detection unit that detects corner areas within each of the multiple input images that are estimated to contain corners of the medium, a judgment unit that determines whether or not an image is missing in the corner area by comparing the corner areas detected by the detection unit, and an output unit that outputs information regarding the judgment result by the judgment unit.

[0010] In an image processing method according to one aspect of the present invention, an image processing device acquires an input image of a medium, detects multiple corner areas within the input image that are estimated to each contain a corner of the medium, compares the detected multiple corner areas to determine whether or not an image is missing in each of the multiple corner areas, and outputs information related to the result of the determination.

[0011] In an image processing method according to one aspect of the present invention, an image processing device acquires multiple input images of a medium, detects corner areas within each of the multiple input images that are estimated to contain the corners of the medium, and compares the detected corner areas to determine whether or not an image is missing in the corner area, and outputs information related to the result of the determination.

[0012] A control program according to one aspect of the present invention is a control program for an image processing device that causes the image processing device to acquire an input image of a medium, detect multiple corner areas within the input image that are estimated to each contain a corner of the medium, compare the detected multiple corner areas to determine whether or not an image is missing in each of the multiple corner areas, and output information related to the result of the determination.

[0013] A control program according to one aspect of the present invention is a control program for an image processing device that acquires multiple input images of a medium, detects corner areas within each of the multiple input images that are estimated to contain corners of the medium, compares the detected corner areas to determine whether or not an image is missing in the corner area, and outputs information related to the result of the determination. Effect of the Invention

[0014] According to the present invention, the image processing device, image processing method, and control program are capable of easily and highly accurately determining whether or not image loss has occurred in a corner area of ​​a medium in an image. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an image processing system 1. [Diagram 2] 2 is a diagram for explaining a transport path inside the image reading device 100. FIG. [Diagram 3] 3 is a schematic diagram for explaining each sensor for detecting a medium. FIG. [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of an image reading device 100 and the like. [Diagram 5] 1A is a diagram showing a schematic configuration of a first processing circuit 140 and the like, and FIG. 1B is a diagram showing a schematic configuration of a second processing circuit 220 and the like. [Figure 6] 10 is a flowchart showing an example of an operation of an image reading process. [Figure 7] 13 is a flowchart showing an example of the operation of a determination process. [Figure 8] 2 is a schematic diagram showing an example of an input image N. FIG. [Figure 9] 13 is a flowchart showing an example of the operation of an image loss determination process. [Figure 10] 13 is a flowchart showing an example of the operation of a content determination process. [Figure 11] Graphs (A), (B), and (C) show an example of the gradation values ​​of pixels on the edge of a medium, and graph (D) is a histogram of the graph shown in (A). [Figure 12] 13 is a flowchart illustrating an example of an operation of a content correction process. [Figure 13] 13 is a flowchart showing an example of the operation of another image loss determination process. [Figure 14] 13 is a flowchart showing an example of the operation of still another image loss determination process. [Figure 15] FIG. 11 is a schematic diagram for explaining a corner distance. [Figure 16] 13 is a flowchart showing an example of the operation of still another image loss determination process. [Figure 17] 13 is a flowchart showing an example of another operation of a content determination process. [Figure 18] 13 is a flowchart showing an example of the operation of still another content determination process. [Figure 19] 13 is a flowchart showing an example of the operation of another content correction process. [Figure 20] 13 is a flowchart showing an example of the operation of still another content correction process. [Figure 21]1A is a diagram showing a schematic configuration of another first processing circuit 340, and FIG. 1B is a diagram showing a schematic configuration of another second processing circuit 420. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an image processing device, an image processing method, and a control program according to one aspect of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to the embodiments, but extends to the inventions described in the claims and their equivalents.

[0017] FIG. 1 is a diagram showing an example of a configuration of an image processing system 1 according to an embodiment.

[0018] The image processing system 1 includes an image reading device 100 and an information processing device 200. The image reading device 100 and the information processing device 200 are each an example of an image processing device. The image reading device 100 and the information processing device 200 are connected to each other for communication. The image reading device 100 is an image scanner or the like that conveys a medium that is an original and captures an image. The medium is paper, thin paper, thick paper, a card, a booklet, a passport, or the like. The image reading device 100 may be a facsimile, a copier, a printer multifunction machine (MFP, Multifunction Peripheral), or the like. The image reading device 100 may also be a flatbed type image scanner, a facsimile, a copier, an MFP, or the like that captures an image without conveying a medium. The image reading device 100 may also be a mobile phone, a smartphone, a tablet computer, a notebook personal computer, or the like that captures an image of a person, an object, a landscape, or the like. The information processing device 200 is a personal computer, or the like. The information processing device 200 may be a mobile phone, a smartphone, a tablet computer, a notebook personal computer, etc. The information processing device 200 may be a server. In this case, a plurality of image reading devices 100 may be communicatively connected to one information processing device 200.

[0019] 1, arrow A1 indicates the medium transport direction, and arrow A2 indicates the width direction perpendicular to the medium transport direction. In the following, upstream refers to the upstream side in the medium transport direction A1, and downstream refers to the downstream side in the medium transport direction A1.

[0020] The image reading device 100 includes a lower housing 101, an upper housing 102, a placement table 103, a discharge table 104, a first operation device 105, a first display device 106, and the like.

[0021] The upper housing 102 is disposed at a position that covers the top surface of the image reading device 100, and engages with the lower housing 101 by a hinge so as to be openable and closable when loading a medium or cleaning the inside of the image reading device 100, for example.

[0022] The placement table 103 engages with the lower housing 101. The placement table 103 has a placement surface 103a on which the medium is placed, and places the medium to be fed and transported. On the placement surface 103a, a side guide 103b is provided movably in a width direction A2 perpendicular to the medium transport direction. The side guide 103b is positioned according to the width of the medium placed on the placement table 103, and regulates the width direction of the medium. In the example shown in FIG. 1, two side guides 103b are movably arranged with a gap between them so that the medium is placed in the center in the width direction A2. Only one side guide 103b may be movably arranged so that the medium is placed at one end in the width direction A2.

[0023] The ejection platform 104 engages with the upper housing 102 and places the ejected media thereon. The ejection platform 104 may be provided to engage with the lower housing 101.

[0024] The first operation device 105 has an input device such as a button and an interface circuit that acquires a signal from the input device, accepts an input operation by a user, and outputs an operation signal according to the input operation by the user.

[0025] The first display device 106 is an example of an output unit. The first display device 106 has a display including a liquid crystal, an organic EL (Electro-Luminescence), or the like, and an interface circuit that outputs image data to the display, and displays the image data on the display. The first display device 106 and the first operating device 105 may be provided integrally using a touch panel type input device.

[0026] FIG. 2 is a diagram for explaining a transport path inside the image reading device 100. As shown in FIG.

[0027] The transport path inside the image reading device 100 includes a first media sensor 111, a feed roller 112, a separation roller 113, a second media sensor 114, a first transport roller 115, a second transport roller 116, a third media sensor 117, an imaging device 118, a third transport roller 119, and a fourth transport roller 120.

[0028] The number of each of the feed roller 112, separation roller 113, first conveyor roller 115, second conveyor roller 116, third conveyor roller 119, and / or fourth conveyor roller 120 is not limited to one, and may be more than one. In this case, the multiple feed rollers 112, separation rollers 113, first conveyor roller 115, second conveyor roller 116, third conveyor roller 119, and / or fourth conveyor roller 120 are arranged at intervals in the width direction A2.

[0029] The image reading device 100 has a so-called straight path. The top surface of the lower housing 101 forms a lower guide 101a of the medium transport path, and the bottom surface of the upper housing 102 forms an upper guide 102a of the medium transport path.

[0030] The first medium sensor 111 is disposed upstream of the feed roller 112 and the separation roller 113. The first medium sensor 111 has a contact detection sensor and detects whether or not a medium is placed on the placement table 103. The first medium sensor 111 generates and outputs a placement signal whose signal value changes depending on whether or not a medium is placed on the placement table 103. Note that the first medium sensor 111 is not limited to a contact detection sensor, and any other sensor capable of detecting the presence or absence of a medium, such as a light detection sensor, may be used as the first medium sensor 111.

[0031] The feed roller 112 is provided in the lower housing 101, and separates and feeds the media placed on the mounting table 103 from the bottom up. The separation roller 113 is a so-called brake roller or retard roller, and is provided in the upper housing 102, disposed opposite the feed roller 112, and provided so as to be rotatable or stoppable in the direction A4 opposite to the medium feeding direction.

[0032] The first conveying roller 115 and the second conveying roller 116 are disposed downstream of the feed roller 112 and facing each other, and convey the medium fed by the feed roller 112 and the separation roller 113 to the imaging device 118. Either the first conveying roller 115 or the second conveying roller 116 may be a driven roller that rotates following the rotation of the other roller.

[0033] The imaging device 118 is disposed downstream of the first conveying roller 115 and the second conveying roller 116, and captures an image of the medium conveyed by the first conveying roller 115 and the second conveying roller 116. The imaging device 118 includes a first imaging device 118a and a second imaging device 118b disposed opposite each other across the medium conveying path. The first imaging device 118a has a line sensor based on a CIS (Contact Image Sensor) of a life-size optical system type having imaging elements based on CMOS (Complementary Metal Oxide Semiconductor) linearly arranged in the main scanning direction. The first imaging device 118a also has a lens that forms an image on the imaging element, an A / D converter that amplifies and analog-to-digital (A / D) converts the electrical signal output from the imaging element, and a backing (reference member) that faces the line sensor of the second imaging device 118b. The first imaging device 118a generates and outputs an input image by capturing an image of the surface of the medium being conveyed according to control from a processing circuit described later.

[0034] Similarly, the second imaging device 118b has a line sensor using a CIS of a life-size optical system type having CMOS imaging elements arranged in a line in the main scanning direction. The second imaging device 118b also has a lens that forms an image on the imaging element, an A / D converter that amplifies and analog-to-digital (A / D) converts the electrical signal output from the imaging element, and a backing (reference member) that faces the line sensor of the first imaging device 118a. The second imaging device 118b generates and outputs an input image by imaging the back side of the medium being conveyed according to control from a processing circuit described later.

[0035] The image reading device 100 may have only one of the first imaging device 118a and the second imaging device 118b arranged, and may read only one side of the medium. Also, instead of a CIS line sensor of an equal magnification optical system type having a CMOS imaging element, a CIS line sensor of an equal magnification optical system type having a CCD (Charge Coupled Device) imaging element may be used. Also, a reduction optical system type line sensor having a CMOS or CCD imaging element may be used.

[0036] The third conveying roller 119 and the fourth conveying roller 120 are disposed facing each other downstream of the imaging device 118, and discharge the medium that has been conveyed by the first conveying roller 115 and the second conveying roller 116 and imaged by the imaging device 118 onto the discharge tray 104. Either one of the third conveying roller 119 and the fourth conveying roller 120 may be a driven roller that rotates following the rotation of the other roller.

[0037] The medium placed on the placement table 103 is transported between the lower guide 101a and the upper guide 102a in the medium transport direction A1 by the rotation of the feed roller 112 in the direction of the arrow A3 in Fig. 2, i.e., the medium feed direction. The separation roller 113 rotates or stops in the direction of the arrow A4, i.e., the opposite direction to the medium feed direction, during medium transport. This limits the transport of media other than the separated medium (preventing double feeding).

[0038] The medium is fed between the first conveyor roller 115 and the second conveyor roller 116 while being guided by the lower guide 101a and the upper guide 102a. The medium is fed between the first imaging device 118a and the second imaging device 118b as the first conveyor roller 115 and the second conveyor roller 116 rotate in the directions of the arrows A5 and A6, respectively. The medium read by the imaging device 118 is discharged onto the discharge tray 104 as the third conveyor roller 119 and the fourth conveyor roller 120 rotate in the directions of the arrows A7 and A8, respectively.

[0039] FIG. 3 is a schematic diagram for explaining each sensor for detecting the medium.

[0040] 3, the number of each of the feed rollers 112, the second conveyor rollers 116, and the fourth conveyor rollers 120 is two. In this case, the number of each of the separation rollers 113, the first conveyor rollers 115, and the third conveyor rollers 119 arranged opposite the feed rollers 112, the second conveyor rollers 116, and the fourth conveyor rollers 120, respectively, is also two. In addition, the number of the second medium sensors 114 is two, and the number of the third medium sensors 117 is one. Note that the number of the second medium sensors 114 and / or the third medium sensors 117 may be any number.

[0041] The second medium sensor 114 detects the medium transported to its arrangement position. In the example shown in FIG. 3, two second medium sensors 114 are arranged side by side with an interval in the width direction A2. The second medium sensor 114 is arranged downstream of the nip portion N1 between the feed roller 112 and the separation roller 113 in the medium transport direction A1, and on each of both sides of the medium transport path in the width direction A2 perpendicular to the medium transport direction. In particular, the second medium sensor 114 is arranged upstream of the imaging device 118 in the medium transport direction A1, and particularly upstream of the nip portion N2 between the first transport roller 115 and the second transport roller 116. In addition, the second medium sensor 114 is arranged inside (toward the center) the side wall 101b of the medium transport path in the width direction A2. The second medium sensor 114 is disposed in the width direction A2 outside (towards the side wall 101b) of a medium of the maximum size supported by the image reading device 100 when placed on the placement table 103, that is, outside the position of the inner side surface of the side guide 103b when placed at the outermost position. The second medium sensor 114 is also disposed outside the end position of the imaging range of the imaging device 118 in the width direction A2.

[0042] The second medium sensor 114 includes a light emitter and a light receiver provided on one side of the medium transport path, and a light guide tube provided at a position facing the light emitter and the light receiver across the medium transport path. The light emitter is an LED (Light Emitting Diode) or the like, and emits light toward the medium transport path. On the other hand, the light receiver is a photodiode or the like, and receives the light emitted by the light emitter and guided by the light guide tube. When a medium is present at a position facing at least one of the light emitter and the light receiver, the light emitted from the light emitter is blocked by the medium, so that the light receiver does not detect the light emitted from the light emitter. The second medium sensor 114 generates and outputs a second medium signal whose signal value changes depending on whether a medium is present or not at the position of the second medium sensor 114 based on the intensity of the light received by the light receiver.

[0043] The third media sensor 117 is disposed downstream of the nip portion N1 between the feed roller 112 and the separation roller 113 in the media transport direction A1, and at the center of the media transport path in the width direction A2 perpendicular to the media transport direction. In particular, the third media sensor 117 is disposed downstream of the second media sensor 114 in the media transport direction A1, particularly downstream of the nip portion N2 between the first transport roller 115 and the second transport roller 116, and upstream of the imaging device 118. In addition, the third media sensor 117 is disposed between the multiple feed rollers 112 (nip portion N1) and / or between the multiple first transport rollers 115 (nip portion N2) in the width direction A2.

[0044] The third medium sensor 117 includes a light emitter and a light receiver provided on one side of the medium transport path, and a light guide tube provided at a position facing the light emitter and the light receiver across the medium transport path. The light emitter is an LED or the like, and emits light toward the medium transport path. On the other hand, the light receiver is a photodiode or the like, and receives the light emitted by the light emitter and guided by the light guide tube. When a medium is present at a position facing at least one of the light emitter and the light receiver, the light emitted from the light emitter is blocked by the medium, so that the light receiver does not detect the light emitted from the light emitter. The third medium sensor 117 generates and outputs a third medium signal whose signal value changes depending on whether a medium is present or not at the position of the third medium sensor 117, based on the intensity of the light received by the light receiver.

[0045] In addition, in second medium sensor 114 and / or third medium sensor 117, a reflective member such as a mirror may be used instead of a light guide tube. In second medium sensor 114 and / or third medium sensor 117, the light emitter and the light receiver may be provided facing each other across the medium transport path. In addition, second medium sensor 114 and / or third medium sensor 117 may detect the presence of the medium using a contact detection sensor or the like that passes a predetermined current when the medium is in contact or when the medium is not in contact. In addition, second medium sensor 114 and / or third medium sensor 117 may detect the presence of the medium using an ultrasonic sensor.

[0046] FIG. 4 is a block diagram showing a schematic configuration of the image reading device 100 and the information processing device 200. As shown in FIG.

[0047] In addition to the above-mentioned components, the image reading device 100 further includes a motor 121, a first communication device 122, a first storage device 130, a first processing circuit 140, and the like.

[0048] The motor 121 includes one or more motors. The motor 121 rotates the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and / or the fourth conveyor roller 120 in response to a control signal from the first processing circuit 140 to convey the medium.

[0049] The first communication device 122 is an example of an output unit and a communication unit. The first communication device 122 has an antenna for transmitting and receiving wireless signals, and a wireless communication interface circuit for transmitting and receiving signals through a wireless communication line according to a communication protocol such as a wireless LAN (Local Area Network). The first communication device 122 can communicate with the information processing device 200, and is communicatively connected to the information processing device 200 to transmit and receive various images and information. The first communication device 122 may have a wired communication interface circuit according to a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol), and may be connected to the information processing device 200 via a network. The first communication device 122 may also have an interface circuit conforming to a serial bus such as a Universal Serial Bus (USB), and may be connected to the information processing device 200 via a wired cable such as a USB cable.

[0050] The first storage device 130 includes a memory device such as a random access memory (RAM) or a read only memory (ROM), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or an optical disk. The first storage device 130 also stores computer programs, databases, tables, and the like used for various processes of the image reading device 100. The computer programs may be installed in the first storage device 130 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or the like.

[0051] The first processing circuit 140 operates based on a program stored in advance in the first storage device 130. The processing circuit is, for example, a CPU (Central Processing Unit). As the first processing circuit 140, a DSP (digital signal processor), an LSI (large scale integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like may be used.

[0052] The first processing circuit 140 is connected to the first operation device 105, the first display device 106, the first medium sensor 111, the second medium sensor 114, the third medium sensor 117, the imaging device 118, the motor 121, the first communication device 122, the first storage device 130, and the like, and controls each of these components. The first processing circuit 140 performs drive control of the motor 121, imaging control of the imaging device 118, and the like, based on each signal received from each sensor, acquires an input image from the imaging device 118, and transmits it to the information processing device 200 via the first communication device 122.

[0053] The information processing device 200 includes a second display device 202, a second operation device 201, a second communication device 203, a second storage device 210, a second processing circuit 220, and the like.

[0054] The second operation device 201 has input devices such as a keyboard and a mouse, and an interface circuit that acquires signals from the input devices.

[0055] The second display device 202 is an example of an output unit. The second display device 202 has a display made of liquid crystal, organic EL, or the like, and an interface circuit that outputs image data to the display, and displays image data on the display according to an instruction from the second processing circuit 220. The second display device 202 and the second operating device 201 may be provided integrally using a touch panel type input device.

[0056] The second communication device 203 is an example of an output unit and a communication unit. The second communication device 203 has an interface circuit similar to that of the first communication device 122. The second communication device 203 is capable of communicating with the image reading device 100, and is communicatively connected to the image reading device 100 to transmit and receive various images and information.

[0057] The second storage device 210 is an example of a storage unit, and includes a memory device such as a RAM or a ROM, a fixed disk device such as a hard disk, or a portable storage device such as an optical disk. The second storage device 210 also stores computer programs, databases, tables, and the like used for various processes of the information processing device 200. The computer programs may be installed in the second storage device 210 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like.

[0058] The second processing circuit 220 operates based on a program previously stored in the second storage device 210. The processing circuit is, for example, a CPU. The second processing circuit 220 may be, for example, a DSP, an LSI, an ASIC, an FPGA, or the like.

[0059] The second processing circuit 220 is connected to the second operation device 201, the second display device 202, the second communication device 203, the second storage device 210, etc., and controls each of these units. The second processing circuit 220 controls each device, acquires an input image from the image reading device 100, and performs image processing on the acquired input image.

[0060] FIG. 5A is a diagram showing a schematic configuration of the first storage device 130 and the first processing circuit 140 of the image reading device 100. As shown in FIG.

[0061] 5(A), the first storage device 130 stores a control program 131, a transmission program 132, and the like. Each of these programs is a functional module implemented by software that runs on a processor. The first processing circuit 140 reads each program stored in the first storage device 130 and operates according to the read program. In this way, the first processing circuit 140 functions as a control unit 141 and a transmission unit 142.

[0062] FIG. 5B is a diagram showing a schematic configuration of the second storage device 210 and the second processing circuit 220 of the information processing device 200. As shown in FIG.

[0063] As shown in FIG. 5B, the second storage device 210 stores an acquisition program 211, an extraction program 212, a detection program 213, an image loss determination program 214, a content determination program 215, a correction program 216, a notification program 217, an output control program 218, and the like. Each of these programs is a functional module implemented by software that runs on a processor. The second processing circuit 220 reads each program stored in the second storage device 210 and operates according to each program that has been read. As a result, the second processing circuit 220 functions as an acquisition unit 221, an extraction unit 222, a detection unit 223, an image loss determination unit 224, a content determination unit 225, a correction unit 226, a notification unit 227, and an output control unit 228. The image loss determination unit 224 is an example of a determination unit.

[0064] FIG. 6 is a flowchart showing an example of the operation of the image reading process of the image reading device 100.

[0065] An example of the operation of the image reading process of the image reading device 100 will be described below with reference to the flowchart shown in Fig. 6. The flow of the operation described below is executed mainly by the first processing circuit 140 in cooperation with each element of the image reading device 100 based on a program stored in advance in the first storage device 130.

[0066] First, the control unit 141 waits until a user inputs an instruction to read a medium using the first operating device 105 or an information processing device, and an operation signal instructing the user to read a medium is received from the first operating device 105 or the first communication device 122 (step S101).

[0067] Next, control unit 141 acquires a first medium signal from first medium sensor 111, and determines whether or not a medium is placed on placement table 103 based on the acquired first medium signal (step S102). If no medium is placed on placement table 103, control unit 141 ends the series of steps.

[0068] On the other hand, when a medium is placed on the placement table 103, the control unit 141 drives the motor 121. As a result, the control unit 141 rotates the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and / or the fourth conveyor roller 120 to feed and convey the medium (step S103).

[0069] Next, the control unit 141 sets the detection flag to OFF (step S104). The detection flag is a flag that indicates whether the second medium sensor 114 has detected a medium, that is, whether the transported medium has passed outside the imaging range of the imaging device 118.

[0070] Next, control unit 141 determines whether the medium has passed outside the imaging range of imaging device 118 (step S105). Control unit 141 periodically acquires a second medium signal from each second medium sensor 114. When the signal value of the second medium signal indicates that a medium is present, control unit 141 determines that the medium has passed the position of second medium sensor 114 that transmitted the second medium signal, and determines that the medium has passed outside the imaging range of imaging device 118. When the medium has not passed outside the imaging range of imaging device 118, control unit 141 transitions to the process of step S107.

[0071] On the other hand, if the medium passes outside the imaging range of imaging device 118, control unit 141 sets the detection flag to ON and stores information indicating second medium sensor 114 that detected the medium in first storage device 130. Control unit 141 also stores the imaging start time of the medium and the detection time when second medium sensor 114 detected the medium in first storage device 130 (step S106).

[0072] Next, control unit 141 determines whether the rear end of the medium has passed the imaging position of imaging device 118 (step S107). Control unit 141 periodically acquires a third medium signal from third medium sensor 117. Control unit 141 determines that the rear end of the medium has passed the position of third medium sensor 117 when the signal value of the third medium signal changes from a value indicating the presence of the medium to a value indicating the absence of the medium. Control unit 141 determines that the rear end of the medium has passed the imaging position when a predetermined time has elapsed since the rear end of the medium passed the position of third medium sensor 117. The predetermined time is set to the time required for the medium to move from the position of third medium sensor 117 to the imaging position of imaging device 118. Note that control unit 141 may determine that the rear end of the medium has passed the imaging position when a predetermined time has elapsed since feeding of the medium began.

[0073] Next, the transmission unit 142 acquires an input image from the imaging device 118, stores the acquired input image in the first storage device 130, and outputs it by transmitting it to the information processing device 200 via the first communication device 122 (step S108). Note that the transmission unit 142 may acquire image data from the imaging device 118 for each predetermined unit (e.g., three lines) and transmit it to the information processing device 200, rather than acquiring the entire input image from the imaging device 118 and transmitting it to the information processing device 200. In this case, the information processing device 200 combines each image data to generate an input image.

[0074] Next, control unit 141 determines whether or not a medium remains on mounting table 103 based on a first medium signal obtained from first medium sensor 111 (step S109). If a medium remains on mounting table 103, control unit 141 returns the process to step S104, and repeats the processes of steps S104 to S109.

[0075] On the other hand, if no media remain on the placement table 103, the control unit 141 stops the motor 121. As a result, the control unit 141 stops the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and / or the fourth conveyor roller 120 (step S110), and the series of steps ends.

[0076] It is noted that the processes of steps S104 to S106 may be omitted, and the control unit 141 may not need to determine whether or not the medium has passed outside the imaging range of the imaging device 118.

[0077] FIG. 7 is a flowchart showing an example of the operation of the determination process of the information processing device 200.

[0078] An example of the operation of the determination process of the information processing device 200 will be described below with reference to the flowchart shown in Fig. 7. The flow of the operation described below is executed mainly by the second processing circuit 220 in cooperation with each element of the information processing device 200 based on a program previously stored in the second storage device 210. The determination process is executed every time an input image is received from the image reading device 100.

[0079] First, the acquisition unit 221 acquires an input image by receiving it from the image reading device 100 via the second communication device 203, and stores it in the second storage device 210 (step S201). As described above, the determination process is executed every time an input image is received from the image reading device 100, and the acquisition unit 221 acquires a plurality of input images and stores them in the second storage device 210. An input image acquired before or after the input image to be processed, i.e., an input image in which a medium is captured before or after the input image to be processed, is an example of a second input image.

[0080] Next, the extraction unit 222 extracts edge pixels from the input image (step S202). The extraction unit 222 calculates, for each horizontal line extending in the horizontal direction (main scanning direction) in the input image, the absolute value of the difference in gradation values ​​between the pixels adjacent to each pixel in each horizontal line, starting from the left end, as an adjacent difference value. The extraction unit 222 extracts pixels in each horizontal line whose adjacent difference value exceeds a gradation threshold as edge pixels. The gradation value is a luminance value or a color value (R value, G value, or B value), etc. The gradation threshold is set to a luminance value difference (e.g., 20) that allows a person to visually distinguish the difference in luminance on the image. The extraction unit 222 determines the edge pixel detected first in each horizontal line, i.e., the pixel located at the leftmost position, as the left edge pixel, and the edge pixel detected last in each horizontal line, i.e., the pixel located at the rightmost position, as the right edge pixel. The extraction section 222 may detect edge pixels in order from the right end side, and may determine that the first edge pixel detected in each horizontal line is the right end edge pixel.

[0081] Similarly, the extraction unit 222 calculates, for each vertical line extending in the vertical direction (sub-scanning direction) in the input image, the absolute value of the difference in gradation values ​​between the pixels adjacent to each pixel in each vertical line in order from the top end side as an adjacent difference value. The extraction unit 222 extracts pixels in each vertical line whose adjacent difference value exceeds a gradation threshold value as edge pixels. The extraction unit 222 determines the edge pixel detected first in each vertical line, i.e., the pixel located at the top, as the top edge pixel, and the edge pixel detected last in each vertical line, i.e., the pixel located at the bottom, as the bottom edge pixel. The extraction unit 222 may detect edge pixels in order from the bottom end side, and determine the edge pixel detected first in each vertical line as the bottom edge pixel. Hereinafter, the left edge pixel, the right edge pixel, the top edge pixel, and the bottom edge pixel may be collectively referred to as end edge pixels.

[0082] The extraction unit 222 may calculate, as the adjacent difference value, the absolute value of the difference in gradation values ​​between two pixels that are a predetermined distance away from each pixel in the input image in the horizontal or vertical direction. The extraction unit 222 may also extract edge pixels by comparing the gradation value of each pixel in the input image with a threshold. For example, when the gradation value of a specific pixel is less than the threshold and the gradation value of a pixel adjacent to the specific pixel in the horizontal or vertical direction or a pixel that is a predetermined distance away from the specific pixel is equal to or greater than the threshold, the extraction unit 222 extracts the specific pixel as an edge pixel.

[0083] The extraction unit 222 may extract edge pixels at regular intervals (e.g., four pixels) in the horizontal or vertical direction in the input image, instead of extracting edge pixels from all pixels in the input image. The extraction unit 222 extracts target lines for detecting edge pixels at regular intervals from among horizontal or vertical lines in the input image, and extracts edge pixels from the extracted target lines. The extraction unit 222 may also group pixels in the input image into groups of pixels arranged continuously in the horizontal or vertical direction, and extract edge pixels based on the statistical values ​​(average, median, minimum, maximum, or variance, etc.) of the gradation values ​​of each pixel in each group. In this case, the extraction unit 222 calculates, as the adjacent difference value, the absolute value of the difference between the statistical value of each group and the statistical value of a group adjacent to that group, in order from the left end side or the top end side, for each horizontal or vertical line in the input image. The extraction unit 222 detects, as an edge pixel, a representative pixel of a group whose adjacent difference value exceeds a gradation threshold (e.g., a pixel arranged in the center of the group, or a pixel arranged in a position closest to an adjacent group). As a result, the extraction unit 222 can reduce the time required to detect the edge of the medium, and can reduce the processing time and processing load of the medium reading process. The extraction unit 222 may extract edge pixels using any known edge detection method other than the above-mentioned methods.

[0084] FIG. 8 is a schematic diagram showing an example of an input image N obtained by capturing an image of a medium.

[0085] 8 includes a medium M having rounded corners. In the medium M, the upper left corner, lower left corner, and lower right corner are rounded, but the upper right corner is cut diagonally (in a straight line). In the input image N, a left edge pixel E1 is detected from the left side of the medium M, a right edge pixel E2 is detected from the right side of the medium M, an upper edge pixel E3 is detected from the upper side of the medium M, and a lower edge pixel E4 is detected from the lower side of the medium M.

[0086] Next, the detection unit 223 detects four straight lines corresponding to the four sides of the medium from the end edge pixels extracted by the extraction unit 222 (step S203). The detection unit 223 detects a left end straight line corresponding to the left side of the medium from the left end edge pixels, a right end straight line corresponding to the right side of the medium from the right end edge pixels, an upper end straight line corresponding to the upper side of the medium from the upper end edge pixels, and a lower end straight line corresponding to the lower side of the medium from the lower end edge pixels. The detection unit 223 detects straight lines (line segments) from each end edge pixel using, for example, the least squares method. The detection unit 223 may detect straight lines using a Hough transform.

[0087] When multiple straight lines are detected from each of the left end edge pixels, the right end edge pixels, the top end edge pixels, or the bottom end edge pixels, the detection unit 223 detects the straight line having the largest number of end edge pixels present within a predetermined distance from each straight line as the left end straight line, the right end straight line, the top end straight line, or the bottom end straight line. The predetermined distance is set to a sufficiently small distance (e.g., 3 pixels). Alternatively, the detection unit 223 extracts all combinations of four straight lines detected from each end edge pixel. The detection unit 223 detects the straight lines included in the combination having the smallest sum of the difference between the angle formed by each of the straight lines that are approximately perpendicular to each other and 90° among the extracted combinations as the left end straight line, the right end straight line, the top end straight line, and the bottom end straight line. The combination of either the left end straight line or the right end straight line and either the top end straight line or the bottom end straight line is an example of two straight lines that are approximately perpendicular to each other detected from edge pixels.

[0088] In the input image N shown in Fig. 8, each corner of the medium M is rounded or cut diagonally (straight), so some of the end edge pixels do not lie on a straight line. Therefore, multiple straight lines are detected from each end edge pixel. In the input image N, a left edge straight line L1, a right edge straight line L2, an upper edge straight line L3, and a lower edge straight line L4 are detected from the left edge pixel E1, the right edge pixel E2, the upper edge pixel E3, and the lower edge pixel E4, respectively.

[0089] Next, the detection unit 223 detects a plurality of corner regions, particularly four corner regions, each of which is estimated to include a corner of the medium in the input image (step S204). The detection unit 223 detects the intersection of the left end line and the top end line as the upper left intersection, the intersection of the right end line and the top end line as the upper right intersection, the intersection of the left end line and the bottom end line as the lower left intersection, and the intersection of the right end line and the bottom end line as the lower right intersection. The upper left intersection, the upper right intersection, the lower left intersection, and the lower right intersection are examples of the intersection of two lines that are approximately perpendicular to each other and detected from edge pixels.

[0090] The detection unit 223 detects an area in the input image that includes the upper left intersection as an upper left corner area that is estimated to include the upper left corner of the medium. The detection unit 223 detects an area in the input image that includes the upper right intersection as an upper right corner area that is estimated to include the upper right corner of the medium. The detection unit 223 detects an area in the input image that includes the lower left intersection as a lower left corner area that is estimated to include the lower left corner of the medium. The detection unit 223 detects an area in the input image that includes the lower right intersection as a lower right corner area that is estimated to include the lower right corner of the medium. The detection unit 223 sets square areas of the same size as each corner area. The detection unit 223 sets each corner area such that the distances from the ends of the corner areas opposite the center of the medium to each intersection are the same in the horizontal and vertical directions, and the distances from the ends of the corner areas closer to the center of the medium to each intersection are the same in the horizontal and vertical directions. It is preferable that each corner region is set so that each intersection point is located near the end of each corner region opposite the center of the medium.

[0091] The detection unit 223 stores position information (coordinate information) within the input image, which indicates the detected corner region, in the second storage device 210. That is, the detection unit 223 detects corner regions that are estimated to include corners of the medium within each of a plurality of input images, and stores position information indicating the detected corner regions in the second storage device 210.

[0092] 8, an upper left intersection point V1, an upper right intersection point V2, a lower left intersection point V3, and a lower right intersection point V4 are detected from the left end line L1 and the upper end line L3, the right end line L2 and the upper end line L3, the left end line L1 and the lower end line L4, and the right end line L2 and the lower end line L4, respectively. Also, square regions including the upper left intersection point V1, the upper right intersection point V2, the lower left intersection point V3, and the lower right intersection point V4, respectively, are detected as the upper left corner region R1, the upper right corner region R2, the lower left corner region R3, and the lower right corner region R4.

[0093] Next, the image missing determination unit 224 executes an image missing determination process (step S205). In the image missing determination process, the image missing determination unit 224 determines whether or not image missing occurs in the corner area detected by the detection unit 223, and identifies the corner area where image missing occurs. Image missing refers to a state in which a part of the medium, particularly a corner of the medium, is not included in the image. The image missing determination process will be described in detail later.

[0094] The following processing of steps S206 to S214 is performed for each corner region detected by detection unit 223. First, content determination unit 225 determines whether or not image loss determination unit 224 has determined that image loss has occurred in the target corner region in the image loss determination process (step S206). If it has been determined that no image loss has occurred in the target corner region, content determination unit 225 does not perform any particular processing, and performs the processing of steps S206 to S214 for the other corner regions.

[0095] On the other hand, if it is determined that image loss occurs in the target corner region, the content determination unit 225 executes a content determination process (step S207). In the content determination process, the content determination unit 225 determines whether or not content exists around the edge of the medium in the corner region determined by the image loss determination unit 224 as having image loss. Furthermore, if the content determination unit 225 determines that content exists, it identifies the type of content (characters, lines, patterns, images, etc.). The content determination process will be described in detail later.

[0096] Next, the content determination unit 225 determines whether or not it has been determined in the content determination process that content exists around the edge of the medium in the corner region in which the image loss determination unit 224 has determined that image loss has occurred (step S208).

[0097] When the content determination unit 225 determines that no content exists around the edge of the medium in the corner area to be determined, the correction unit 226 forms (restores, corrects) the corner in the corner area determined to have image loss, and corrects the corner area (step S209). For example, the correction unit 226 corrects the corner area determined to have image loss by using the corner area determined to have no image loss by the content determination unit 225. The correction unit 226 generates a correction corner area whose orientation (arrangement direction) matches that of the corner area where image loss occurs by rotating the corner area where image loss does not occur by 90°, 180°, or 270°, or by inverting it vertically, horizontally, or diagonally. The correction unit 226 corrects the corner area where image loss occurs by arranging (copying) the correction corner area in the corner area where image loss occurs.

[0098] The correction unit 226 may correct only the missing portion in the corner region where the image loss occurs. In this case, the correction unit 226 sets the area outside each end edge pixel (opposite the center of the medium) in the corner region where the image loss occurs as the correction target area. In the upper left corner region, the area outside each end edge pixel is the area to the left and above each end edge pixel. In the upper right corner region, the area outside each end edge pixel is the area to the right and above each end edge pixel. In the lower left corner region, the area outside each end edge pixel is the area to the left and below each end edge pixel. In the lower right corner region, the area outside each end edge pixel is the area to the right and below each end edge pixel. The correction target area is set to include the end edge pixel itself. The correction target area may be set not to include the end edge pixel itself. The correction unit 226 corrects the corner region where the image loss occurs by arranging (copying) a corresponding area of ​​the correction corner region in the correction target area of ​​the corner region where the image loss occurs. This allows the correction unit 226 to restore the medium edge in the corner region while maintaining the content (non-missing portion) included in the corner region where the image loss occurs.

[0099] The correction unit 226 may also determine whether or not content exists in a corner area where no image loss occurs in the same manner as in the content determination process, and correct the corner area where image loss occurs using the corner area where it is determined that no content exists. This allows the correction unit 226 to prevent erroneous copying of content included in another corner area to the corner area where image loss occurs.

[0100] In the example shown in FIG. 8, it is determined that no image loss occurs in the upper left corner region R1, the lower left corner region R3, and the lower right corner region R4, and that image loss occurs in the upper right corner region R2. The correction unit 226 corrects the upper right corner region R2 by rotating the upper left corner region R1 by 90° clockwise and arranging the upper left intersection point V1 in the upper right corner region R2 so that it coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by rotating the lower left corner region R3 by 180° clockwise and arranging the lower left intersection point V3 in the upper right corner region R2 so that it coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by rotating the lower right corner region R4 by 270° clockwise and arranging the lower right intersection point V4 in the upper right corner region R2 so that it coincides with the upper right intersection point V2.

[0101] Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the upper left corner region R1 horizontally across a predetermined vertical line, and then arranging the upper left intersection point V1 in the upper right corner region R2 so that the upper left intersection point V1 coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the lower left corner region R3 across a diagonal line that extends from the upper left to the lower right and is inclined at 45° with respect to the horizontal and vertical lines, and then arranging the lower left intersection point V3 in the upper right corner region R2 so that the lower left intersection point V3 coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the lower right corner region R4 vertically across a predetermined horizontal line, and then arranging the lower right intersection point V4 in the upper right corner region R2 so that the lower right intersection point V4 coincides with the upper right intersection point V2.

[0102] On the other hand, if the content determination unit 225 determines in step S208 that content exists around the medium edge in the target corner region, the correction unit 226 determines whether or not the content is correctable (step S210). If the type of content identified by the content determination unit 225 is a character, a ruled line, or a pattern, the correction unit 226 determines that the content is correctable. On the other hand, if the type of content identified by the content determination unit 225 is something other than a character, a ruled line, or a pattern (such as an image), the correction unit 226 determines that the content is not correctable.

[0103] If it is determined that the content can be corrected, the correction unit 226 corrects the corner area in which it is determined that image loss has occurred, in the same manner as in the process of step S209 (step S211).

[0104] Next, the correction unit 226 executes a content correction process (step S212). In the content correction process, the correction unit 226 corrects a corner region in which the image loss determination unit 224 has determined that an image loss has occurred and in which the content determination unit 225 has determined that content exists around the edge of the medium, based on the surrounding pixels of the edge of the medium in the corner region. Details of the content correction process will be described later.

[0105] On the other hand, if it is determined in step S210 that the content cannot be corrected, the correction unit 226 does not correct the corner region (step S213). In this way, the correction unit 226 corrects a corner region determined by the image loss determination unit 224 as having image loss and determined by the content determination unit 225 as having no content around the medium edge. On the other hand, the correction unit 226 does not correct a corner region determined by the content determination unit 225 as having content around the medium edge and determined by the image loss determination unit 224 as having image loss. In this way, the correction unit 226 can appropriately restore a corner region damaged in the input image, while suppressing damage to the content in the input image by correcting the corner region.

[0106] Next, the notification unit 227 notifies the user that image loss has occurred by displaying the information on the second display device 202 or transmitting the information to the terminal device used by the user via the second communication device 203 (step S214). In this way, the notification unit 227 notifies the user that image loss has occurred when there is a corner area where the content determination unit 225 has determined that content exists around the edge of the medium, particularly when there is a corner area where irreparable content exists. This allows the user to recognize that image loss has occurred in the input image. As a result, the user can repair the state of the medium as necessary and then have the image reading device 100 reread the medium, and the image processing system 1 can improve the convenience for the user.

[0107] When the processes of steps S206 to S214 have been performed for all corner regions detected by the detection unit 223, the output control unit 228 outputs the determination result of the input image and / or image loss (step S215) and ends the series of steps. The output control unit 228 outputs the determination result of the input image and / or image loss by displaying it on the second display device 202 or transmitting it to a terminal device used by the user via the second communication device 203. When the input image has not been corrected by the correction unit 226, the output control unit 228 outputs the input image received from the image reading device 100, and when the input image has been corrected by the correction unit 226, the output control unit 228 outputs the input image corrected by the correction unit 226. The determination result of image loss includes whether or not image loss occurs in the input image, whether or not content exists around the edge of the medium in the corner region where it is determined that image loss occurs, and / or whether or not the corner region where image loss occurs has been corrected. The determination result of the input image and / or image loss is an example of information on the determination result by the image loss determination unit 224.

[0108] The processes of steps S206 to S214 may be omitted, and the second processing circuit 220 may not execute correction processing on the corner region where image loss has occurred. The processes of steps S210 to S212 may be omitted, and the correction unit 226 may not correct the corner region when content exists around the edge of the medium in the corner region, regardless of whether the content is correctable or not. The processes of steps S210, S213, and S214 may be omitted, and the correction unit 226 may correct the corner region when content exists around the edge of the medium in the corner region, regardless of whether the content is correctable or not. The processes of steps S212 and / or S214 may be omitted. In the process of step S204, the detection unit 223 may detect any number of corner regions (any one, two, or three) instead of detecting four corner regions. Furthermore, the processes in steps S206 to S214 do not have to be performed for all corner regions, but may be performed for only an arbitrary number of corner regions (any one, two, or three).

[0109] 9 is a flowchart showing an example of the operation of the image loss determination process. The image loss determination process is executed in step S205 of the determination process shown in FIG.

[0110] First, the image missing determination unit 224 calculates the similarity between the multiple corner regions detected by the detection unit 223 (step S301). The image missing determination unit 224 extracts all combinations (pairs) of two corner regions from among all corner regions detected by the detection unit 223. The image missing determination unit 224 calculates the similarity between the two corner regions included in each combination for all the extracted combinations. The image missing determination unit 224 aligns the orientation of the two corner regions by rotating at least one of the two corner regions by 90°, 180°, or 270°, or by inverting the two corner regions up and down, left and right, or diagonally. The image missing determination unit 224 calculates, for example, the normalized cross-correlation (NCC) of the two corner regions as the similarity. The image missing determination unit 224 may calculate, as the similarity, the inverse of the Sum of Squared Difference (SSD) or the inverse of the Sum of Absolute Difference (SAD) of the two corner regions, or the like.

[0111] Next, the image loss determination unit 224 classifies the multiple corner regions into groups based on the calculated similarity between the multiple corner regions (step S302). The image loss determination unit 224 classifies two corner regions whose similarity is equal to or greater than a similarity threshold into the same group. The image loss determination unit 224 also classifies into a specific group a corner region whose similarity to a corner region classified into the specific group is equal to or greater than a similarity threshold. The similarity threshold is set to a value between the similarity calculated for a corner region without image loss and the similarity calculated for a corner region with image loss, based on a prior experiment.

[0112] Next, the image loss determination unit 224 determines whether or not the multiple corner regions detected by the detection unit 223 have been classified into two or more groups (step S303).

[0113] When all corner regions detected by detection unit 223 are classified into one group, image loss determination unit 224 determines that no image loss has occurred in any corner region (step S304), and ends the series of steps.

[0114] On the other hand, when the corner areas detected by the detection unit 223 are classified into two or more groups, the image loss determination unit 224 determines that image loss has occurred in one of the corner areas (step S305). The media to be transported include media with right-angled corners such as PPC (Plain Paper Copier) paper or business cards, and media with rounded corners such as ID (Identification) cards or passports specified by ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) 7810. For both media with right-angled corners and media with rounded corners, the shapes of the four corners are likely to be the same. On the other hand, it is unlikely that all corners of the medium are transported outside the imaging range of the imaging device 118 in the same way, and it is unlikely that all corners of the medium are folded in the same way, and therefore it is unlikely that the same type of image loss will occur in all corner areas. Therefore, if the shapes of all corner regions are similar, the image loss determination unit 224 determines that no image loss has occurred in any corner region, and if the shape of any corner region is different from the shape of the other corner regions, it determines that an image loss has occurred in one of the corner regions.

[0115] In this way, image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner areas by comparing the multiple corner areas detected by detection unit 223. This allows image loss determination unit 224 to correctly determine whether or not image loss has occurred even if the shape of the corner of the medium being transported is not recognized in advance. Therefore, image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes, such as right-angled corners or rounded corners.

[0116] In particular, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner regions based on the number of groups classified based on the degree of similarity between the multiple corner regions. This allows the image loss determination unit 224 to more accurately determine whether or not image loss has occurred for media having corners of various shapes, such as right angles or rounded corners.

[0117] Next, the image loss determination unit 224 identifies a corner region where image loss occurs based on the number of corner regions belonging to each classified group (step S306), and ends the series of steps. The image loss determination unit 224 determines that no image loss occurs in a corner region belonging to the group with the largest number of corner regions among the classified groups, and determines that image loss occurs in corner regions belonging to other groups. As described above, since the same type of image loss is unlikely to occur in each corner region, it is highly likely that no image loss occurs in corner regions having similar shapes, and that image loss occurs in corner regions having shapes dissimilar to the shapes of other corner regions. The image loss determination unit 224 can identify a corner region where image loss occurs with high accuracy by using the number of corner regions belonging to each classified group.

[0118] Furthermore, in the process of step S204, when the detection unit 223 detects only one corner region, the image loss determination unit 224 may determine whether or not image loss occurs in that corner region based only on that corner region. For example, when the distance between the intersection of two straight lines corresponding to the edge of the medium and the edge edge pixel located closest to the intersection in that corner region is equal to or less than a predetermined threshold, the image loss determination unit 224 determines that the corner is a right angle and that image loss does not occur. The threshold is set to a sufficiently small value (e.g., 3 pixels). On the other hand, when the distance between the intersection of two straight lines corresponding to the edge of the medium and the edge edge pixel located closest to the intersection in that corner region is greater than the threshold, the image loss determination unit 224 determines that the corner is not a right angle and that image loss occurs.

[0119] 10 is a flowchart showing an example of the operation of the content determination process, which is executed in step S207 of the determination process shown in FIG.

[0120] First, the content determination unit 225 detects the medium edge in the corner region determined to have image loss by the image loss determination unit 224 (step S401). The content determination unit 225 detects the edge pixel included in the corner region determined to have image loss as the medium edge.

[0121] The content determination unit 225 may detect only the end edge pixels corresponding to the missing portion among the end edge pixels included in the corner region where image loss occurs as the medium end. In this case, the content determination unit 225 detects only the end edge pixels not corresponding to the end edge pixels included in the corner region where image loss occurs as the medium end among the end edge pixels included in the corner region where image loss occurs. The content determination unit 225 selects one of the corner regions where image loss does not occur. The content determination unit 225 generates a correction corner region for the selected corner region, in the same manner as the process of step S209, with the orientation of the selected corner region being aligned with the corner region where image loss occurs. The content determination unit 225 detects, as the medium end, the end edge pixels that do not have a pixel corresponding to the end edge pixel in the correction corner region within the first distance among the end edge pixels included in the corner region where image loss occurs. The first distance is set to the number of pixels (for example, 10 pixels) that can be considered to have the same shape as the corner.

[0122] Next, the content determination unit 225 calculates characteristic values ​​of a plurality of pixels corresponding to the medium edge in the corner region where it is determined that image loss has occurred (step S402). The content determination unit 225 extracts pixels located in the corner region inside (toward the center of the medium) each end edge pixel detected in step S401 and within a second distance from each end edge pixel as a plurality of pixels corresponding to the medium edge. In the upper left corner region, the region inside each end edge pixel is the region to the right and below each end edge pixel. In the upper right corner region, the region inside each end edge pixel is the region to the left and below each end edge pixel. In the lower left corner region, the region inside each end edge pixel is the region to the right and above each end edge pixel. In the lower right corner region, the region inside each end edge pixel is the region to the left and above each end edge pixel. The plurality of pixels corresponding to the medium edge are set so as not to include the end edge pixel itself. The plurality of pixels corresponding to the medium edge may be set so as to include the end edge pixel itself. Hereinafter, the pixels corresponding to the edge of the medium may be referred to as edge surrounding pixels.

[0123] For example, the content determination unit 225 calculates the difference between the maximum and minimum gradation values ​​of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 may calculate the difference between the maximum or minimum gradation value and the average or median gradation value of each edge peripheral pixel as the characteristic value of the edge peripheral pixel.

[0124] FIG. 11A is a graph G1 showing an example of the gradation values ​​of pixels on the edge of a medium that includes characters or lines as content.

[0125] The horizontal axis of the graph G1 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in the graph G1, a white background (gradation value U1) exists in the area S1, a black ruled line (gradation value U2) exists in the area S2, and a black character (gradation value U3) exists in the area S3. In the graph G1, the difference between the maximum and minimum gradation values ​​of the pixels around the edge is the difference between the gradation value U1 of the background and the gradation value U3 of the character. If this difference is large, it is highly likely that the pixels around the edge contain content such as characters or rules, and if this difference is small, it is highly likely that the pixels around the edge do not contain content such as characters or rules. The content determination unit 225 can determine with high accuracy whether or not content such as characters or rules is included in the corner area determined to have image loss by using the difference in gradation values ​​of the pixels around the edge as the characteristic value of the pixels around the edge.

[0126] Furthermore, the content determination unit 225 may calculate the variance (or standard deviation) of the grayscale values ​​of each of the edge peripheral pixels as the characteristic value of the edge peripheral pixels.

[0127] FIG. 11B is a graph G2 showing an example of the gradation values ​​of pixels on the edge of a medium that includes a photograph as content.

[0128] The horizontal axis of graph G2 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in graph G2, a background exists in area S4, and a photograph exists in area S5. In the area containing the photograph, pixels having various gradation values ​​exist, and the gradation value of each pixel varies greatly. Therefore, when the variance of the gradation values ​​is large, it is highly likely that the edge surrounding pixels contain content such as a photograph, and when the variance of the gradation values ​​is small, it is highly likely that the edge surrounding pixels do not contain content such as a photograph. The content determination unit 225 can determine with high accuracy whether or not a corner area determined to have an image missing contains content such as a photograph by using the variance of the gradation values ​​of each edge surrounding pixel as the characteristic value of the edge surrounding pixel.

[0129] The content determination unit 225 may also calculate the periodicity of the gradation values ​​of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 calculates the periodicity so that the more pixels having the same gradation value exist periodically, the higher the value, and the more pixels having the same gradation value exist randomly, the lower the value. For example, the content determination unit 225 calculates the inverse of the variance (or standard deviation) of the number of consecutive pixels having gradation values ​​within a predetermined range in the horizontal or vertical direction as the periodicity of the gradation values ​​of each edge peripheral pixel. The predetermined range is a range of gradation values ​​that is considered to indicate the same color (for example, 200 or more and 255 or less for white, and 0 or more and 55 or less for black). The content determination unit 225 may set a plurality of predetermined ranges and calculate the average value of the inverse of the variance (or standard deviation) of the number of consecutive pixels having gradation values ​​within each predetermined range in the horizontal or vertical direction as the periodicity of the gradation values ​​of each edge peripheral pixel. The plurality of predetermined ranges are ranges of gradation values ​​corresponding to each of a plurality of colors such as white, gray, black, etc.

[0130] FIG. 11C is a graph G3 showing an example of the gradation values ​​of pixels on the edge of a medium that includes a pattern (background pattern) as content.

[0131] The horizontal axis of the graph G3 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in the graph G3, a hatched pattern exists throughout the image, white pixels (gradation value U6) exist in the area S6, and gray pixels (gradation value U7) exist in the area S7. As shown in the graph G3, pixels having a specific gradation value exist periodically in the area including the pattern such as hatching, dots, and lines. Therefore, when the periodicity of the gradation value is high, it is highly likely that the edge surrounding pixels include the content such as the pattern, and when the periodicity of the gradation value is low, it is highly likely that the edge surrounding pixels do not include the content such as the pattern. The content determination unit 225 can determine with high accuracy whether or not the corner area determined to have an image missing includes the content such as the pattern by using the periodicity of the gradation value of each edge surrounding pixel as the characteristic value of the edge surrounding pixel.

[0132] Furthermore, the content determination unit 225 may calculate the distribution of gradation values ​​of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 calculates the frequency of each gradation value as the distribution of gradation values ​​of each edge peripheral pixel. The content determination unit 225 may calculate the number of gradation values ​​whose frequency is equal to or greater than a frequency threshold, or the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than a frequency threshold, as the distribution of gradation values ​​of each edge peripheral pixel. The frequency threshold is set to a value (e.g., 3) that is sufficiently larger than the number of gradation values ​​that may occur due to noise or the like.

[0133] FIG. 11D is a graph G4 showing a histogram of the graph G1 shown in FIG. 11A.

[0134] The horizontal axis of graph G4 indicates the gradation value, and the vertical axis indicates the frequency of each gradation value. In the example shown in graph G4, the frequency is high around gradation value U1 corresponding to the white background, around gradation value U2 corresponding to the black ruled lines, and around gradation value U3 corresponding to the black characters, and is equal to or higher than the frequency threshold T1. That is, in the example shown in graph G4, there are three groups of consecutive gradation values ​​whose frequency is equal to or higher than the frequency threshold: a group of gradation values ​​corresponding to the white background, a group of gradation values ​​corresponding to the black ruled lines, and a group of gradation values ​​corresponding to the black characters.

[0135] When the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than the frequency threshold is 1, it is highly likely that the edge peripheral pixels contain only background and do not contain content. On the other hand, when the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than the frequency threshold is 2 or more and is small, it is highly likely that the edge peripheral pixels contain content such as characters, lines, or patterns. On the other hand, when the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than the frequency threshold is large, it is highly likely that the edge peripheral pixels contain content such as photos. The content determination unit 225 can determine with high accuracy whether or not content is contained in a corner region determined to have image loss by using the distribution of gradation values ​​of each edge peripheral pixel as a characteristic value of the edge peripheral pixel.

[0136] The characteristic value may include at least one of the above-mentioned characteristic values, and may include a plurality of characteristic values.

[0137] Next, the content determination unit 225 determines whether each characteristic value of the edge peripheral pixel is within a specified range indicating the absence of content (step S403). The specified range is set, for example, to a range equal to or less than a characteristic threshold value. Each characteristic threshold value is set, based on a prior experiment, to a value between each characteristic value when the edge peripheral pixel contains content and each characteristic value when the edge peripheral pixel does not contain content.

[0138] If all the characteristic values ​​of the edge surrounding pixels are within the specified range, the content determination unit 225 determines that no content exists around the medium edge in the target corner region (step S404), and ends the series of steps.

[0139] On the other hand, if the characteristic value of any of the edge surrounding pixels is outside the specified range, the content determination unit 225 determines that content exists around the medium edge in the corner region to be determined (step S405).

[0140] In this way, the content determination unit 225 determines whether or not content exists around the medium edge in the corner region determined to have image loss by the image loss determination unit 224. In particular, the content determination unit 225 determines whether or not content exists around the medium edge based on the difference, variance, periodicity, or distribution of gradation values ​​of multiple pixels corresponding to the medium edge in the corner region determined to have image loss by the image loss determination unit 224. This allows the content determination unit 225 to determine with high accuracy whether or not content exists around the medium edge.

[0141] Next, the content determination unit 225 identifies the type of content present around the edge of the medium (step S406), and ends the series of steps. If the variance of the gradation values ​​of the pixels around each edge is outside a specified range, the content determination unit 225 determines that the content is an image that cannot be corrected, and if the variance of the gradation values ​​of the pixels around each edge is within a specified range, the content determination unit 225 determines that the content is a correctable character, ruled line, or pattern. If the variance of the gradation values ​​of the pixels around each edge is outside a specified range, the content determination unit 225 may determine that the content is a correctable pattern. If the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than the frequency threshold is equal to or greater than the group number threshold, the content determination unit 225 may determine that the content is an image that cannot be corrected. On the other hand, if the number of groups of consecutive gradation values ​​whose frequency is equal to or greater than the frequency threshold is less than the group number threshold, the content determination unit 225 may determine that the content is a correctable character, ruled line, or pattern. The group number threshold is set to a value sufficiently greater than 2 (for example, 4, etc.).

[0142] 12 is a flowchart showing an example of the operation of the content correction process, which is executed in step S212 of the determination process shown in FIG.

[0143] First, the correction unit 226 sets the area inside (toward the center of the medium) of the original end edge pixels (before being corrected in step S211) in the corner area to be corrected where image loss occurs as the medium area. The correction unit 226 also sets the area outside (opposite the center of the medium) of the original end edge pixels and inside of the end edge pixels corrected in step S211 in the corner area to be corrected where image loss occurs as the correction target area (step S501). The medium area is set so as not to include the end edge pixels themselves, and the correction target area is set so as to include the end edge pixels themselves. The medium area may be set so as to include the end edge pixels themselves, and the correction target area may be set so as not to include the end edge pixels themselves.

[0144] Next, the correction unit 226 transmits a search request to the search server via the second communication device 203 to request a search for a similar image including pixels similar to the medium area in the corner area to be corrected (step S502). The search request includes an image including the medium area in the corner area to be corrected. The search server has a database including various images, and when receiving a search request, detects a similar image similar to the image included in the search request from among the images included in the database. The search server calculates the similarity between the image included in the search request and each image included in the database, and detects the image with the highest similarity as a similar image. The search server calculates the similarity of the NCC, reciprocal of SSD, or reciprocal of SAD, etc. of the two images. The search server transmits the detected similar image to the information processing device 200.

[0145] Next, the correction unit 226 receives the similar image from the search server via the second communication device 203 (step S503).

[0146] Next, the correction unit 226 forms (restores, corrects) the content in the correction target area in the corner area of ​​the correction target based on the received similar image, corrects the correction target area (step S504), and ends the series of steps. The correction unit 226 cuts out an area having the same size as the correction target area from the received similar image while shifting its position to generate a cut-out image. The correction unit 226 may also generate a cut-out image by cutting out an area having a different size from the correction target area and enlarging or reducing it to the same size as the correction target area. The correction unit 226 calculates the similarity between each cut-out image and the correction target area, and corrects the correction target area by replacing the correction target area with the cut-out image having the highest similarity. The correction unit 226 calculates the similarity between the NCC, the reciprocal of SSD, the reciprocal of SAD, or the like of the two images.

[0147] In this way, the correction unit 226 corrects the corner region based on a similar image including pixels similar to the peripheral pixels of the medium edge in the corner region to be corrected. That is, the correction unit 226 corrects a corner region in which the image loss determination unit 224 has determined that an image loss has occurred and the content determination unit 225 has determined that content exists around the medium edge, based on the peripheral pixels of the medium edge in the corner region. This allows the correction unit 226 to appropriately correct a corner region in which an image loss has occurred and content exists around the medium edge.

[0148] Alternatively, the information processing device 200 may store various images in the second storage device 210 in advance, and the correction unit 226 may detect a similar image from among the images stored in the second storage device 210 in steps S501 and S502.

[0149] As described above in detail, the image processing system 1 compares multiple corner regions of a medium in an input image to determine whether or not image loss occurs in each of the multiple corner regions. This makes it possible for the image processing system 1 to determine whether or not image loss occurs in each corner region for media having corners of various shapes. Therefore, the image processing system 1 can easily and accurately determine whether or not image loss occurs in a corner region of a medium in an image.

[0150] Furthermore, the image processing system 1 corrects corner areas where image loss occurs and where no content exists around the edge of the medium, but does not correct corner areas where content exists around the edge of the medium. This makes it possible for the image processing system 1 to prevent content loss caused by correcting corner areas where image loss occurs. Therefore, the image processing system 1 is able to appropriately correct corner areas of the medium in the image.

[0151] Furthermore, the image processing system 1 corrects corner areas where image loss occurs and content exists around the edge of the medium based on the surrounding pixels of the edge of the medium within the corner area, making it possible for the image processing system 1 to satisfactorily restore content in corner areas where image loss occurs.

[0152] In particular, the image processing system 1 can appropriately handle the case where skew of a medium occurs in the image reading device 100 and image loss occurs in the input image, and can tolerate to some extent the occurrence of skew of a medium in the image reading device 100. Therefore, the image reading device 100 can reduce the size of the imaging device 118, and can reduce the device size and device costs.

[0153] 13 is a flowchart showing an example of the operation of image loss determination processing according to another embodiment of the present invention, which is executed in place of the image loss determination processing shown in FIG.

[0154] First, the image loss determination unit 224 calculates the area of ​​the medium area in each corner region detected by the detection unit 223 (step S601). The image loss determination unit 224 specifies, in each corner region, the area inside (toward the center) the edge pixels as the medium area, and calculates the number of pixels in the medium area as the area of ​​the medium area in each corner region.

[0155] Next, the image loss determination unit 224 determines whether or not there is a combination of corner regions in which the difference in the area of ​​the medium areas is greater than an area threshold (step S602). The image loss determination unit 224 extracts all combinations (pairs) of two corner regions from all corner regions detected by the detection unit 223. The image loss determination unit 224 calculates the difference in the area of ​​the medium areas of the two corner regions included in each of the extracted combinations for all combinations. The area threshold is set to a value between the difference in the area of ​​the medium areas calculated for two corner regions where no image loss occurs and the difference in the area of ​​the medium areas calculated for a corner region where no image loss occurs and a corner region where image loss occurs, based on a prior experiment.

[0156] If there is no combination of corner regions in which the difference in area of ​​the medium regions is greater than the area threshold, the image loss determining unit 224 determines that no image loss has occurred in any corner region (step S603), and ends the series of steps.

[0157] On the other hand, if there is a combination of corner regions in which the difference in the area of ​​the medium region is greater than the area threshold, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions (step S604). As described above, the shapes of the four corners of the medium are highly likely to be the same, and if no image loss has occurred, the areas of the medium regions in all corner regions are highly likely to be similar. On the other hand, the possibility of image loss occurring in the same manner in all corner regions is low, and the possibility of image loss occurring such that the areas of the medium regions in each corner region are similar is low. Therefore, if the areas of the medium regions in all corner regions are similar, the image loss determination unit 224 determines that no image loss has occurred in any of the corner regions. On the other hand, if the area of ​​the medium region in any corner region is significantly different from the area of ​​the medium region in the other corner regions, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions.

[0158] In this way, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner regions by comparing the areas of the medium areas in the multiple corner regions detected by the detection unit 223. This allows the image loss determination unit 224 to determine whether or not image loss has occurred even if the shape of the corner of the medium being transported is not recognized in advance. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

[0159] Next, the image missing determination unit 224 identifies a corner region where image missing occurs based on the area of ​​the medium area in each corner region (step S605), and ends the series of steps. The image missing determination unit 224 determines that image missing does not occur in a corner region where the area of ​​the medium area is the largest, and in a corner region where the difference between the area of ​​the medium area and the largest area is equal to or less than the area threshold. On the other hand, the image missing determination unit 224 determines that image missing occurs in a corner region where the difference between the area of ​​the medium area and the largest area is greater than the area threshold. As described above, the same type of image missing is unlikely to occur in all corner regions, and image missing is unlikely to occur in each corner region such that the areas of the medium areas are similar. Therefore, it is highly likely that image missing does not occur in a corner region where the area of ​​the medium area is close to that of the other corner regions, and it is highly likely that image missing occurs in a corner region where the area of ​​the medium area is significantly different from that of the other corner regions. The image missing determination unit 224 can identify a corner region where image missing occurs with high accuracy by using the area of ​​the medium area in each corner region.

[0160] As described above in detail, the information processing device 200 is now able to easily and accurately determine whether or not an image is missing in a corner region of a medium in an image, even when comparing the area of ​​the medium region in multiple corner regions.

[0161] 14 is a flowchart showing an example of the operation of image loss determination processing according to still another embodiment of the present invention, which is executed in place of the image loss determination processing shown in FIG.

[0162] First, the image loss determination unit 224 calculates, for each of a plurality of corner regions, the distance between the intersection of two straight lines corresponding to two sides of the medium that form the corner corresponding to each corner region and a predetermined edge pixel as the corner distance (step S701). The image loss determination unit 224 calculates the corner distance between each of the upper left intersection, upper right intersection, lower left intersection, and lower right intersection detected in step S203 of the determination process and the predetermined edge pixel for each of the upper left corner region, upper right corner region, lower left corner region, and lower right corner region. The end edge pixel closest to each intersection in each corner region is set as the predetermined edge pixel. The end edge pixel closest to the straight line passing through each intersection and a point located diagonally opposite each intersection in each corner region may also be set as the predetermined edge pixel.

[0163] The image loss determination unit 224 calculates the Euclidean distance between each intersection and a specified edge pixel as the corner distance. The image loss determination unit 224 may calculate the Manhattan distance between each intersection and a specified edge pixel, that is, the sum of the distance in the horizontal direction and the distance in the vertical direction, as the corner distance. This allows the image loss determination unit 224 to calculate the corner distance efficiently and accurately. The image loss determination unit 224 may also calculate the Chebyshev distance between each intersection and a specified edge pixel, that is, the larger of the distance in the horizontal direction and the distance in the vertical direction, as the corner distance. This allows the image loss determination unit 224 to calculate the corner distance efficiently.

[0164] FIG. 15 is a schematic diagram for explaining the corner distance.

[0165] FIG. 15 shows the input image N shown in FIG. 8. In this example, the end edge pixel E5 closest to the upper left intersection V1 is extracted as the specified edge pixel, and the corner distance D1 from the upper left intersection V1 to the end edge pixel E5 is calculated. Also, the end edge pixel E6 closest to the upper right intersection V2 is extracted as the specified edge pixel, and the corner distance D2 from the upper right intersection V2 to the end edge pixel E6 is calculated. Also, the end edge pixel E7 closest to the lower left intersection V3 is extracted as the specified edge pixel, and the corner distance D3 from the lower left intersection V3 to the end edge pixel E7 is calculated. Also, the end edge pixel E8 closest to the lower right intersection V4 is extracted as the specified edge pixel, and the corner distance D4 from the lower right intersection V4 to the end edge pixel E8 is calculated.

[0166] 15, the corner distances D1, D3, and D4 calculated for the upper left intersection V1, the lower left intersection V3, and the lower right intersection V4, which have no bent corners, are substantially the same as one another. On the other hand, the corner distance D2 calculated for the upper right intersection V2, which has a bent corner, is significantly different from the corner distances D1, D3, and D4. Therefore, the image loss determination unit 224 can accurately determine whether or not image loss has occurred in each corner area based on the corner distances.

[0167] Next, the image loss determination unit 224 determines whether or not there is a combination of corner regions in which the difference in corner distance is greater than the distance threshold (step S702). The image loss determination unit 224 extracts all combinations (pairs) consisting of two corner regions from all corner regions detected by the detection unit 223. For all extracted combinations, the image loss determination unit 224 calculates the difference in corner distances calculated for the two corner regions included in each combination. The distance threshold is set to a value between the difference in corner distances calculated for two corner regions in which no image loss occurs and the difference in corner distances calculated for a corner region in which no image loss occurs and a corner region in which image loss occurs, based on a previous experiment. The image loss determination unit 224 may set a predetermined number of end edge pixels in each corner region in order of proximity to each intersection as the predetermined edge pixels, and calculate a statistical value (average value, median value, mode, maximum value, minimum value, etc.) of the distance between each intersection and each predetermined edge pixel as the corner distance.

[0168] If there is no combination of corner regions in which the difference in corner distance is greater than the distance threshold, image loss determining section 224 determines that no image loss has occurred in any corner region (step S703), and ends the series of steps.

[0169] On the other hand, if there is a combination of corner regions in which the difference in corner distance is greater than the distance threshold, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions (step S704). As described above, the shapes of the four corners of the medium are highly likely to be the same, and if no image loss has occurred, the corner distances are highly likely to be similar in all corner regions. On the other hand, the possibility of image loss occurring in the same manner in all corner regions is low, and the possibility of image loss occurring such that the corner distances in each corner region are similar is low. Therefore, if the corner distances in all corner regions are similar, the image loss determination unit 224 determines that image loss has not occurred in all corner regions. On the other hand, if the corner distance in any corner region is significantly different from the corner distance in the other corner regions, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions.

[0170] In this way, image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner regions by comparing the distances calculated for each of the multiple corner regions detected by detection unit 223. This allows image loss determination unit 224 to determine whether or not image loss has occurred even if it does not recognize in advance the shape of the corners of the medium being transported. Therefore, image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

[0171] Next, the image loss determination unit 224 identifies a corner region where image loss occurs based on the corner distance calculated for each of the multiple corner regions (step S705), and ends the series of steps. The image loss determination unit 224 determines that image loss does not occur in a corner region where the corner distance is the smallest and in a corner region where the difference between the corner distance and the smallest corner distance is equal to or less than the distance threshold. On the other hand, the image loss determination unit 224 determines that image loss occurs in a corner region where the difference between the corner distance and the smallest corner distance is greater than the distance threshold. As described above, it is unlikely that the same type of image loss occurs in all corner regions, and it is unlikely that image loss occurs in each corner region such that the corner distances are similar. Therefore, it is highly likely that image loss does not occur in a corner region where the corner distance is similar to the corner distance in other corner regions, and it is highly likely that image loss occurs in a corner region where the corner distance is significantly different from the corner distance in other corner regions. The image loss determination section 224 can specify the corner area where the image loss occurs with high accuracy by using the corner distance in each corner area.

[0172] As described above in detail, the information processing device 200 is now able to easily and accurately determine whether or not image loss has occurred in a corner region of a medium in an image, even when comparing corner distances in multiple corner regions.

[0173] 16 is a flowchart showing an example of the operation of image loss determination processing according to yet another embodiment of the present invention. The image loss determination processing shown in FIG.

[0174] First, the image missing determination unit 224 reads out an input image captured before the input image to be determined from the second storage device 210 as a second input image (step S801). For example, the image missing determination unit 224 reads out an input image captured immediately before the input image to be determined as the second input image.

[0175] When an input image captured before the input image to be determined is not acquired, the image loss determination unit 224 waits until an input image captured immediately after the input image to be determined is acquired, and uses the input image captured immediately after the input image to be determined as the second input image. Also, the image loss determination unit 224 may use the input image captured immediately after the input image to be determined as the second input image even when image loss occurs in a corner area detected from the input image captured immediately before the input image to be determined. The image loss determination unit 224 may use an input image captured a predetermined number of times before or after the input image to be determined as the second input image, instead of an input image captured immediately before or after the input image to be determined.

[0176] Also, in a booklet or the like made up of a plurality of pages, the shape of the cover may be different from the shapes of the other pages. The image missing determination unit 224 may not use the input image generated first in the medium reading process as the input image in which the cover is captured as the second input image. Also, in a booklet or the like made up of a plurality of pages, the shape of the inner cover may be different from the shapes of the other pages. For example, the image missing determination unit 224 may execute character recognition processing in the input image using a known OCR (Optical Character Recognition) technology. When the image missing determination unit 224 recognizes characters of a predetermined font size (for example, 20 points) or more in the center of the input image, it considers the input image to be the input image in which the inner cover is captured, and does not use it as the second input image. Also, in a booklet or the like made up of a plurality of pages, the shape of the odd-numbered pages may be different from the shape of the even-numbered pages. The image missing determination unit 224 may use the input image captured two pages before or two pages after the input image to be determined as the second input image. With these, the image missing determination unit 224 can determine whether or not image missing has occurred with a higher degree of accuracy.

[0177] Furthermore, the image loss determination unit 224 may use as the second input image only an input image in which a medium of approximately the same size as the medium included in the input image to be determined is captured. In this case, the image loss determination unit 224 detects edge pixels in each input image in the same manner as the process in step S202 in Fig. 7, and detects the number of pixels included in an area surrounded by the detected edge pixels as the size of the medium. The image loss determination unit 224 determines whether the sizes of the media included in each input image are approximately the same depending on whether the difference in the detected medium sizes is equal to or smaller than a predetermined threshold value.

[0178] Also, the image loss determination unit 224 may use as the second input image only an input image obtained by capturing a medium having a document layout (or ruled lines) substantially identical to the document layout (or ruled lines) in the medium included in the input image to be determined. In this case, the image loss determination unit 224 further detects edge pixels in the area surrounded by the edge pixels of each input image in the same manner as the process of step S202 in FIG. 7. The image loss determination unit 224 detects straight lines (line segments) from each edge pixel using the least squares method, Hough transform, or the like. The image loss determination unit 224 generates a binary image in which only pixels corresponding to the detected straight lines are regarded as valid pixels and other pixels are regarded as invalid pixels. The image loss determination unit 224 calculates the NCC, reciprocal of SSD, reciprocal of SAD, or the like of two binary images among the generated binary images as the similarity degree of the two input images corresponding to each binary image. The image loss determination unit 224 determines whether the document layouts of the media contained in each input image are substantially the same, depending on whether the calculated similarity is equal to or greater than a predetermined threshold value.

[0179] In addition, a mark such as a logo, a fixed character such as a header / footer, or a page number may be written at the edge of a document on which characters are printed, and it is highly likely that such a mark, fixed character, or page number is written at the same position on all pages. The image loss determination unit 224 may use as the second input image only an input image in which a medium having the same type of mark, fixed character, or page number as the mark, fixed character, or page number written on the medium included in the input image to be determined is captured. In this case, the image loss determination unit 224 calculates the NCC, the reciprocal of SSD, or the reciprocal of SAD, etc. of the upper end region or the lower end region (blank area) of the medium of the two input images as the degree of similarity. The image loss determination unit 224 determines whether the mark or fixed character written on the medium included in each input image is the same or not depending on whether the calculated degree of similarity is equal to or greater than a predetermined threshold. Alternatively, the image loss determination unit 224 performs character recognition processing in the upper end region or the lower end region of the medium of the two input images using a known OCR technique. The image loss determination unit 224 determines whether or not a page number is written on each medium based on whether numbers are detected at the same position in each area and whether the difference between the detected numbers matches the difference in the capture order of each input image.

[0180] Next, the image loss determination unit 224 calculates the degree of similarity between multiple corner regions (step S802). The image loss determination unit 224 calculates the degree of similarity between each corner region detected from the input image to be determined and a corner region at a corresponding position detected from the second input image. The image loss determination unit 224 calculates the reciprocal of the NCC, SSD, or SAD of the two corner regions as the degree of similarity.

[0181] Next, the image loss determination unit 224 determines whether or not there is a corner region whose similarity is greater than a similarity threshold (step S803). The similarity threshold is set to a value between the similarity calculated for two corner regions where no image loss occurs and the similarity calculated for a corner region where no image loss occurs and a corner region where image loss occurs, based on a prior experiment.

[0182] If there is no corner region whose similarity degree is smaller than the similarity threshold value, the image loss determining unit 224 determines that no image loss has occurred in any corner region (step S804), and ends the series of steps.

[0183] On the other hand, if there is a corner region whose similarity is smaller than the similarity threshold, the image missing determination unit 224 determines that image missing has occurred in that corner region (step S805) and ends the series of steps. Media placed together on the placement table 103 and transported, particularly media transported continuously, are likely to be the same type of media, and the shapes of corresponding corners are likely to be the same. On the other hand, in media placed together on the placement table 103 and transported, particularly media transported continuously, the possibility that corresponding corners are transported outside the imaging range of the imaging device 118 in the same way, and the possibility that corresponding corners are bent in the same way is low. Therefore, when corresponding corner regions in multiple input images are similar to each other, the image missing determination unit 224 determines that image missing has not occurred in that corner region. On the other hand, when corresponding corner regions in multiple input images are not similar to each other, the image missing determination unit 224 determines that image missing has occurred in that corner region.

[0184] In this way, the image loss determination unit 224 determines whether or not image loss has occurred in a corner area by comparing corresponding corner areas detected by the detection unit 223 in multiple input images. This allows the image loss determination unit 224 to determine whether or not image loss has occurred even if the shape of the corner of the medium being transported is not recognized in advance. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

[0185] The image missing determination unit 224 may determine whether or not image missing has occurred using parameters other than the degree of similarity between a plurality of corner regions. For example, the image missing determination unit 224 may calculate the area of ​​the medium area in each corner region in the same manner as in the process of step S601 in Fig. 13, and determine whether or not image missing has occurred based on whether or not the difference in the area of ​​the medium area in corresponding corner regions is greater than an area threshold. The image missing determination unit 224 may also calculate the corner distance in each corner region in the same manner as in the process of step S601 in Fig. 13, and determine whether or not image missing has occurred based on whether or not the difference in the corner distance in corresponding corner regions is greater than a distance threshold.

[0186] Furthermore, the image loss determining section 224 may determine whether or not image loss has occurred in only one, two, or three of the corner regions, rather than in all four corner regions.

[0187] As described above in detail, the information processing device 200 is now able to easily and accurately determine whether or not an image is missing in a corner area of ​​a medium in an image, even when comparing corresponding corner areas in multiple input images.

[0188] The image loss determination unit 224 may determine whether or not image loss occurs in a corner area in the input image by using another method. For example, the image loss determination unit 224 determines whether or not image loss occurs in a corner area in the input image based on whether or not the medium conveyed in the medium reading device 100 passes outside the imaging range of the imaging device 118. If the medium passes outside the imaging range of the imaging device 118, in step S106 of FIG. 6, the control unit 141 transmits image loss occurrence information indicating the occurrence of image loss to the information processing device 200 via the first communication device 122. The image loss occurrence information includes information indicating the second medium sensor 114 that detected the medium, the imaging start time of the medium, the detection time when the second medium sensor 114 detected the medium, and the transport speed of the medium. If the image loss determination unit 224 receives image loss occurrence information from the medium reading device 100 via the second communication device 203, it determines that image loss occurs in a corner area in the input image. The image loss determination unit 224 identifies the end in the main scanning direction where the image loss occurs in the input image based on information indicating the second medium sensor 114 that detected the medium. The image loss determination unit 224 also identifies the end in the sub-scanning direction where the image loss occurs in the input image based on the image capture start time of the medium, the detection time when the second medium sensor 114 detected the medium, and the medium transport speed.

[0189] In addition, the image loss determination unit 224 may determine that no image loss occurs in the corresponding corner area when the corner of the medium in the input image is an approximately right angle, and may determine that image loss occurs in the corresponding corner area when the corner of the medium in the input image is not an approximately right angle. In this case, the image loss determination unit 224 detects a left end straight line, a right end straight line, an upper end straight line, and a lower end straight line from the input image in the same manner as the process of step S203 in FIG. 7. The image loss determination unit 224 generates a corrected image in which the detected left end straight line, right end straight line, upper end straight line, and lower end straight line are extended to a position where they intersect with each other. The image loss determination unit 224 calculates the NCC, the reciprocal of SSD, the reciprocal of SAD, or the like between each corner area in the input image and each corresponding corner area in the corrected image as a similarity degree. The image loss determination unit 224 determines whether the corner of the medium in each corner area is an approximately right angle or not depending on whether the calculated similarity degree is equal to or greater than a predetermined threshold value.

[0190] Fig. 17 is a flowchart showing an example of the operation of a content determination process according to another embodiment. The content determination process shown in Fig. 17 is executed instead of or in addition to the content determination process shown in Fig. 10. When the content determination process shown in Fig. 10 is not executed, the process of step S210 in Fig. 7 may be omitted, and either the process of steps S211-S212 or the process of steps S213-S214 may be fixed and executed regardless of the type of content.

[0191] First, the content determination unit 225 detects the medium edge in the corner region where it has been determined that an image loss has occurred (step S901), in the same manner as in the process of step S401 in FIG.

[0192] Next, the content determination unit 225 executes character recognition processing in the input image (step S902). That is, the content determination unit 225 recognizes characters in an area including a corner area determined by the image loss determination unit 224 to have an image loss. The content determination unit 225 detects characters from the input image using a known OCR technique, and detects a plurality of characters having a predetermined positional relationship as a character string. The plurality of characters having a predetermined positional relationship are, for example, characters that overlap each other in the vertical direction and are adjacent to each other within a certain range in the horizontal direction and are connected together. The certain range is set to, for example, the number of pixels corresponding to a margin. The plurality of characters having a predetermined positional relationship includes characters that are connected with a line feed in between. That is, the plurality of characters having a predetermined positional relationship includes a character located at the right end in the horizontal direction and a character located adjacent to the character below the character in the vertical direction within a certain range and located at the left end in the horizontal direction. The characters having a predetermined positional relationship include a character located at the bottom in the vertical direction and a character located adjacent to the character located at the left of the character in the horizontal direction within a certain range and located at the top in the vertical direction. The content determination unit 225 stores the detected characters (character strings) and position information (coordinate information) of the characters in the input image in the second storage device 210.

[0193] Next, the content determination unit 225 determines whether or not a character has been detected around the edge of the medium in each corner region (step S903). For example, the content determination unit 225 determines that a character has been detected around the edge of the medium when the character is detected within a range of a predetermined number of pixels from the edge of the medium. The predetermined number of pixels is set to, for example, the number of pixels equivalent to the vertical or horizontal length of one or two characters.

[0194] If no characters are detected around the edge of the medium in any of the corner regions, the content determination unit 225 determines that no content exists around the edge of the medium in each corner region (step S904), and ends the series of steps.

[0195] On the other hand, when a character is detected around the edge of the medium in any of the corner regions, the content determination unit 225 determines whether or not the character string including the character detected around the edge of the medium is a natural language (step S905). The content determination unit 225 determines whether or not the character string is a natural language by using a known natural language analysis technique (morphological analysis, syntactic analysis, semantic analysis, contextual analysis). The content determination unit 225 may determine whether or not the character string is a natural language by using a learning model that has been trained to output whether or not the input character string is a natural language when the character string is input. The learning model is pre-trained by deep learning or the like using various character strings in natural languages ​​and / or character strings that are not natural languages ​​as teacher data. The content determination unit 225 inputs the detected character string into the learning model and determines whether or not the character string is a natural language based on the output value output from the learning model.

[0196] If the character string including the characters detected around the medium edge in each corner region is a natural language, the content determination unit 225 determines that no content exists around the medium edge in each corner region (step S904), and ends the series of steps. In other words, if the character string including the characters detected around the medium edge in each corner region is a natural language, the content determination unit 225 determines that the characters around the medium edge in each corner region are not lost due to image loss.

[0197] On the other hand, if the character string including the characters detected around the medium edge of any corner region is not a natural language, the content determination unit 225 determines that content exists around the medium edge in that corner region (step S906) and ends the series of steps. That is, if the character string including the characters detected around the medium edge of each corner region is not a natural language, the content determination unit 225 determines that the characters around the medium edge of each corner region have been lost due to image loss. In this case, the content determination unit 225 determines that the content is correctable characters.

[0198] Note that the process of step S905 may be omitted, and if characters are detected near the edge of the medium in any corner area, the content determination unit 225 may determine that content exists near the edge of the medium.

[0199] In this way, when the content determination unit 225 recognizes characters around the edge of the medium in a corner area where the image loss determination unit 224 has determined that image loss has occurred, the content determination unit 225 determines that content exists around the edge of the medium. In general, there is a high possibility that margins are set at the edges of a document on which characters are printed, and there is a high possibility that no characters exist around the edge of the medium. When characters exist around the edge of the medium, it is highly likely that at least the margin portion has been lost due to image loss, and that the portion lost due to image loss also contains characters. When the content determination unit 225 recognizes characters around the edge of the medium, it determines that content exists around the edge of the medium, thereby making it possible to determine with high accuracy that characters have been lost due to image loss.

[0200] Furthermore, if the characters recognized in the area including the corner area determined to have image loss are not a natural language, the content determination unit 225 determines that content exists around the edge of the medium in the corner area. In general, character strings written in a document are likely to be natural languages ​​that humans use on a daily basis for communication, and are unlikely to be meaningless random character strings. If the recognized character string is not a natural language, the content determination unit 225 determines that part of the character string has been lost due to image loss, thereby making it possible to determine with high accuracy that characters have been lost due to image loss.

[0201] As described above in detail, the image processing system 1 is capable of appropriately correcting corner areas of a medium in an image even when character recognition technology is used.

[0202] 18 is a flowchart showing an example of the operation of a content determination process according to still another embodiment of the present invention, which is executed in place of the content determination process shown in FIG.

[0203] First, the content determination unit 225 reads out the second input image from the second storage device 210 in the same manner as in step S801 of FIG. 16 (step S1001).

[0204] Next, the content determination unit 225 determines whether or not content exists in a corner region in the second input image corresponding to the corner region determined to have image loss in the input image to be determined (step S1002). The content determination unit 225 calculates characteristic values ​​of multiple pixels included in each corner region in the second input image in the same manner as the process of step S402 in FIG. 10. If the characteristic value is within a specified range indicating that no content exists, the content determination unit 225 determines that no content exists in the corresponding corner region in the second input image. On the other hand, if the characteristic value is not within the specified range, the content determination unit 225 determines that content exists in the corresponding corner region in the second input image.

[0205] If no content exists in the corner region in the second input image, the content determination unit 225 determines that no content exists around the edge of the medium in the corresponding corner region of the input image to be determined (step S1003), and ends the series of steps.

[0206] On the other hand, if content exists in the corner region in the second input image, the content determination unit 225 determines that content exists around the edge of the medium in the corresponding corner region of the input image to be determined (step S1004).

[0207] In this way, when content exists in a corner area in the second input image corresponding to the corner area determined by the image loss determination unit 224 as having image loss, the content determination unit 225 determines that content exists around the edge of the medium in the corner area determined by the image loss determination unit 224 as having image loss. As described above, marks, static characters, page numbers, etc. may be written on the edge of a document on which characters are printed, and it is highly likely that such marks, static characters, or page numbers are written in the same position on all pages. The content determination unit 225 can determine with high accuracy whether or not marks, static characters, or page numbers exist around the edge of the medium by using a second input image in which the medium is imaged before or after the input image to be determined.

[0208] Next, the content determination unit 225 identifies the type of content present around the medium edge (step S1005), and ends the series of steps. The content determination unit 225 identifies the type of content present around the medium edge in the same manner as the process of step S406 in Fig. 10. However, the content determination unit 225 identifies the type of content present around the medium edge of each corner region based on characteristic values ​​of multiple pixels included in each corner region in the second input image. The content determination unit 225 determines that the type of content present around the medium edge of each corner region in the input image to be determined is the same as the identified type of content.

[0209] In addition, when it is determined in the content determination process of this embodiment that image content exists near the edge of the medium, the content is highly likely to be a logo or the like that is common to all pages. Therefore, in step S210 of Fig. 7, the correction unit 226 may determine that the content can be corrected regardless of the type of content.

[0210] As described above in detail, the image processing system 1 is capable of appropriately correcting corner areas of a medium in an image even when the second input image is used.

[0211] 19 is a flowchart showing an example of the operation of the content correction process according to another embodiment of the present invention, which is executed instead of the content correction process shown in FIG.

[0212] First, the correction unit 226 sets a medium area and a correction target area in the corner area to be corrected where image loss occurs, in the same manner as in the process of step S501 in FIG. 12 (step S1101).

[0213] Next, the correction unit 226 forms (restores, corrects) a background in the correction target area, and corrects the correction target area (step S1102). When the content is determined to be a pattern (background pattern) in the content determination process, the correction unit 226 corrects the pattern (background pattern) in the correction target area. In addition, when the difference between the color value (RGB value) of each pixel in the medium area in the corner area of ​​the correction target and the color value of each pixel in the correction target area is equal to or greater than a predetermined threshold, the correction unit 226 may correct the background color in the correction target area. The correction unit 226 corrects the background in the correction target area by cutting out an image of the same size as the correction target area from the medium area and arranging (copying) it in the correction target area. This allows the correction unit 226 to properly restore the background damaged by image loss.

[0214] Next, the correction unit 226 forms (restores, corrects) characters in the correction target area and corrects the correction target area (step S1103). When the content is determined to be characters in the content determination process, the correction unit 226 corrects the characters in the correction target area. The correction unit 226 detects a plurality of characters having a predetermined positional relationship as a character string from the input image in the same manner as the process of step S902 in FIG. 10. The correction unit 226 converts a character string including the characters detected in the medium area into a character string in natural language using a known natural language analysis technique (morphological analysis, syntactic analysis, semantic analysis, context analysis). The correction unit 226 corrects the characters in the correction target area by arranging (writing) in the correction target area a character that is not included in the character string before conversion and is connected to a character that exists around the edge of the medium, among the characters included in the converted character string. The correction unit 226 may convert a character string including the characters detected in the medium area into a character string in natural language using a learning model that has been trained to output a character string obtained by converting an input character string into a natural language when a character string is input. The learning model is pre-trained using a set of various non-natural language character strings and their corresponding natural language character strings as training data by deep learning, etc. The correction unit 226 inputs a character string including characters detected within the medium area to the learning model, and acquires a natural language character string based on an output value output from the learning model.

[0215] In this way, the correction unit 226 recognizes characters in an area including a corner area determined by the image loss determination unit 224 as having an image loss, and corrects the corner area based on a natural language similar to the recognized characters. This allows the correction unit 226 to properly restore characters damaged by image loss.

[0216] Next, the correction unit 226 forms (restores, corrects) a ruled line in the correction target area, corrects the correction target area (step S1104), and ends the series of steps. When the content is determined to be a ruled line in the content determination process, the correction unit 226 corrects the ruled line in the correction target area. The correction unit 226 extracts edge pixels from the medium area in the corner area to be corrected in the same manner as the process of step S202 in FIG. 7, and detects straight lines from the extracted edge pixels in the same manner as the process of step S203. The correction unit 226 detects, among the detected straight lines, a straight line that extends to the original end edge pixel (before being corrected in step S211) as a ruled line cut at the end of the medium in the corner area. The correction unit 226 corrects the ruled line in the correction target area by further extending the straight line detected as the ruled line to a predetermined position in the correction target area. The predetermined position is set, for example, to a position inward (toward the center of the medium) by the number of pixels corresponding to the margin from the end edge pixel corrected in step S211.

[0217] When the correction unit 226 detects a line extending horizontally and a line extending vertically as a cut ruled line, the correction unit 226 may correct the ruled line by extending each line to a position where the lines intersect with each other. This makes it possible to properly restore the ruled line in the table format that has been damaged due to image loss.

[0218] In this manner, the correction unit 226 corrects the corner area by extending the ruled line cut at the medium edge in the corner area determined to have image loss by the image loss determination unit 224. In this way, the correction unit 226 can properly restore the ruled line damaged by the image loss.

[0219] As described above in detail, the image processing system 1 is capable of satisfactorily restoring content in corner areas where image loss occurs, even when correcting background patterns, characters and / or lines.

[0220] 20 is a flowchart showing an example of the operation of content correction processing according to still another embodiment of the present invention, which is executed instead of the content correction processing shown in FIG.

[0221] First, the correction unit 226 reads out a second input image from the second storage device 210 (step S1201) in the same manner as in the process of step S801 in Fig. 16. The correction unit 226 reads out a plurality of second input images.

[0222] Next, the correction unit 226 determines whether or not content exists in the corner region in each corresponding second input image (step S1202), in the same manner as in the process of step S1002 in Fig. 18. If it is determined that content does not exist in the corner region in each corresponding second input image, the correction unit 226 ends the series of steps without performing correction.

[0223] On the other hand, if it is determined that content exists in the corner regions in each corresponding second input image, the correction unit 226 determines whether the content is a mark, a static character, or a background pattern (step S1203). The correction unit 226 calculates the similarity of the corner regions in each corresponding second input image in the same manner as the process of step S802 in Fig. 16. If the similarity of each corner region is equal to or greater than a threshold, the correction unit 226 determines that each corner region contains the same mark, static character, or background pattern, and if the similarity of each corner region is less than the threshold, the correction unit 226 determines that each corner region does not contain the same mark, static character, or background pattern.

[0224] If it is determined that the content is a mark, a static character, or a background pattern, the correction unit 226 forms (restores, corrects) the mark, static character, or background pattern in a corner area in the input image to be corrected, corrects the corner area (step S1204), and ends the series of steps. The correction unit 226 corrects the mark, static character, or background pattern by arranging (copying) the corresponding corner area in the second input image in the corner area in the input image to be corrected. In this way, the correction unit 226 can properly restore the mark, static character, or background pattern that has been damaged due to image loss.

[0225] In addition, even if there is no content in a corner area in the input image to be corrected, the correction unit 226 may correct the background color by placing (copying) the corresponding corner area in the second input image in the corner area in the input image to be corrected.

[0226] On the other hand, if it is determined that the content is not a mark, the correction unit 226 determines whether or not the content is a page number (step S1205). The correction unit 226 executes character recognition processing in the corner regions of the corresponding second input images in the same manner as the processing in step S902 of FIG. 17. The correction unit 226 determines that the content is a page number if a number is detected in the corner region of each second input image and the difference between the numbers detected in the corner region of each second input image matches the difference between the imaging orders of each second input image. On the other hand, the correction unit 226 determines that the content is not a page number if a number is not detected in the corner region of any of the second input images or if the difference between the numbers detected in the corner region of each second input image does not match the difference between the imaging orders of each second input image. If it is determined that the content is not a page number, the correction unit 226 does not execute correction and ends the series of steps.

[0227] On the other hand, if it is determined that the content is a page number, the correction unit 226 forms (restores, corrects) the page number in the corner region of the input image to be corrected, corrects the corner region (step S1206), and ends the series of steps. The correction unit 226 identifies the page number of the input image to be corrected from the relationship of the numbers detected in the corner regions of each second input image, and corrects the page number by arranging (writing) the identified page number in the corner region of the input image to be corrected. This allows the correction unit 226 to properly restore the page number that has been damaged due to image loss.

[0228] In this way, the correction unit 226 corrects a corner region in a specific input image among the multiple input images, where the image loss determination unit 224 has determined that an image loss has occurred and where the content determination unit 225 has determined that content exists around the medium edge, based on the corresponding corner region in the other input image. This allows the correction unit 226 to properly restore the content lost due to the image loss.

[0229] As described above in detail, the image processing system 1 is able to satisfactorily restore content in corner areas where image loss occurs, even when the second input image is used.

[0230] 21(A) is a diagram showing a schematic configuration of a first processing circuit 340 in an image reading device according to another embodiment. The first processing circuit 340 is used in place of the first processing circuit 140 of the image reading device 100, and executes image reading processing and the like in place of the first processing circuit 140. The first processing circuit 340 has a control circuit 341, a transmission circuit 342, and the like. Each of these components may be formed by an independent integrated circuit, microprocessor, firmware, and the like.

[0231] The control circuit 341 is an example of a control unit, and has the same functions as the control unit 141. The control circuit 341 receives an operation signal from the first operation device 105 or the first communication device 122, a first medium signal from the first medium sensor 111, a second medium signal from the second medium sensor 114, and a third medium signal from the third medium sensor 117, and controls the motor 121 and the imaging device 118 based on each of the received signals.

[0232] The transmission circuit 342 is an example of a transmission unit, and has the same function as the transmission unit 142. The transmission circuit 342 acquires an input image from the imaging device 118, and outputs it to the first communication device 122.

[0233] As described above in detail, even when the image reading device uses the first processing circuit 340, the image processing system 1 can easily and accurately determine whether or not image loss occurs in a corner area of ​​the medium in the image. Also, the image processing system 1 can appropriately correct the corner area of ​​the medium in the image. Also, the image processing system 1 can satisfactorily restore the content in the corner area where image loss occurs.

[0234] 21B is a diagram showing a schematic configuration of a second processing circuit 420 in an information processing device according to another embodiment. The second processing circuit 420 is used in place of the second processing circuit 420 of the information processing device 200, and executes a determination process and the like in place of the second processing circuit 220. The second processing circuit 420 has an acquisition circuit 421, an extraction circuit 422, a detection circuit 423, an image loss determination circuit 424, a content determination circuit 425, a correction circuit 426, a notification circuit 427, an output control circuit 428, and the like. Each of these units may be composed of an independent integrated circuit, microprocessor, firmware, and the like.

[0235] The acquisition circuit 421 is an example of an acquisition unit, and has the same function as the acquisition unit 221. The acquisition circuit 421 receives an input image from the second communication device 203, and stores it in the second storage device 210.

[0236] The extraction circuit 422 is an example of an extraction unit, and has the same function as the extraction unit 222. The extraction circuit 422 reads the input image from the second storage device 210, extracts edge pixels from the input image, and stores the extraction result in the second storage device 210.

[0237] The detection circuit 423 is an example of a detection unit, and has the same function as the detection unit 223. The detection circuit 423 reads out the extraction results of the input image and the edge image from the second storage device 210, detects corner regions based on the read information, and stores the detection results in the second storage device 210.

[0238] The image loss determination circuit 424 is an example of an image loss determination section, and has the same function as the image loss determination section 224. The image loss determination circuit 424 reads out the input image and the detection results of the corner region from the second storage device 210, determines whether or not image loss has occurred in the corner region based on the read information, and stores the determination result in the second storage device 210.

[0239] The content judgment circuit 425 is an example of a content judgment unit, and has the same function as the content judgment unit 225. The content judgment circuit 425 reads out the input image and the detection result of the corner region from the second storage device 210, judges whether or not content exists around the medium edge in the corner region based on each piece of read information, and stores the judgment result in the second storage device 210.

[0240] The correction circuit 426 is an example of a correction unit, and has the same function as the correction unit 226. The correction circuit 426 reads out the input image and the detection result of the corner area from the second storage device 210, corrects the corner area based on each piece of read information, and stores the corrected input image in the second storage device 210.

[0241] The notification circuit 427 is an example of a notification unit, and has the same function as the notification unit 227. The notification circuit 427 reads out the determination results of the image loss and the content from the second storage device 210, and outputs to the second display device 202 or the second communication device 203 that an image loss has occurred.

[0242] The output control circuit 428 is an example of an output control unit, and has the same function as the output control unit 228. The output control circuit 428 reads out the input image or the determination result of image loss and content from the second storage device 210, and outputs information related to the determination result to the second display device 202 or the second communication device 203.

[0243] As described above in detail, even when the information processing device uses the second processing circuit 420, the image processing system 1 can easily and accurately determine whether or not image loss occurs in a corner area of ​​the medium in the image. Also, the image processing system 1 can appropriately correct the corner area of ​​the medium in the image. Also, the image processing system 1 can satisfactorily restore the content in the corner area where image loss occurs.

[0244] Although the preferred embodiments have been described above, the embodiments are not limited thereto. For example, the image reading device may have a so-called U-turn path, and the medium placed on the loading table may be fed and conveyed from the top to the bottom, and then discharged onto the discharge table. In this case, the feed roller is disposed above the separation roller and opposite to the separation roller.

[0245] 7 may be executed by the image reading device 100 instead of the information processing device 200. In that case, the first processing circuit 140 of the image reading device 100 has each unit of the second processing circuit 220 of the information processing device 200, and executes the determination process. The acquisition unit of the first processing circuit 140 acquires the input image and various information by reading them from the first storage device 130. Also, the image processing system 1 may have a plurality of information processing devices 200 instead of one, and each information processing device 200 may cooperate to share each process in the overall process and the second process. Also, in that case, a plurality of information processing devices 200 may be distributed and arranged on a network so that an image processing service can be provided in the form of cloud computing.

[0246] Furthermore, in steps S209 and / or S211 in Fig. 7, the correction unit 226 may correct a corner region determined to have image loss by using a corner region in the second input image corresponding to the corner region. In this case, the correction unit 226 reads out the second input image in the same manner as in the process of step S801 in Fig. 16, and corrects the corner region in which image loss has occurred by arranging (copying) the corner region in the second input image corresponding to the corner region in which image loss has occurred. In this case, the correction unit 226 may correct only the missing portion in the corner region in which image loss has occurred. (Appendix 1) an acquisition unit that acquires an input image obtained by capturing an image of a medium; a detection unit that detects a corner region that is estimated to include a corner of a medium within the input image; an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit; a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred; a correction unit that corrects corner areas in which it has been determined by the image loss determination unit that image loss has occurred, where the content determination unit has determined that no content exists around an edge of a medium, and does not correct corner areas in which it has been determined by the content determination unit that content exists around an edge of the medium; an output unit that outputs the input image corrected by the correction unit; 13. An image processing device comprising: (Appendix 2) 2. The image processing device according to claim 1, further comprising a notification unit that notifies a user that an image is missing when there is a corner area in which the content determination unit determines that content exists around an edge of a medium. (Appendix 3) The image processing device according to claim 1 or 2, wherein the content determination unit determines whether or not content is present near the edge of a medium based on a difference, variance, periodicity, or distribution of gradation values ​​of multiple pixels corresponding to the edge of a medium in a corner region where the image loss determination unit has determined that an image loss has occurred. (Appendix 4) The image processing device according to claim 1 or 2, wherein the content determination unit determines that content is present around the edge of the medium when characters are recognized around the edge of the medium in a corner area in which the image loss determination unit has determined that an image loss has occurred. (Appendix 5) The image processing device described in Appendix 1 or 2, wherein the content determination unit recognizes characters within an area including a corner area in which the image loss determination unit has determined that an image loss has occurred, and if the recognized characters are not a natural language, determines that content is present around the edge of the medium within the corner area. (Appendix 6) The acquisition unit further acquires a second input image in which a medium is captured before or after the input image, The image processing device according to claim 1 or 2, wherein when content is present in a corner area in the second input image corresponding to the corner area in which the image loss determination unit has determined that an image loss has occurred, the content determination unit determines that content is present around the edge of a medium in the corner area in which the image loss determination unit has determined that an image loss has occurred. (Appendix 7) The image processing device An input image of a medium is obtained; Detecting a corner region within the input image that is estimated to include a corner of a medium; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; Among the corner areas determined to have image loss, the corner areas determined to have no content around the edge of the medium are corrected, and the corner areas determined to have content around the edge of the medium are not corrected; outputting the corrected input image; 13. An image processing method comprising: (Appendix 8) A control program for an image processing device, An input image of a medium is obtained; Detecting a corner region within the input image that is estimated to include a corner of a medium; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; Among the corner areas determined to have image loss, the corner areas determined to have no content around the edge of the medium are corrected, and the corner areas determined to have content around the edge of the medium are not corrected; outputting the corrected input image; A control program for causing the image processing device to execute the above steps. (Appendix 9) an acquisition unit that acquires an input image obtained by capturing an image of a medium; a detection unit that detects four corner regions in the input image, each of which is estimated to include a corner of a medium; an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit; a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred; a correction unit that corrects a corner region in which the image loss determination unit has determined that an image loss has occurred and in which the content determination unit has determined that content exists around the edge of the medium, based on peripheral pixels of the edge of the medium within the corner region; an output unit that outputs the input image corrected by the correction unit; 13. An image processing device comprising: (Appendix 10) Further comprising a communication unit, 10. The image processing device according to claim 9, wherein the correction unit receives an image including pixels similar to the surrounding pixels via the communication unit, and corrects the corner region based on the received image. (Appendix 11) The image processing device according to claim 9 or 10, wherein the correction unit recognizes characters within an area including a corner area in which the image loss determination unit has determined that an image loss has occurred, and corrects the corner area based on a natural language similar to the recognized character. (Appendix 12) 11. The image processing device according to claim 9, wherein the correction unit corrects the corner area by extending a ruled line cut at an edge of a medium in the corner area where the image loss determination unit has determined that an image loss has occurred. (Appendix 13) an acquisition unit that acquires a plurality of input images obtained by capturing an image of a medium; a detection unit that detects a corner region that is estimated to include a corner of a medium in each of the plurality of input images; an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit; a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred; a correction unit that corrects a corner area in a specific input image among the plurality of input images, the corner area being determined by the image loss determination unit as having an image loss and in which the content determination unit has determined that content exists around an edge of a medium, based on a corresponding corner area in another input image; an output unit that outputs the input image corrected by the correction unit; 13. An image processing device comprising: (Appendix 14) The image processing device An input image of a medium is obtained; Detecting four corner regions in the input image, each of which is estimated to include a corner of a medium; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; A corner region where it is determined that an image is missing and that content exists around the edge of the medium is corrected based on the surrounding pixels of the edge of the medium within the corner region; outputting the corrected input image; 13. An image processing method comprising: (Appendix 15) The image processing device A plurality of input images of a medium are obtained; Detecting a corner region that is estimated to include a corner of a medium in each of the plurality of input images; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; correcting a corner area in which it is determined that an image is missing in a specific input image among the plurality of input images and that content is present around an edge of the medium, based on a corresponding corner area in another input image; outputting the corrected input image; 13. An image processing method comprising: (Appendix 16) A control program for an image processing device, An input image of a medium is obtained; Detecting four corner regions in the input image, each of which is estimated to include a corner of a medium; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; A corner region where it is determined that an image is missing and that content exists around the edge of the medium is corrected based on the surrounding pixels of the edge of the medium within the corner region; outputting the corrected input image; A control program for causing the image processing device to execute the above steps. (Appendix 17) A control program for an image processing device, A plurality of input images of a medium are obtained; Detecting a corner region that is estimated to include a corner of a medium in each of the plurality of input images; determining whether or not an image defect occurs in the detected corner area; determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred; correcting a corner area in which it is determined that an image is missing in a specific input image among the plurality of input images and that content is present around an edge of the medium, based on a corresponding corner area in another input image; outputting the corrected input image; A control program for causing the image processing device to execute the above steps. [Explanation of symbols]

[0247] 1 image processing system, 100 image reading device, 200 information processing device, 221 acquisition unit, 222 extraction unit, 223 detection unit, 224 image loss determination unit, 225 content determination unit, 226 correction unit, 227 notification unit, 228 output control unit

Claims

1. an acquisition unit that acquires an input image obtained by capturing an image of a medium; a detection unit that detects a plurality of corner regions including corners of a medium in the input image; a determination unit that determines whether or not image loss occurs in each of the plurality of corner areas by comparing the plurality of corner areas detected by the detection unit; an output unit that outputs information related to the determination result by the determination unit; 1. An image processing device comprising:

2. An acquisition unit that acquires an input image obtained by capturing an image of a medium; a detection unit that detects a plurality of corner regions including corners of a medium in the input image; a correction unit that corrects a corner region where image loss occurs based on a comparison of the plurality of corner regions detected by the detection unit; 1. An image processing device comprising:

3. 2. The image processing device according to claim 1, wherein the determination unit classifies the plurality of corner regions into groups based on the degree of similarity of the plurality of corner regions, and determines whether or not image loss occurs in the plurality of corner regions based on the number of groups into which the plurality of corner regions are classified.

4. The image processing device according to claim 3 , wherein the determining unit identifies a corner region in which image loss occurs based on the number of corner regions belonging to each of the classified groups.

5. The image processing device according to claim 1 , wherein the determining unit determines whether or not an image defect occurs in each of the plurality of corner regions by comparing areas of the medium regions in the plurality of corner regions.

6. further comprising an extraction unit that extracts edge pixels from the input image; The determination unit calculating a distance between a predetermined edge pixel and an intersection of two straight lines detected from the edge pixels and substantially perpendicular to each other, for each of the plurality of corner regions; The image processing device according to claim 1 , wherein the distances calculated for each of the plurality of corner regions are compared to determine whether or not image loss occurs in each of the plurality of corner regions.

7. The acquisition unit acquires a plurality of input images, the detection unit detects the corner region in each of the plurality of input images; The image processing device according to claim 1 , wherein the determining unit determines whether or not an image defect occurs in the corner area by comparing the corner areas detected in each of the plurality of input images.

8. The acquisition unit acquires a plurality of input images, the detection unit detects the corner region in each of the plurality of input images; The image processing device according to claim 2 , wherein the correction unit corrects a corner area where image loss occurs based on a comparison of the corner areas detected in each of the plurality of input images.

9. A control program for an image processing device, An input image of the medium is acquired, Detecting a plurality of corner regions including corners of a medium in the input image; By comparing the detected corner areas, it is determined whether or not image loss occurs in each of the corner areas; outputting information regarding the result of the determination; a control program for causing the image processing device to execute the above steps;

10. A control program for an image processing device, An input image of the medium is acquired, Detecting a plurality of corner regions including corners of a medium in the input image; correcting the corner area where image loss occurs based on a comparison of the detected plurality of corner areas; a control program for causing the image processing device to execute the above steps;

Images