Computer program, recording medium, and image processing device

Abstract

To provide a computer program, a recording medium, and an image processing device which appropriately detect an area representing an object of a line from an image.SOLUTION: A computer program includes: a step S220 of detecting an object area representing an object by executing first labeling processing of target image data; steps S240 and S260 of determining whether or not the object area satisfies a line inclusive condition; and steps S250 and S270 of detecting a first direction line candidate area, determining whether or not the first direction line candidate area satisfies a first direction target condition, and changing respective color values of a plurality of pixels included in the first direction line candidate area determined to satisfy the first direction target condition to a color value of a background, by executing second labeling processing using a plurality of pixels included in a partial area satisfying a first direction candidate condition among pixels of the object area determined to satisfy the line inclusive condition.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present specification relates to techniques for processing image data. [Background technology]

[0002] Conventionally, reading devices such as scanners have been used to read objects such as manuscripts. In addition, in order to improve the accuracy of character recognition of the read image, a process of removing ruled lines within the image may be performed. For example, Patent Document 1 proposes a technique for removing ruled lines by detecting continuous parts of black pixels that are continuous in the horizontal direction and removing continuous parts having a length equal to or greater than a predetermined threshold value from the image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-322510 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, a scanned image may represent various objects, such as a table, a graph, characters, etc., in addition to ruled lines. There is room for improvement in appropriately detecting an area representing a line object from such an image.

[0005] This specification discloses techniques for detecting regions representing line objects from an image. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] A computer program for a computer that processes image data, comprising: a first type detection function that detects J object regions (J is an integer equal to or greater than 1) that represent objects by executing a first labeling process on target image data that represents a target image, which is an image of a document optically read by a reading device; a first judgment function that judges whether the object regions satisfy a line inclusion condition that indicates that the object regions represent objects that include lines; and a first judgment function that judges whether the object regions satisfy a line inclusion condition that indicates that the object regions include a line extending in a first direction, which is either the horizontal direction or the vertical direction, among a plurality of pixels contained in K line candidate regions (K is an integer equal to or greater than 1 and equal to or less than J), which are object regions that have been judged to satisfy the line inclusion condition. a first direction candidate detection function that detects L (L is an integer equal to or greater than 1) first direction line candidate regions that are candidates for object regions indicating a line extending in the first direction by performing a second labeling process using a plurality of pixels included in a partial region that satisfies a first direction candidate condition; a first direction judgment function that judges whether the first direction line candidate regions satisfy a first direction target condition that indicates that the first direction line candidate regions represent a line object extending in the first direction; and a first direction change function that changes the color values ​​of each of a plurality of pixels included in a first direction line candidate region that is determined to satisfy the first direction target condition, among a plurality of pixels of the target image, to the color value of the background.

[0008] According to this configuration, J object regions are detected by performing a first labeling process on the target image data, and L first direction line candidate regions are detected by performing a second labeling process using a plurality of pixels included in partial regions that satisfy the first direction candidate condition among a plurality of pixels included in K line candidate regions, which are object regions determined to satisfy the line inclusion condition, and the color values ​​of each of the plurality of pixels included in the first direction line candidate regions determined to satisfy the first direction target condition are changed to the color value of the background, so that areas representing line objects can be detected more appropriately than when areas representing line objects are detected directly from the target image.

[0009] The technology disclosed in this specification can be realized in various forms, for example, in the form of an image processing method and an image processing device, a computer program for realizing the functions of those methods or devices, a recording medium on which that computer program is recorded (e.g., a non-temporary recording medium), etc. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a system 1000. [Diagram 2] 10 is a flowchart illustrating an example of a read image process. [Diagram 3] (A) shows an example of a scanned image, and (B) shows an example of a label image. [Figure 4] 13 is a flowchart illustrating an example of a line removal process. [Diagram 5] 13 is a flowchart illustrating an example of a process for determining a line candidate region. [Figure 6] 13A to 13D are diagrams illustrating examples of object regions. [Figure 7] 1A is an explanatory diagram showing an example of a line candidate region image, FIG. 1B is an example of a label image, and FIG. 1C is an example of a label image. [Figure 8] 13 is a flowchart illustrating an example of a horizontal line removal process. [Figure 9] 13 is a flowchart illustrating an example of a horizontal line removal process. [Figure 10] 13A to 13D are explanatory diagrams showing an example of processing of a focus line candidate region. [Figure 11] 13 is a flowchart illustrating an example of a vertical line removal process. [Figure 12] 13 is a flowchart illustrating an example of a vertical line removal process. [Figure 13] 1 shows an example of a line removal image. [Figure 14] 13 is a flowchart illustrating an example of a condition determination process. [Figure 15] 11 is a flow chart illustrating another embodiment of a line removal process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First Example: A1. System 1000 consists of: 1 is a block diagram showing the configuration of a system 1000. The system 1000 includes a multifunction device 200 and a user terminal 500. The multifunction device 200 and the user terminal 500 are connected to a network NT. The network NT is a so-called local area network. The multifunction device 200 and the user terminal 500 are capable of communicating with each other via the network NT. The network NT may include the so-called Internet.

[0012] The multifunction device 200 includes a control unit 299, a display unit 240, an operation unit 250, a communication interface 270, a reading execution unit 280, and a printing execution unit 290. The control unit 299 includes a processor 210 and a storage device 215. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a touch panel, a button, a lever, etc., which are arranged on top of the display unit 240. The user can input various instructions to the multifunction device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices (for example, a USB interface, a wired LAN interface, or an IEEE802.11 wireless interface). In this embodiment, the communication interface 270 is connected to a network NT.

[0014] The reading execution unit 280 is a reading device configured to optically read an object such as a document using a photoelectric conversion element such as a CCD or a CMOS. The reading execution unit 280 generates read image data (e.g., RGB bitmap data) that represents the read image (referred to as a "read image").

[0015] The print execution unit 290 is a device that prints an image on paper (an example of a print medium) using a printing material in a predetermined manner (for example, a laser method or an inkjet method). The print execution unit 290 executes printing according to the control of the processor 210.

[0016] The processor 210 is a device configured to perform data processing, and may be, for example, a CPU. The volatile storage device 220 may be, for example, a DRAM, and the non-volatile storage device 230 may be, for example, a flash memory.

[0017] The non-volatile storage device 230 stores a program 232. The processor 210 executes the program 232 to realize various functions for controlling the multifunction device 200. The processor 210 temporarily stores various intermediate data used in the execution of the program 232 in a storage device (e.g., either the volatile storage device 220 or the non-volatile storage device 230). In this embodiment, the program 232 is stored in advance in the non-volatile storage device 230 as firmware by the manufacturer of the multifunction device 200.

[0018] The user terminal 500 is a device that executes image processing using the multifunction device 200 (e.g., a smartphone, a tablet computer, etc.). The user terminal 500 includes a processor 510, a storage device 515, a display unit 540, an operation unit 550, and a communication interface 570. These elements are connected to each other via a bus. The storage device 515 includes a volatile storage device 520 and a non-volatile storage device 530.

[0019] The display unit 540 is a device that displays images, and is, for example, a liquid crystal display. The operation unit 550 is a device that receives operations by a user, and includes, for example, a touch panel that is overlaid on the display unit 540. The user can input various instructions to the user terminal 500 by operating the operation unit 550. The communication interface 570 is an interface for communicating with other devices (for example, a wired LAN interface, an IEEE802.11 wireless interface). In this embodiment, the communication interface 570 is connected to a network NT.

[0020] The processor 510 is a device configured to process data, such as a CPU. The volatile storage device 520 is a DRAM, and the non-volatile storage device 530 is a flash memory.

[0021] The non-volatile storage device 530 stores a program 531. The processor 510 executes the program 531 to execute a function as an image processing application. The processor 510 temporarily stores various intermediate data used in the execution of the program 531 in a storage device (for example, the volatile storage device 520 or the non-volatile storage device 530). Details of the processing executed by the processor 510 will be described later. In this embodiment, the program 531 of the image processing application is provided by a server (not shown). Alternatively, the program 531 may be provided by a portable recording medium such as a memory card.

[0022] A2. Image reading processing: 2 is a flowchart showing an example of read image processing. In this embodiment, the user terminal 500 causes the multifunction device 200 to read an object such as a document, and acquires read image data from the multifunction device 200. The user terminal 500 then executes line removal processing and character recognition processing of the read image. The processing by the user terminal 500 is performed by the processor 510 executing the program 531.

[0023] In S110, the user places an original in a portion of the reading execution unit 280 of the multifunction device 200 (FIG. 1) where the object to be read is to be placed (for example, the original is placed on the platen of the reading execution unit 280). The user then operates the operation unit 550 of the user terminal 500 to input an instruction to start processing. The processor 510 of the user terminal 500 supplies a reading instruction to the multifunction device 200. In response to the reading instruction, the processor 210 of the multifunction device 200 causes the reading execution unit 280 to read the original. The reading execution unit 280 reads the original and generates read image data representing an image of the read original.

[0024] In S120, the processor 210 of the multifunction device 200 acquires the read image data from the reading execution unit 280, and transmits the acquired read image data to the user terminal 500. The processor 510 of the user terminal 500 acquires the read image data from the multifunction device 200.

[0025] 3A shows an example of a read image represented by the read image data. The read image I1 is a rectangular image having two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy perpendicular to the horizontal direction Dx. The read image I1 is represented by the color values ​​(in this embodiment, the component values ​​of R (red), G (green), and B (blue)) of a plurality of pixels arranged in a matrix along the horizontal direction Dx and the vertical direction Dy.

[0026] The scanned image I1 in FIG. 3(A) represents a plurality of objects including three ruled lines OL1-OL3, a title Otx1, a group of character strings Otx2, a table Otb, and a bar graph Obg. The table Otb includes a line OL4 that separates cells, and character strings arranged in each cell. The bar graph Obg includes a line OL5 that indicates an axis and a bar, and character strings of scales. Lines arranged near characters may reduce the accuracy of character recognition. In S130 (FIG. 2), the processor 510 of the user terminal 500 executes a line removal process that removes lines from the scanned image I1 (details will be described later). In S140, the processor 510 executes a character recognition process using the remaining image from which the lines have been removed. The character recognition process may be various processes that recognize characters in an image. For example, an OCR engine called "Tesseract" by Google Inc. may be adopted. The processor 510 stores character data indicating the characters recognized by the character recognition process in the storage device 515 (for example, the non-volatile storage device 530). Then, the processor 510 ends the process of FIG.

[0027] 4 is a flowchart showing an example of the line removal process. In S210, the processor 510 of the user terminal 500 determines the direction of the line to be removed (also called the removal target direction) according to a user instruction. The user can select the removal target direction from "horizontal direction", "vertical direction", and "both" by operating the operation unit 550.

[0028] In S220, the processor 510 generates labeling data indicating an object region by executing a labeling process on the read image data. In this embodiment, the processor 510 classifies a plurality of pixels of the read image I1 into object pixels indicating an object and background pixels indicating a background by executing a binarization process on the read image data. The processor 510 then assigns one identifier to one continuous region consisting of one or more continuous object pixels. The processor 510 then assigns different identifiers to a plurality of continuous regions that are separated from each other. The label data indicates the identifier of each pixel. Hereinafter, the continuous region is also referred to as an object region. The processor 510 detects the object region by such a labeling process.

[0029] Various processes can be adopted as the binarization process. In this embodiment, the processor 510 calculates a luminance value from the color value (here, RGB value) of each pixel, classifies pixels having a luminance value less than a threshold value as object pixels, and classifies pixels having a luminance value equal to or greater than the threshold value as background pixels. The threshold value is experimentally determined in advance so as to appropriately classify the background and objects in the read image. Alternatively, the processor 510 may determine the threshold value by analyzing the captured image data. Also, various processes can be adopted as the labeling process of the binary image. In this embodiment, so-called 8-neighborhood labeling is performed. Alternatively, so-called 4-neighborhood labeling may be performed.

[0030] FIG. 3(B) shows an example of a label image obtained by the labeling process of the scanned image I1 (FIG. 3(A)). This label image I2 shows object areas OA1-OA5 and a plurality of object areas OA showing characters. The first object area OA1 shows a plurality of characters and a ruled line OL1 of the title Otx1. In this way, when a line is arranged near the characters, the characters and the line can be included in one object area. The object areas OA2 and OA3 show the ruled lines OL2 and OL3, respectively. The fourth object area OA4 shows the line OL4 of the table Otb. The fifth object area OA5 shows the line OL5 of the bar graph Obg. The plurality of object areas OA show each of the plurality of characters included in the character string group Otx2, the table Otb, and the bar graph Obg. In this way, the processor 510 obtains object areas showing objects included in the scanned image I1 by the labeling process.

[0031] In S230 (FIG. 4), the processor 510 executes a line candidate region determination process. The line candidate region determination process is a process for determining whether or not an object region is a line candidate object region that is a candidate for a region indicating a line (a line candidate object region is also simply called a line candidate region).

[0032] 5 is a flowchart showing an example of a process for determining a line candidate region. In S310, the processor 510 selects an unprocessed object region from the object regions represented by the labeling data as an object region of interest.

[0033] In S320, the processor 510 acquires the height and width of the target object region. Then, the processor 510 determines whether or not one or both of the height and the width are equal to or greater than the length threshold value ThL.

[0034] Figures 6(A) to 6(D) are explanatory diagrams showing examples of object areas, each showing object areas OA2, OA3, OA6, and OA5, respectively.

[0035] The hatched second object area OA2 in FIG. 6(A) shows a ruled line OL2 approximately parallel to the horizontal direction Dx. The rectangle R2 is formed by two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy, and is the smallest rectangle that encloses the second object area OA2 (also called the minimum enclosing rectangle R2). In the figure, for the sake of explanation, the entire contour of the minimum enclosing rectangle R2 is shown apart from the contour of the second object area OA2. In reality, at least a part of the contour of the minimum enclosing rectangle R2 is in contact with the contour of the second object area OA2 (the same applies to FIG. 6(B) and FIG. 6(D)). The width W2 is the length of the minimum enclosing rectangle R2 in the horizontal direction Dx, and the height H2 is the length of the minimum enclosing rectangle R2 in the vertical direction Dy. The width and height of the minimum enclosing rectangle indicate the width and height of the object area enclosed by the minimum enclosing rectangle, respectively. When the object area OA2 shows a ruled line OL2 that is approximately parallel to the horizontal direction Dx, the width W2 is approximately the same as the length of the ruled line OL2 and may be a large value, and the height H2 may be a small value.

[0036] The hatched third object area OA3 in Fig. 6(B) shows a ruled line OL3 that is approximately parallel to the vertical direction Dy. In the figure, the width W3 and height H3 of the minimum bounding rectangle R3 that includes the third object area OA3 are shown. When the object area OA3 shows a ruled line OL3 that is approximately parallel to the vertical direction Dy, the height H3 is approximately the same as the length of the ruled line OL3 and may be a large value. The width W3 may be a small value.

[0037] The sixth object area OA6 in Fig. 6(C) shows one character CH6 (here, a Japanese Kanji character). In the figure, the width W6 and height H6 of the minimum bounding rectangle R6 that includes the sixth object area OA6 are shown. When the object area OA6 shows the character CH6, both the height H6 and the width W6 can be large values.

[0038] The fifth object region OA5 in Fig. 6(D) shows the line OL5 of the bar graph Obg (Fig. 3(A)). The width W5 and height H5 of the minimum bounding rectangle R5 that includes the fifth object region OA5 are shown in the figure. When the object region OA5 shows the line OL5 of the bar graph Obg, both the height H6 and the width W6 can be large values.

[0039] In S320 (FIG. 5), the processor 510 compares each of the height and width with a length threshold ThL. In this embodiment, the length threshold ThL is determined in advance. The length threshold ThL is set to, for example, the minimum value of the length of the line to be removed.

[0040] If both the height and width are less than the length threshold ThL (S320: No), the target object region indicates an object that is small compared to the line to be selected as the line candidate region. In S360, the processor 510 determines that the target object region is a non-line candidate region, and stores candidate determination data indicating the determination result in the memory device 515 (in this embodiment, the non-volatile memory device 530) (not shown). Then, the processor 510 proceeds to S370.

[0041] If one or both of the height and width are equal to or greater than the length threshold ThL (S320: Yes), in S330, processor 510 determines whether the angle of the diagonal of the minimum bounding rectangle is within the horizontal range or the vertical range.

[0042] Lines DL2, DL3, DL6, and DL5 in Figures 6(A) to 6(D) are diagonals of the minimum bounding rectangles R2, R3, R6, and R5, respectively. Angles Ag2, Ag3, Ag6, and Ag5 are the angles of the diagonals DL2, DL3, DL6, and DL5. In this embodiment, the angle of the diagonal indicates an acute angle among the angles formed by a straight line parallel to the horizontal direction Dx and the diagonal. Therefore, the angle of the diagonal is a value of 0 degrees or more and 90 degrees or less.

[0043] The horizontal range is the range of the angle of the diagonal when the diagonal is approximately parallel to the horizontal direction Dx. For example, the horizontal range may be a range of 0 degrees or more and 19 degrees or less. When the attention object region shows a line approximately parallel to the horizontal direction Dx, such as the ruled line OL2 in FIG. 6(A), the angle of the diagonal is within the horizontal range.

[0044] The vertical range is the range of the angle of the diagonal when the diagonal is approximately parallel to the vertical direction Dy. For example, the vertical range may be a range of 71 degrees or more and 90 degrees or less. When the attention object region shows a line approximately parallel to the vertical direction Dy, such as the ruled line OL3 in FIG. 6(B), the angle of the diagonal is within the vertical range.

[0045] If the angle of the diagonal line is within the horizontal range or the vertical range (FIG. 5: S330: Yes), the target object region is highly likely to show a line roughly parallel to the horizontal direction Dx or roughly parallel to the vertical direction Dy, as in the object regions OA2 and OA3 in FIG. 6(A) and FIG. 6(B). In this case, in S350, the processor 510 determines that the target object region is a line candidate region, and stores candidate determination data indicating the determination result in the non-volatile storage device 530 (not shown). Then, the processor 510 proceeds to S370. In this embodiment, the object regions OA2 and OA3 are determined to be line candidate regions.

[0046] If the diagonal angle is outside the horizontal range and outside the vertical range (S330: No), the target object area indicates an object that extends in both the horizontal direction Dx and the vertical direction Dy, such as object areas OA6 and OA5 in Figures 6(C) and 6(D). In this case, processor 510 determines in S340 whether the pixel ratio is less than the character threshold value ThT. The pixel ratio is the ratio of the number of pixels in the object area to the number of pixels in the minimum character bounding rectangle.

[0047] The object area OA6 in Fig. 6C shows the character CH6. The pixels showing the character CH6 (i.e., the pixels in the object area) are distributed over almost the entire minimum bounding rectangle R6. Therefore, the pixel proportion RP may be a large value. Similarly, if the object area shows a character different from the character CH6 in Fig. 6C, the pixel proportion RP may be a large value.

[0048] The object area OA5 in Fig. 6(D) shows the line OL5 of the bar graph Obg. The minimum bounding rectangle R5 includes background portions disposed between the lines. Therefore, the pixel ratio RP may be a small value. In cases where the object area shows various objects including lines, such as the line OL4 of the table Otb (Fig. 3(B)), not limited to the line OL5 of the bar graph Obg, the pixel ratio RP may be a small value.

[0049] In this embodiment, the character threshold value ThT is experimentally determined in advance (for example, ThT=30%) so as to appropriately distinguish between character object regions and non-character object regions (such as tables and graphs).

[0050] If the pixel ratio is less than the character threshold ThT (FIG. 5: S340: Yes), in S350, the processor 510 determines that the target object region is a line candidate region, and stores the candidate determination data in the non-volatile storage device 530. Then, the processor 510 proceeds to S370. In this embodiment, the fifth object region OA5 (FIG. 6(D)) is determined to be a line candidate region.

[0051] If the pixel ratio is equal to or greater than the character threshold ThT (S340: No), in S360, the processor 510 determines that the target object region is a non-line candidate region, and stores the candidate determination data in the non-volatile storage device 530. Then, the processor 510 proceeds to S370. In this embodiment, the sixth object region OA6 (FIG. 6(C)) is determined to be a non-line candidate region.

[0052] In S370, processor 510 determines whether all object regions have been processed. If an unprocessed object region remains (S370: No), processor 510 proceeds to S310 and executes processing of a new target object region. If all object regions have been processed (S370: Yes), processor 510 ends the processing of Fig. 5, i.e., the processing of S230 in Fig. 4.

[0053] 7(A) is an explanatory diagram showing an example of a line candidate region image showing the line candidate regions when the label image I2 (FIG. 3(B)) is processed. As shown in this line candidate region image I3, object regions OA1, OA2, and OA3 including ruled lines OL1, OL2, and OL3, a fourth object region OA4 including a line OL4 of the table Otb, and a fifth object region OA5 including a line OL5 of the bar graph Obg are determined to be line candidate regions.

[0054] 5, the condition for an object region to be determined to be a line candidate region is that one or both of "S320: Yes and S330: Yes" and "S320: Yes and S330: No and S340: Yes" are satisfied. This condition is an example of a line inclusion condition indicating that the object region represents an object that includes a line (such as a line, a table, or a graph).

[0055] In S240 (FIG. 4), processor 510 determines whether or not a horizontal line is to be removed. If the removal target direction determined in S210 is "horizontal" or "both" (S250: Yes), processor 510 executes horizontal line removal processing in S250.

[0056] 8 and 9 are flowcharts showing an example of the horizontal line removal process. In S410, the processor 510 uses the labeling data to generate image data of an image for extracting a horizontal line (also called horizontal image data). The processor 510 refers to the candidate determination data described in S350 and S360 (FIG. 5) and generates image data indicating a line candidate region among a plurality of object regions included in the label image as horizontal image data. When the label image I2 in FIG. 3(B) is processed, the horizontal image data indicates the line candidate region image I3 in FIG. 7(A).

[0057] In S415, the processor 510 selects an unprocessed line candidate region from the line candidate regions represented by the horizontal image data as a line candidate region of interest.

[0058] 10(A)-10(D) are explanatory diagrams showing an example of processing of the attention line candidate region. The hatched region OAx in FIG. 10(A) indicates the attention line candidate region OAx. The attention line candidate region OAx indicates multiple lines of a table (including lines L1-L6). The rectangle Rx is the minimum enclosing rectangle Rx that encloses the attention line candidate region OAx. The left end RxL is the end of the minimum enclosing rectangle Rx on the side opposite to the horizontal direction Dx. The right end RxR is the end of the minimum enclosing rectangle Rx on the horizontal direction Dx side.

[0059] In S420 (FIG. 8), the processor 510 selects a horizontal partial area of ​​interest. FIG. 10(A) shows a horizontal partial area PA. The horizontal partial area PA is a rectangular area extending in the horizontal direction Dx from the left end RxL to the right end RxR of the minimum bounding rectangle Rx. Although not shown, the minimum bounding rectangle Rx is formed by a plurality of horizontal partial areas PA arranged in the vertical direction Dy. That is, the minimum bounding rectangle Rx can be divided into a plurality of horizontal partial areas PA. As described later, the processor 510 determines whether or not each horizontal partial area PA includes a line extending in the horizontal direction Dx for each horizontal partial area PA. In S420, a horizontal partial area of ​​interest that is a horizontal partial area to be processed is selected from unprocessed horizontal partial areas.

[0060] In S425, the processor 510 counts the number of processing areas including pixels of the attention line candidate area among the multiple processing areas included in the attention horizontal partial area. FIG. 10(A) shows a part of multiple processing areas CA among the multiple processing areas CA included in the horizontal partial area PA. The processing areas CA are rectangular areas. The width of the processing area CA in the horizontal direction Dx is, for example, 5 pixels, and the height in the vertical direction Dy is, for example, 5 pixels. Although not shown, the multiple processing areas CA arranged in the horizontal direction Dx form the horizontal partial area PA. That is, the horizontal partial area PA can be divided into multiple processing areas CA. The processor 510 determines whether the processing area CA includes pixels of the attention line candidate area OAx for each of the multiple processing areas CA. For example, the processor 510 determines whether the processing area CA includes pixels of the attention line candidate area OAx for each processing area CA in turn from the left end RxL toward the horizontal direction Dx. Then, the processor 510 counts the number of processing areas CA including pixels of the attention line candidate area OAx.

[0061] 10A includes a first line L1 extending in the horizontal direction Dx from the left end RxL to the right end RxR. Therefore, each of the processing areas CA of the first horizontal partial area PA1 includes pixels of the attention line candidate area OAx.

[0062] 10B shows a horizontal partial area PA (called the second horizontal partial area PA2) adjacent to the first horizontal partial area PA1 (FIG. 10A). The second horizontal partial area PA2 does not include any lines extending in the horizontal direction Dx, but includes portions of each of the lines L4-L6 extending in the vertical direction Dy. Therefore, among the multiple processing areas CA of the second horizontal partial area PA2, the processing areas CAa, CAb, and CAc that overlap the lines L4-L6 include pixels of the attention line candidate area OAx, and the remaining multiple processing areas CA do not include pixels of the attention line candidate area OAx.

[0063] In S430 (FIG. 8), the processor 510 determines whether the area ratio RA of the horizontal partial area of ​​interest is less than the area threshold ThP. The area ratio RA is the ratio of the number of processing areas CA including pixels of the attention line candidate area OAx to the total number of processing areas CA included in the horizontal partial area of ​​interest. Since the first horizontal partial area PA1 in FIG. 10(A) includes the first line L1 extending in the horizontal direction Dx, many processing areas CA in the first horizontal partial area PA1 include pixels of the attention line candidate area OAx. Therefore, the area ratio RA of the first horizontal partial area PA1 can be a large value. Since the second horizontal partial area PA2 in FIG. 10(B) does not include a line extending in the horizontal direction Dx, many processing areas CA in the second horizontal partial area PA2 do not include pixels of the attention line candidate area OAx. Therefore, the area ratio RA of the second horizontal partial area PA2 can be a small value.

[0064] In this embodiment, the region threshold ThP is experimentally determined in advance (e.g., ThP=40%) so that the region proportion RA is equal to or greater than the region threshold ThP when the horizontal partial region PA includes a line extending in the horizontal direction Dx. In this manner, the region proportion RA being equal to or greater than the region threshold ThP is an example of a horizontal candidate condition CCx indicating that the horizontal partial region PA includes a line extending in the horizontal direction Dx.

[0065] If the area ratio RA is less than the area threshold ThP (S430: Yes), in S435, the processor 510 removes all pixels of the horizontal partial area of ​​interest from the line candidate area in the image for horizontal line extraction. For example, for the second horizontal partial area PA2 in FIG. 10(B), the pixel ratio RP is less than the area threshold ThP. Therefore, all pixels in the second horizontal partial area PA2 are removed from the line candidate area OAx. FIG. 10(C) shows the line candidate area OAx after the removal process. The pixels included in the second horizontal partial area PA2 have been removed from the line candidate area OAx. After S435, the processor 510 proceeds to S440.

[0066] If the region proportion RA is equal to or greater than the region threshold ThP (S430: No), the processor 510 skips S435 and proceeds to S440. For example, for the first horizontal partial region PA1 in Fig. 10(A), the pixel proportion RP is equal to or greater than the region threshold ThP. Therefore, the pixels in the first horizontal partial region PA1 are not removed from the line candidate region OAx but are maintained (see Figs. 10(B) and 10(C)).

[0067] In S440, processor 510 determines whether or not the processing of all horizontal partial regions included in the attention line candidate region is completed. If an unprocessed horizontal partial region remains (S440: No), processor 510 proceeds to S420 and processes a new horizontal partial region.

[0068] When all horizontal partial regions have been processed (S440: Yes), in S445, the processor 510 determines whether all line candidate regions represented by the horizontal image data have been processed. When an unprocessed line candidate region remains (S445: No), the processor 510 proceeds to S415 and processes a new line candidate region.

[0069] FIG. 10(D) shows the processed line candidate area OAx. As shown in the figure, lines L4-L6 extending in the vertical direction Dy have been removed from the line candidate area OAx. In this way, pixels of the horizontal partial area that do not include a line extending in the horizontal direction Dx are removed from the line candidate area OAx. For example, if the horizontal partial area does not include a line extending in the horizontal direction Dx but includes a part of a character, the part of the character is removed. Lines L1-L3 extending in the horizontal direction Dx remain. The processed line candidate area OAx shows a plurality of pixels included in the horizontal partial area PA that satisfy the horizontal candidate condition CCx (FIG. 8) among the plurality of pixels of the line candidate area OAx.

[0070] If the processing of all line candidate regions has been completed (S445: Yes), the processor 510 proceeds to S450 in FIG.

[0071] In S450, the processor 510 generates horizontal labeling data indicating an object region by executing a labeling process on the horizontal image data representing the processed image for extracting horizontal lines. In the image for extracting horizontal lines to be subjected to the labeling process, as shown in FIG. 10(D), the parts other than the lines extending in the horizontal direction Dx are removed. Therefore, the labeling process in S450 can detect an object region representing a line extending in the horizontal direction Dx. For example, three object regions representing three lines L1, L2, and L3 are detected from the image in FIG. 10(D). Hereinafter, the object region detected in S450 is referred to as a horizontal line candidate region. The horizontal labeling data represents the detected horizontal line candidate region. The method of the labeling process in S450 is the same as the method of the labeling process of the binary image in S220 in FIG. 4.

[0072] FIG. 7(B) shows an example of a label image represented by horizontal labeling data when the line candidate region image I3 (FIG. 7(A)) is processed. The label image I4 shows a plurality of object regions OA1a, OA2, OA4a, and OA5a, which are a plurality of horizontal line candidate regions. The first object region OA1a shows a ruled line OL1 that is a remainder after removing a plurality of characters of the title Otx1 from the first object region OA1 in FIG. 7(A). The second object region OA2 is the same as the second object region OA2 in FIG. 7(A) and shows a ruled line OL2 extending in the horizontal direction Dx. The plurality of fourth object regions OA4a show a plurality of horizontal lines that are a remainder after removing a plurality of vertical lines from the fourth object region OA4 (i.e., the lines of the table Otb) in FIG. 7(A). The plurality of object regions OA5a show a plurality of horizontal lines that are a remainder after removing a plurality of vertical lines from the fifth object region OA5 (i.e., the lines of the bar graph Obg) in FIG. 7(A). In this way, parts of the object region that indicate elements other than lines extending in the horizontal direction Dx (for example, characters, vertical lines, etc.) are removed.

[0073] In S455 (FIG. 9), the processor 510 selects an unprocessed horizontal line candidate region from the horizontal line candidate regions represented by the horizontal labeling data as a target horizontal line candidate region.

[0074] In S460, the processor 510 determines whether the target horizontal line candidate region satisfies a horizontal target condition. The horizontal target condition is a condition indicating that the horizontal line candidate region represents a line object extending in the horizontal direction Dx. In this embodiment, the horizontal target condition is that all of the following three conditions C11, C12, and C13 are satisfied. (First condition C11) The length in the horizontal direction Dx of the horizontal line candidate region is equal to or greater than the length threshold value ThL. (Second condition C12) The length in the vertical direction Dy of the horizontal line candidate region is equal to or less than the line width threshold ThW. (Third condition C13) The angle of the diagonal of the minimum bounding rectangle of the horizontal line candidate region is within the horizontal range. The length threshold ThL of the first condition C11 is the same as the length threshold ThL of S320 in FIG. 5. The line width threshold ThW of the second condition C12 is predetermined and is set to, for example, the maximum width of the line to be removed. The horizontal range of the third condition C13 is the same as the horizontal range of S330 in FIG. 5. The horizontal symmetry condition composed of these conditions C11-C13 selects an object region that includes one line extending in the horizontal direction Dx and does not include other elements, unlike the line inclusion condition in FIG. 5. For example, with respect to the image in FIG. 10(D), three object regions representing three lines L1, L2, and L3, respectively, may satisfy the horizontal symmetry condition.

[0075] If the target horizontal line candidate area satisfies the horizontal target condition (S460: Yes), in S465, processor 510 changes the color value of each pixel in the target horizontal line candidate area of ​​the read image to a background color value. In this embodiment, the background color value is a color value indicating a color close to white, and is determined in advance. Alternatively, processor 510 may determine the background color value using the color values ​​of each of multiple pixels in the background portion of the read image (for example, the background color value may be an average value of the color values ​​of multiple pixels). Processor 510 removes various horizontal lines shown in FIG. 7(B) from read image I1 (FIG. 3(A)) by the processing of S465. After S465, processor 510 proceeds to S470.

[0076] If the noted horizontal line candidate region does not satisfy the horizontal object condition (S460: No), processor 510 skips S465 and proceeds to S470.

[0077] In S470, processor 510 determines whether or not processing of all horizontal line candidate regions represented by the horizontal labeling data has been completed. If unprocessed horizontal line candidate regions remain (S470: No), processor 510 proceeds to S455 and processes new horizontal line candidate regions. If processing of all horizontal line candidate regions has been completed (S470: Yes), processor 510 ends the processing of Figs. 8 and 9, i.e., the processing of S250 in Fig. 4. Then, processor 510 proceeds to S260 in Fig. 4.

[0078] If the horizontal line is not to be removed (S240: No), processor 510 skips S250 and proceeds to S260.

[0079] In S260, processor 510 determines whether or not vertical lines are to be removed. If the removal target direction determined in S210 is "vertical direction" or "both" (S260: Yes), processor 510 executes vertical line removal processing in S270.

[0080] 11 and 12 are flowcharts showing an example of vertical line removal processing. The only difference from the horizontal line removal processing in FIG. 8 and FIG. 9 is that the horizontal direction Dx and the vertical direction Dy are switched. The reference numerals of each step in FIG. 11 and FIG. 12 are the same as the reference numerals of the corresponding steps in FIG. 8 and FIG. 9 except that the hundreds digit of the numerals is changed from "4" to "5". For example, S510 in FIG. 11 corresponds to S410 in FIG. 8. The processing of each step in FIG. 11 and FIG. 12 is the same as the processing of the corresponding steps in FIG. 8 and FIG. 9 (detailed description will be omitted) except that the horizontal direction Dx and the vertical direction Dy are switched.

[0081] In S510, the processor 510 uses the labeling data and the candidate determination data to generate image data for a vertical line extraction image (also called vertical image data). When the label image I2 in FIG. 3B is processed, the vertical image data represents the line candidate region image I3 in FIG. 7A.

[0082] In S520-S525, the processor 510 calculates an area ratio RA of a vertical partial area (not shown) extending in the vertical direction Dy, instead of a horizontal partial area PA (FIGS. 10(A)-10(C)) extending in the horizontal direction Dx. If the area ratio RA is less than the area threshold ThP (S530: Yes), in S535, the processor 510 removes all pixels of the vertical partial area of ​​interest from the line candidate area in the image for vertical line extraction. For example, when the line candidate area OAx in FIG. 10(A) is processed, the line L1-L3 extending in the horizontal direction Dx is removed from the line candidate area OAx (not shown). In S515-S545, the processor 510 processes all vertical partial areas of all line candidate areas. The processing area CA and the area threshold ThP are the same as the processing area CA and the area threshold ThP in FIG. 8 and FIG. 9, respectively. Moreover, the region ratio RA being equal to or greater than the region threshold value ThP (S530: No) is an example of a vertical direction candidate condition CCy indicating that the vertical partial region includes a line extending in the vertical direction Dy.

[0083] In S550 (FIG. 12), the processor 510 generates vertical labeling data indicating an object region by executing a labeling process on the vertical image data representing the image for extracting vertical lines. Hereinafter, the object region detected in S550 is referred to as a vertical line candidate region.

[0084] FIG. 7(C) shows an example of a label image represented by vertical labeling data when the line candidate region image I3 (FIG. 7(A)) is processed. The label image I5 shows a plurality of object regions OA3, OA4b, and OA5b that are a plurality of vertical line candidate regions. The third object region OA3 is the same as the third object region OA3 in FIG. 7(A) and shows a ruled line OL3 extending in the vertical direction Dy. The plurality of fourth object regions OA4b show a plurality of vertical lines that are the remainder after removing a plurality of horizontal lines from the fourth object region OA4 (i.e., the lines of the table Otb) in FIG. 7(A). The plurality of object regions OA5b show a plurality of vertical lines that are the remainder after removing a plurality of horizontal lines from the fifth object region OA5 (i.e., the lines of the bar graph Obg) in FIG. 7(A). In this way, the portions of the object regions that show elements (e.g., characters, horizontal lines, etc.) different from the lines extending in the vertical direction Dy are removed.

[0085] Processor 510 selects a target vertical line candidate region in S555 (FIG. 12), and determines in S560 whether the target vertical line candidate region satisfies a vertical object condition. The vertical object condition is a condition indicating that the vertical line candidate region represents a line object extending in the vertical direction Dy. In this embodiment, the vertical object condition is that all of the following three conditions C21, C22, and C23 are satisfied. (First condition C21) The length of the vertical line candidate region in the vertical direction Dy is equal to or greater than the length threshold value ThL. (Second condition C22) The length in the horizontal direction Dx of the vertical line candidate region is equal to or smaller than the line width threshold ThW. (Third condition C23) The angle of the diagonal of the minimum bounding rectangle of the vertical line candidate region is within the vertical range. The length threshold ThL and line width threshold ThW are the same as the length threshold ThL and line width threshold ThW, respectively, of S460 in Fig. 9. The vertical range is the same as the vertical range of S330 in Fig. 5. The vertical target condition composed of these conditions C21-C23 differs from the line inclusion condition in Fig. 5 in that it selects an object region that includes one line extending in the vertical direction Dy and does not include other elements.

[0086] If the target vertical line candidate region satisfies the vertical target condition (S560: Yes), in S565, processor 510 changes the color value of each pixel in the target vertical line candidate region of the scanned image to the background color value. By the process of S565, processor 510 removes various vertical lines shown in FIG. 7C from scanned image I1 (FIG. 3A).

[0087] In S555-S570, processor 510 processes all vertical line candidate regions represented by the vertical labeling data. If the processing of all vertical line candidate regions is completed (S570: Yes), processor 510 ends the processing of Fig. 11 and Fig. 12, i.e., the processing of S270 in Fig. 4. Then, processor 510 proceeds to S280 in Fig. 4.

[0088] If the vertical line is not to be removed (S260: No), processor 510 skips S270 and proceeds to S280.

[0089] In S280, the processor 510 stores the line-removed image data representing the line-removed image, which is the scanned image subjected to the line removal processing by one or both of S250 and S270, in the storage device 515 (in this embodiment, the non-volatile storage device 530). Then, the processor 510 ends the processing of FIG. 4, i.e., S130 of FIG. 2.

[0090] FIG. 13 shows an example of a line-removed image. This line-removed image I6 shows an example in which both the horizontal line removal process (S250) and the vertical line removal process (S270) are performed on the read image I1 (FIG. 3(A)). The horizontal lines shown in FIG. 7(B) and the vertical lines shown in FIG. 7(C) are removed from the read image I1. The line-removed image I6 represents the title Otx1, the character string group Otx2, a plurality of characters included in the table Otb, and a plurality of characters included in the bar graph Obg. In this way, since the lines near the characters are removed, the processor 510 can perform appropriate character recognition in S140 of FIG. 2.

[0091] As described above, in this embodiment, in order to detect an object region representing a line extending in the horizontal direction Dx, the processor 510 extracts the object region by a two-stage labeling process of S220 in FIG. 4 and S450 in FIG. 9. When detecting an object region representing a line extending in the horizontal direction Dx in one extraction process, the time required for the process may be long due to the complexity of the process (for example, detailed analysis of the shape of the object is performed). In this embodiment, the processor 510 can quickly detect an object region representing a line extending in the horizontal direction Dx. Similarly, when detecting an object region representing a line extending in the vertical direction Dy, the processor 510 executes a two-stage labeling process of S220 in FIG. 4 and S550 in FIG. 12. Therefore, the processor 510 can quickly detect an object region representing a line extending in the vertical direction Dy.

[0092] Also, as described in FIG. 10(A)-FIG. 10(D), in S430 (FIG. 8) and S530 (FIG. 11), processor 510 extracts a plurality of pixels included in a partial region that satisfies a condition (horizontal candidate condition CCx or vertical candidate condition CCy) indicating that a line extending in the removal target direction is included. Then, processor 510 executes the second stage labeling process (S450 (FIG. 9), S550 (FIG. 12)) using the extracted plurality of pixels. The candidate conditions CCx and CCy can exclude a plurality of pixels in a partial region that does not include a line extending in the removal target direction and includes noise from the target of the second stage labeling process. Therefore, when a read image (for example, read image I1 in FIG. 3(A)) includes noise, the effect of the noise is mitigated.

[0093] Also, in S340 of FIG. 5, the processor 510 uses the pixel ratio RP to determine whether the target object region is a line candidate region. If the object region contains a lot of noise, the pixel ratio RP may be large. In S340, the processor 510 can exclude the region containing a lot of noise from the line candidate region. Therefore, if the scanned image (for example, the scanned image I1 in FIG. 3(A)) contains noise, the effect of the noise is mitigated.

[0094] In this embodiment, the processor 510 executes the following process. In S220 of FIG. 4, the processor 510 executes a labeling process (also called a first labeling process) of target image data representing a target image (for example, the read image I1 of FIG. 3(A)) which is an image of a document optically read by the reading execution unit 280 (FIG. 1) which is an example of a reading device. As shown in FIG. 3(B), the processor 510 detects J object regions (J is an integer of 1 or more) which represent objects by the first labeling process. In S230 of FIG. 4, the processor 510 executes a line candidate region determination process (FIG. 5). As described in FIG. 5, the processor 510 determines whether the object region satisfies a line inclusion condition which indicates that the object region represents an object including a line.

[0095] Furthermore, processor 510 executes one or both of S250 and S270 in Fig. 4. In S250, processor 510 executes the horizontal line removal process in Fig. 8 and Fig. 9. In S450 (Fig. 9), processor 510 executes a labeling process to detect L (L is an integer equal to or greater than 1) horizontal line candidate regions (Fig. 7(B)) that are candidates for object regions indicating lines extending in the horizontal direction Dx (hereinafter, this labeling process is referred to as a horizontal labeling process). The image data to be subjected to the horizontal labeling process is horizontal image data representing an image for extracting horizontal lines that has been processed by the process in Fig. 8. The process of FIG. 8 extracts a plurality of pixels (FIG. 10(D), FIG. 7(B)) included in a horizontal partial area that satisfies a horizontal candidate condition CCx (FIG. 8: S430: No) indicating that a line extending in the horizontal direction Dx is included among a plurality of pixels included in K (K is an integer of 1 or more and J or less) line candidate areas (FIG. 7(A)) that are object areas determined to satisfy the line inclusion condition (FIG. 5). The horizontal labeling process (FIG. 8: S450) uses the extracted plurality of pixels. This enables processor 510 to detect a horizontal line candidate area that is a candidate for an object area indicating a line extending in the horizontal direction Dx (FIG. 7(B)).

[0096] At S460 in Fig. 9, processor 510 determines whether the horizontal line candidate region satisfies a horizontal object condition indicating that the horizontal line candidate region represents a line object extending in the horizontal direction Dx. Then, at S465, processor 510 changes the color values ​​of each of the multiple pixels included in the horizontal line candidate region determined to satisfy the horizontal object condition, among the multiple pixels of the target image (e.g., scanned image I1 (Fig. 3(A)), to the color value of the background (Fig. 13). In this way, processor 510 can appropriately remove the horizontal lines from the target image.

[0097] If multiple pixels whose color values ​​should be changed (i.e., multiple pixels representing horizontal lines) are directly detected from the target image, complex processing is performed to confirm that the pixel of interest does not represent an element other than a horizontal line (e.g., a character, etc.). For example, a process of detecting a crossing pattern of multiple ruled lines may be performed. Analysis of the shape of such an object may be susceptible to noise. In this embodiment, the processor 510 detects the area representing the horizontal line object using a simple process of a two-stage labeling process (S220 in FIG. 4 and S450 in FIG. 9) and the configuration of the object area obtained by the labeling process, so that the time required for processing can be shortened. In addition, since the influence of noise can be mitigated, the area representing the horizontal line object can be appropriately detected. Note that, as the configuration of the object area in this embodiment, the length in the horizontal direction Dx, the length in the vertical direction Dy, the angle of the diagonal of the minimum bounding rectangle, and the area ratio RA are used as described in FIG. 5, FIG. 8, and FIG. 9.

[0098] The same is true when the process of S270 (FIG. 7) is executed. As described above, the process of S270 is obtained by swapping the horizontal direction Dx and the vertical direction Dy in the process of S250. In S270, processor 510 executes the vertical line removal process of FIGS. 11 and 12. In S550 (FIG. 12), processor 510 executes a labeling process to detect M (M is an integer equal to or greater than 1) vertical line candidate regions (FIG. 7(C)) that are candidates for object regions indicating lines extending in the vertical direction Dy (hereinafter, this labeling process is referred to as a vertical labeling process). The image data to be subjected to the vertical labeling process is horizontal image data representing an image for extracting vertical lines that has been processed by the process of FIG. 11. The process of FIG. 11 extracts a plurality of pixels (FIG. 7(C)) included in a vertical partial area that satisfies a vertical candidate condition CCy (FIG. 11: S530: No) indicating that a line extending in the vertical direction Dy is included among a plurality of pixels included in K (K is an integer between 1 and J) line candidate areas (FIG. 7(A)) that are object areas determined to satisfy the line inclusion condition (FIG. 5). The vertical labeling process (FIG. 12: S550) uses the extracted plurality of pixels. This enables processor 510 to detect a vertical line candidate area that is a candidate for an object area indicating a line extending in the vertical direction Dy (FIG. 7(C)).

[0099] In S560 of FIG. 12, processor 510 determines whether the vertical line candidate region satisfies a vertical object condition indicating that the vertical line candidate region represents a line object extending in the vertical direction Dy. Then, in S565, processor 510 changes the color values ​​of each of a plurality of pixels included in the vertical line candidate region determined to satisfy the vertical object condition, among a plurality of pixels of the target image (e.g., scanned image I1 (FIG. 3(A))), to the color value of the background (FIG. 13). In this way, processor 510 can appropriately remove vertical lines from the target image. Processor 510 can also reduce the time required for processing to detect vertical lines.

[0100] 5, in this embodiment, the line inclusion condition is related to the length of the object region in the horizontal direction Dx (S320), the length of the object region in the vertical direction Dy (S320), and the direction in which the diagonal of the minimum bounding rectangle that is a rectangle consisting of two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy and that encloses the object region (S330). Therefore, the processor 510 can appropriately extract an object region that represents an object that includes a line as a line candidate region.

[0101] 5, the line inclusion condition is further related to a pixel ratio RP (S340) which is the ratio of the total number of pixels in the object region enclosed by the minimum bounding rectangle to the total number of pixels in the minimum bounding rectangle. Thus, the processor 510 can appropriately extract object regions representing objects that include lines as line candidate regions.

[0102] 9, the horizontal symmetry condition is related to the length of the object region in the horizontal direction Dx (first condition C11), the length of the object region in the vertical direction Dy (second condition C12), and the angle of the diagonal of the minimum bounding rectangle that is a rectangle consisting of two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy and that encloses the object region (third condition C13). Therefore, processor 510 can appropriately extract the object region representing the horizontal line object.

[0103] 12, the vertical symmetry condition is related to the length of the object region in the vertical direction Dy (first condition C21), the length of the object region in the horizontal direction Dx (second condition C22), and the angle of the diagonal of the minimum bounding rectangle that is a rectangle consisting of two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy and that encloses the object region (third condition C23). Thus, processor 510 can appropriately extract the object region representing the vertical line object.

[0104] B. Second Example: Fig. 14 is a flowchart showing an example of a condition determination process. The condition determination process is a process for determining the character threshold value ThT used in S340 of Fig. 5. In this embodiment, the processor 510 of the user terminal 500 (Fig. 1) determines the character threshold value ThT using a text document including characters whose color values ​​should not be changed. The program 531 is configured to execute the condition determination process in addition to the read image process of Fig. 2.

[0105] In S610, the user places a text document including characters whose color values ​​should not be changed in a portion of the reading execution unit 280 of the multifunction device 200 (FIG. 1) where the object to be read is to be placed. Then, the user operates the operation unit 550 of the user terminal 500 to input an instruction to start processing. The processor 510 of the user terminal 500 supplies a reading instruction to the multifunction device 200. In response to the reading instruction, the processor 210 of the multifunction device 200 causes the reading execution unit 280 to read the text document. The reading execution unit 280 reads the text document and generates read image data representing a text document image, which is an image of the read text document. Hereinafter, this read image data will also be referred to as text document image data.

[0106] In S620, the processor 210 of the multifunction device 200 acquires the text document image data from the reading execution unit 280, and transmits the acquired text document image data to the user terminal 500. The processor 510 of the user terminal 500 acquires the text document image data from the multifunction device 200.

[0107] In S630, processor 510 executes labeling processing of the text document image data to generate labeling data indicating a text object area, which is an object area indicating a text. The labeling processing method is the same as the labeling processing method of S220 in Fig. 4. As a result, a text object area is detected, such as object area OA6 in Fig. 6(C).

[0108] In S640 (FIG. 14), processor 510 calculates the character pixel ratio of the character object region. The method of calculating the character pixel ratio is the same as the method of calculating the pixel ratio RP described in S340 of FIG. 5. The character pixel ratio is the ratio of the total number of pixels in the character object region surrounded by the minimum character enclosing rectangle to the total number of pixels in the minimum character enclosing rectangle, which is the smallest enclosing rectangle that surrounds the character object region.

[0109] In S650, the processor 510 uses the calculated character pixel ratio to determine the character threshold ThT. The processor 510 determines the character threshold ThT to a value smaller than the character pixel ratio (e.g., ThT=RP*0.9) so that a character object region having the character pixel ratio does not satisfy the line inclusion condition. If multiple character object regions are detected in S630, the processor 510 uses the smallest character pixel ratio among the multiple character pixel ratios to determine the character threshold ThT.

[0110] In S660, the processor 510 stores character threshold data indicating the determined character threshold value ThT in the storage device 515 (the non-volatile storage device 530 in this embodiment). Then, the processor 510 ends the processing of Fig. 14. Thereafter, when executing the processing of Fig. 5, the processor 510 uses the character threshold value ThT indicated by the character threshold data.

[0111] As described above, in this embodiment, the processor 510 executes the following processes. In S630, the processor 510 detects a text object region that represents a text object by executing a labeling process on the text document image data. The text document image data is image data that represents a text document image that is an image of a text document that includes text whose color value should not be changed and is optically read by a reading device (the reading execution unit 280 in this embodiment) (S610, S620). In S640-S650, the processor 510 uses the text pixel ratio to determine the text threshold value ThT used to determine the line inclusion condition so that the text object region does not satisfy the line inclusion condition. Therefore, the processor 510 can suppress the change in the color value of the pixel that represents the text. The appropriate text threshold value ThT may differ depending on the attributes of the text, such as the font of the text and the size of the text. In this embodiment, the processor 510 can use the text threshold value ThT that is suitable for the attributes of the text.

[0112] C. Third Example: Fig. 15 is a flow chart showing another embodiment of the line removal process. In the figure, S710, S720, and S730 are shown which are inserted between S220 and S230 in Fig. 4. The other parts of the line removal process are the same as the corresponding parts of the embodiment in Fig. 4 (illustration and description of the same parts are omitted).

[0113] 4, in S710 in FIG. 15, the processor 510 (FIG. 1) causes the display unit 540 to display the object region represented by the labeling data generated in S220 (FIG. 4). In this embodiment, the processor 510 causes the display unit 540 to display the entire image represented by the labeling data. For example, when the labeling data represents the label image I2 in FIG. 3(B), the processor 510 displays the label image I2 on the display unit 540.

[0114] In S720, the user selects an object region including an area whose color value should not be changed, and inputs information indicating the selected object region by operating operation unit 550. Processor 510 uses the signal from operation unit 550 to obtain selected region data, which is data indicating the input information.

[0115] In S730, the processor 510 removes the object region indicated by the selection region data from the labeling data. Specifically, the pixel values ​​of the pixels indicating the object region are changed from values ​​indicating the object region (in this embodiment, the identifier of the object region) to values ​​indicating background pixels. As a result, the pixels included in the object region are excluded from the targets of the change in color value. After S730, the processor 510 proceeds to S230 in FIG. 4.

[0116] As described above, in this embodiment, the processor 510 further executes the following processes. In S710, the processor 510 causes the display unit 540 to display a label image (for example, the label image I2 (FIG. 3(B))) which is an example of information related to the J object regions represented by the labeling data. In S720, the processor 510 acquires selected region data indicating an object region selected by the user and including a region whose color value should not be changed. In S730, a plurality of pixels included in the object region indicated by the selected region data are excluded from the target of changing the color value. In this way, the processor 510 can suppress the change of the color value of the pixels included in the object region indicated by the selected region data. In addition, the user may select any object region as the object region including a region whose color value should not be changed. For example, the user may select an object region including a table or a flag.

[0117] D. Variations: (1) The line inclusion condition is not limited to the condition described in FIG. 5, and may be various conditions indicating that the object region represents an object including a line. For example, the condition may be that the length in the horizontal direction Dx is equal to or greater than the length threshold ThL and equal to or less than the length upper limit. The same applies to the length in the vertical direction Dy. If the horizontal line is not to be removed, the condition related to the "length in the horizontal direction Dx" in the condition of S320 may be omitted. If the horizontal line is not to be removed, the condition related to the horizontal range in the condition of S330 may be omitted. If the vertical line is not to be removed, the condition related to the "length in the vertical direction Dy" in the condition of S320 may be omitted. If the vertical line is not to be removed, the vertical range in S330 may be omitted. S340 related to the pixel ratio RP may be omitted. For example, if the determination result of S330 is No, the processor 510 may proceed to S360. In addition, S330 related to the diagonal line and S340 related to the pixel ratio RP may be omitted. The line inclusion condition may be various conditions related to the length in the horizontal direction Dx, the length in the vertical direction Dy, and the direction in which the diagonal line extends. In general, the line inclusion condition may be a condition that is satisfied not only when the object region represents a single line, but also when the object region includes other elements (e.g., other lines, characters, etc.) in addition to the single line. For example, when detecting an object region that includes a line that extends in a specific direction, either the horizontal direction Dx or the vertical direction Dy, the line inclusion condition may include that the length of the object region in the specific direction is equal to or greater than a first threshold value.

[0118] (2) The horizontal candidate condition CCx in Fig. 8 is not limited to "RA is equal to or greater than ThP" and may be various conditions indicating that the horizontal partial area PA includes a line extending in the horizontal direction Dx. For example, the horizontal candidate condition CCx may be that a partial pixel ratio, which is the ratio of the total number of pixels of the line candidate area in the horizontal partial area PA to the total number of pixels in the horizontal partial area PA, is equal to or greater than a partial pixel threshold. The partial pixel threshold may be experimentally determined in advance so that the partial pixel ratio is equal to or greater than the partial pixel threshold when the horizontal partial area includes a line extending in the horizontal direction Dx. The same applies to the vertical candidate condition CCy in Fig. 11.

[0119] (3) The horizontal symmetry condition in S460 of FIG. 9 may be various other conditions instead of the conditions configured by C11-C13 described above. For example, the condition may be that the length in the horizontal direction Dx is equal to or greater than the length threshold ThL and equal to or less than the length upper limit. The condition may be that the length in the vertical direction Dy is equal to or less than the line width threshold ThW and equal to or greater than the line width lower limit (the line width lower limit is greater than zero). The condition related to the length in the vertical direction Dy (for example, the second condition C12) may be omitted. The third condition C13 related to the diagonal line may be omitted. The horizontal symmetry condition may include that the length in the horizontal direction Dx of the object region is equal to or greater than the second threshold. The horizontal symmetry condition may be various conditions related to the length in the horizontal direction Dx, the length in the vertical direction Dy, and the angle of the diagonal line. In general, it is preferable that the horizontal symmetry condition is configured so as not to be satisfied when the object region includes elements other than lines extending in the horizontal direction Dx. The same applies to the vertical symmetry condition in S560 of FIG. 12. For example, the vertical symmetric condition may include that the length of the object region in the vertical direction Dy is equal to or greater than a third threshold. The vertical symmetric condition may be various conditions related to the length of the horizontal direction Dx, the length of the vertical direction Dy, and the angle of the diagonal. In general, the vertical symmetric condition is preferably configured such that it is not satisfied when the object region includes elements other than lines extending in the vertical direction Dy.

[0120] (4) The process of determining the character threshold ThT in S650 of Fig. 14 may be various processes that use the character pixel ratio. When multiple character object regions are detected in S630, the processor 510 may determine the character threshold ThT using a comprehensive evaluation value (e.g., average value, median value, etc.) of the multiple character pixel ratios. For example, the character threshold ThT may be a value obtained by multiplying the evaluation value by a predetermined coefficient. Here, one or more of the multiple character pixel ratios may be less than the character threshold ThT.

[0121] (5) The information displayed in S710 of FIG. 15 may be any information related to the J object regions represented by the labeling data. For example, processor 510 may cause display unit 540 to display a list of the J object regions by enlarging each object region individually. Processor 510 may also cause display unit 540 to display a list of the J object regions. In general, it is preferable that information including J pieces of object information respectively related to the J object regions is displayed. A user can select an object region by observing the displayed J pieces of object information. The object information may be any information related to the object region, such as an image, a size, or a shape of the object region.

[0122] (6) The process of S730 in Fig. 15 may be any process of excluding a plurality of pixels included in the object region indicated by the selection region data from the target of changing the color value, instead of removing the object region indicated by the selection region data from the labeling data. For example, in S465 in Fig. 9 and S565 in Fig. 12, the processor 510 may skip changing the color value of the pixel for which the exclusion flag is set. In this case, in S730, the processor 510 may set the exclusion flag to each of the plurality of pixels included in the object region indicated by the selection region data.

[0123] (7) The conditions used in the line removal process may be determined by a user. For example, the processor 510 may use parameters determined by the user as parameters (e.g., ThL, ThW, horizontal range, vertical range) for determining the horizontal object condition (FIG. 9, S460) and the vertical object condition (FIG. 12, S560). Furthermore, the processor 510 may use the parameters for determining the horizontal object condition and the vertical object condition to determine the parameters for determining the line inclusion condition of FIG. 5. The parameters for determining the line inclusion condition may be determined so that an object region that does not satisfy the horizontal object condition and the vertical object condition may satisfy the line inclusion condition. For example, the processor 510 may calculate the length threshold ThL of the line inclusion condition by multiplying the length threshold ThL of the horizontal object condition and the vertical object condition by a coefficient less than 1.

[0124] (8) The line removal process may be various other processes instead of the process of the embodiment shown in FIG. 4. For example, one of the horizontal line removal (S250) and the vertical line removal (S260) may be omitted. In addition, the use of the line removed image data representing the line removed image, which is the read image after the line removal process, is not limited to character recognition and may be any use. For example, the processor 510 may display the line removed image on the display unit 540 for the user to observe. The user can easily read the characters.

[0125] (9) The reading device may have any configuration that generates read image data representing a read image, which is an image of the read document, by optically reading the document instead of the configuration of the reading execution unit 280 (FIG. 1). For example, the reading device may include an automatic document feeder (ADF (Auto Document Feeder)) that automatically transports documents one by one. Also, a single-function reading device may be used instead of the multifunction device 200. Also, a portable device such as a digital camera or a smartphone with a camera may be used as the reading device.

[0126] (10) Instead of the user terminal 500, the control unit 299 of the multifunction device 200 may execute the line removal process and, in turn, the read image process of FIG. 2. In this way, a control device provided inside the reading device, or an image processing device connected to the reading device, may execute the line removal process and, in turn, the read image process of FIG. 2. Also, a plurality of devices (e.g., computers) capable of communicating with each other via a network may share a portion of the function of the line removal process each other and, as a whole, provide the function of the line removal process (a system including these devices corresponds to an image processing device). Similarly, a plurality of devices capable of communicating with each other via a network may share a portion of the function of the read image process in FIG. 2 each other and, as a whole, provide the function of the read image process.

[0127] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the function of the labeling process (S220) in FIG. 4 may be realized by a dedicated hardware circuit.

[0128] Furthermore, when some or all of the functions of the present invention are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0129] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the invention, and equivalents thereof are included in the present invention. [Explanation of symbols]

[0130] 200...multifunction device, 210...processor, 215...storage device, 220...volatile storage device, 230...nonvolatile storage device, 232...program, 240...display unit, 250...operation unit, 270...communication interface, 280...reading execution unit, 290...printing execution unit, 299...control unit, 500...user terminal, 510...processor, 515...storage device, 520...volatile storage device, 530...nonvolatile storage device, 531...program, 540...display unit, 550...operation unit, 570...communication interface, 1000...system

Claims

1. A computer program for a computer that processes image data, comprising: A first type detection function that detects J object regions (J is an integer equal to or greater than 1) representing objects by executing a first labeling process on target image data that represents a target image, which is an image of a document optically read by a reading device; a first determination function for determining whether the object region satisfies a line-containing condition indicating that the object region represents an object that includes a line; a first direction candidate detection function that detects L (L is an integer equal to or greater than 1) first direction line candidate regions that are candidates for object regions that indicate a line extending in the first direction by executing a second labeling process that uses a plurality of pixels included in a partial region that satisfies a first direction candidate condition indicating that the object region includes a line extending in the first direction, which is either a horizontal direction or a vertical direction, out of a plurality of pixels included in K (K is an integer equal to or greater than 1 and equal to or less than J) line candidate regions that are object regions that are determined to satisfy the line inclusion condition; a first direction determination function for determining whether the first direction line candidate region satisfies a first direction target condition indicating that the first direction line candidate region represents a line object extending in the first direction; a first direction change function for changing color values ​​of a plurality of pixels included in a first direction line candidate area determined to satisfy the first direction target condition among a plurality of pixels of the target image to a color value of a background; A computer program that enables a computer to realize the above.

2. 2. The computer program of claim 1, further comprising: a second direction candidate detection function that detects M (M is an integer equal to or greater than 1) second direction line candidate regions that are candidates for an object region indicating a line extending in the second direction by executing a third labeling process that uses a plurality of pixels included in a partial region that satisfies a second direction candidate condition indicating that the partial region includes a line extending in a second direction different from the first direction of the horizontal direction and the vertical direction, out of the plurality of pixels included in the K line candidate regions; a second direction determination function for determining whether the second direction line candidate region satisfies a second direction target condition indicating that the second direction line candidate region represents a line object extending in the second direction; A second direction change function that changes the color values ​​of a plurality of pixels included in a second direction line candidate area determined to satisfy the second direction target condition among a plurality of pixels of the target image to a color value of a background; A computer program that enables a computer to realize the above.

3. A computer program according to claim 1 or 2, The line-containing condition is The horizontal length of the object area; The vertical length of the object area; a direction in which a diagonal line of a minimum bounding rectangle that encloses the object region extends, the minimum bounding rectangle being a rectangle having two sides parallel to the horizontal direction and two sides parallel to the vertical direction; Related to computer programs.

4. 4. A computer program according to claim 3, comprising: The line inclusion condition is further associated with a pixel ratio that is a ratio of a total number of pixels of the object area enclosed by the minimum bounding rectangle to a total number of pixels within the minimum bounding rectangle.

5. 5. A computer program as claimed in claim 4, further comprising: a first type character detection function for detecting a character object area representing an object of the character by executing a fourth labeling process of character document image data representing a character document image, the character document image being an image of a character document including characters whose color values ​​should not be changed and optically read by the reading device; a determination function that determines the line inclusion condition so that the character object region does not satisfy the line inclusion condition by using a character pixel ratio that is the ratio of a total number of pixels in the character object region surrounded by a minimum character enclosing rectangle that is the smallest enclosing rectangle that surrounds the character object region to a total number of pixels in the minimum character enclosing rectangle; A computer program that enables a computer to realize the above.

6. A computer program according to any one of claims 1 to 5, The first direction target condition is: The horizontal length of the object area; The vertical length of the object area; a diagonal angle of a minimum bounding rectangle that is a rectangle having two sides parallel to the horizontal direction and two sides parallel to the vertical direction and that encloses the object region; and Related to computer programs.

7. A computer program according to any one of claims 1 to 6, further comprising: a display function for displaying information about the J object regions on a display device; an acquisition function for acquiring selected area data indicating an object area selected by a user, the selected object area including an area whose color value should not be changed; an exclusion function of excluding a plurality of pixels included in the object area indicated by the selected area data from the target of the change of color value; A computer program that enables a computer to realize the above.

8. A computer-readable recording medium having a computer program for a computer that processes image data recorded thereon, The computer program comprises: A first type detection function that detects J object regions (J is an integer equal to or greater than 1) representing objects by executing a first labeling process on target image data that represents a target image, which is an image of a document optically read by a reading device; a first determination function for determining whether the object region satisfies a line-containing condition indicating that the object region represents an object that includes a line; a first direction candidate detection function that detects L (L is an integer equal to or greater than 1) first direction line candidate regions that are candidates for object regions that indicate a line extending in the first direction by executing a second labeling process that uses a plurality of pixels included in a partial region that satisfies a first direction candidate condition indicating that the object region includes a line extending in the first direction, which is either a horizontal direction or a vertical direction, out of a plurality of pixels included in K (K is an integer equal to or greater than 1 and equal to or less than J) line candidate regions that are object regions that are determined to satisfy the line inclusion condition; a first direction determination function for determining whether the first direction line candidate region satisfies a first direction target condition indicating that the first direction line candidate region represents a line object extending in the first direction; a first direction change function for changing color values ​​of a plurality of pixels included in a first direction line candidate area determined to satisfy the first direction target condition among a plurality of pixels of the target image to a color value of a background; A recording medium that enables a computer to realize the above.

9. An image processing device for processing image data, a first type detection unit that detects J object regions (J is an integer equal to or greater than 1) that represent objects by executing a first labeling process on target image data that represents a target image, which is an image of a document optically read by a reading device; a first determination unit that determines whether the object region satisfies a line inclusion condition indicating that the object region represents an object that includes a line; a first direction candidate detection unit that detects L (L is an integer equal to or greater than 1) first direction line candidate areas that are candidates for object areas indicating lines extending in the first direction by performing a second labeling process using a plurality of pixels included in a partial area that satisfies a first direction candidate condition indicating that the object area includes a line extending in a first direction, which is either a horizontal direction or a vertical direction, out of a plurality of pixels included in K (K is an integer equal to or greater than 1 and equal to or less than J) line candidate areas that are object areas determined to satisfy the line inclusion condition; a first direction determination unit that determines whether the first direction line candidate region satisfies a first direction target condition indicating that the first direction line candidate region represents a line object extending in the first direction; a first direction change unit that changes color values ​​of a plurality of pixels included in a first direction line candidate area determined to satisfy the first direction target condition, among a plurality of pixels of the target image, to color values ​​of a background; An image processing device comprising:

Images