Image processing system and image processing program

Abstract

To provide an image processing system and an image processing program for identifying a frame that has been filled inside more properly.SOLUTION: An image processing system is configured to: generate a first image by removing elements in contact with a boundary of a mark sheet area image from the mark sheet area image; generate a second image by removing, from the first image, elements whose area is equal to or smaller than a specific ratio (less than 1) of the area of an element having the largest area in the first image; identify (S122) an element in the second image whose inside is not filled in as a blank frame; identify (S124), of elements within the mark sheet area image, an element whose position in a direction in which the frames are arranged with respect to a character string that represents an item is within a specific distance from the blank frame as a frame; and identify (S125) a frame that has no blank frame corresponding in terms of distance as a filled-in frame, whose inside is filled in.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing system and an image processing program for determining whether or not a box is filled in on an image of a mark-making sheet. [Background technology]

[0002] Conventionally, in an image of a mark-sheet in which an item is selected by filling in the inside of a frame, there is known an image processing system that determines whether or not the inside of a frame is filled in, which determines that the inside of a frame is filled in if the difference in density between the inside of a filled in frame and the inside of an unfilled frame is equal to or greater than a specific density value (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-250173 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional image processing systems, the average density of each pixel row within a frame is calculated and the maximum value is used as the density value within the frame. Therefore, when there is a lot left unpainted within the frame, such as when only a single thin line extending perpendicular to the extension direction of the pixel rows is drawn within the frame, or when the frame is checked with a tick or an X, the density value within the frame will be low. As a result, despite the person who painted within the frame having intended to do so, the image processing system is unable to properly identify frames that have been painted on the inside.

[0005] Therefore, an object of the present invention is to provide an image processing system and an image processing program that can more appropriately identify frames whose insides are painted than in the past. [Means for solving the problem]

[0006] The image processing system of the present invention is an image processing system that identifies a painted frame, which is a frame whose inside is painted, from a region image as an image of a region read by an image reading device from a mark sheet having an area in which a set of a character string representing an item and a frame that is arranged in a first direction relative to the character string and whose inside is painted when the item represented by the character string is selected, are arranged in a second direction perpendicular to the first direction. The image processing system is characterized in that the image processing system generates a first image in which at least elements that are in contact with the boundary of the region image are removed from the region image, generates a second image in which at least elements whose area is equal to or smaller than a specific ratio less than 1 to the element with the largest area in the first image are removed, identifies the elements in the second image as unpainted frames, which are the frames whose inside is not painted, identifies elements in the region image whose position in the first direction is within a specific distance from the unpainted frame as the frame, and identifies the frame for which there is no unpainted frame that corresponds in distance as the painted frame.

[0007] With this configuration, the image processing system of the present invention generates a first image from the region image in which at least elements that are in contact with the boundary of the region image have been removed, generates a second image from the first image in which at least elements whose area is equal to or less than a specific ratio (less than 1) of the element with the largest area in the first image have been removed, identifies the elements in the second image as unpainted frames, which are frames whose insides are not painted, identifies, among the elements in the region image, elements whose position in a direction perpendicular to the arrangement direction of the frames is within a specific distance from the unpainted frames, and identifies frames with no corresponding unpainted frame in terms of distance as painted frames, which are frames whose insides are painted, thereby making it possible to identify frames with painted insides more appropriately than before.

[0008] The image processing system of the present invention may identify the item corresponding to the painted frame based on a list showing the correspondence between the position of the frame in the second direction and the item among the multiple frames in the area, and based on the position of the painted frame in the second direction among the multiple frames in the area.

[0009] With this configuration, the image processing system of the present invention identifies the item that corresponds to the painted frame based on a list showing the correspondence between the position of the frame in the direction in which the frames are arranged and the items, and based on the position of the painted frame in the direction in which the frames are arranged, among the multiple frames in the area, so that the item selected in the area can be identified more appropriately than before.

[0010] The image processing program of the present invention is an image processing program for identifying a painted frame, which is a frame whose inside is painted, from a region image as an image of a region in an image read by an image reading device from a mark sheet having an area in which a set of a character string representing an item and a frame that is arranged in a first direction relative to the character string and whose inside is painted when the item represented by the character string is selected, is arranged in a second direction perpendicular to the first direction, the image processing program being characterized in that it causes a computer to generate a first image in which at least elements that are in contact with the boundary of the region image are removed from the region image, causes the computer to generate a second image in which at least elements whose area is equal to or smaller than a specific ratio less than 1 to the element with the largest area in the first image are removed, causes the computer to identify the elements in the second image as unpainted frames, which are the frames whose inside is not painted, causes the computer to identify, among the elements in the region image, elements whose position in the first direction is within a specific distance from the unpainted frame as the frame, and causes the computer to identify, as the painted frame, a frame for which there is no unpainted frame that corresponds in distance.

[0011] With this configuration, a computer executing the image processing program of the present invention generates a first image by removing from the region image at least elements that are in contact with the boundary of the region image, generates a second image by removing from the first image at least elements whose area is equal to or less than a specific ratio (less than 1) of the largest area element in the first image, identifies the elements in the second image as unpainted frames, that is, frames whose insides are not painted, identifies, among the elements in the region image, elements whose position in a direction perpendicular to the arrangement direction of the frames is within a specific distance from the unpainted frames as frames, and identifies frames with no corresponding unpainted frame in terms of distance as painted frames, that is, frames whose insides are painted, thereby making it possible to identify frames whose insides are painted more appropriately than before. Effect of the Invention

[0012] The image processing system and the image processing program of the present invention can more appropriately identify frames whose insides are painted than in the past. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an image processing system according to an embodiment of the present invention when configured with one computer. [Diagram 2] 2 is a diagram showing an example of a mark sheet handled by the image processing system shown in FIG. 1; [Diagram 3] 10 is a flowchart of the operation of the image processing system shown in FIG. 1 when identifying an item corresponding to a filled-in frame from a mark sheet image. [Figure 4] 4 is a diagram showing an example of a mark sheet area image generated by the pre-processing shown in FIG. 3. FIG. [Diagram 5] 4 is a flowchart of the main body process shown in FIG. 3. [Figure 6] 6 is a flowchart of the unpainted frame identification process shown in FIG. 5 . [Figure 7] 7 is a flowchart of the unpainted frame only processing shown in FIG. 6. [Figure 8]8 is a diagram showing an example of a positive binary image of a mark sheet area image generated by the uncolored frame only processing shown in FIG. 7. [Figure 9] 9 is a diagram showing an example of an image obtained by removing all continuous white pixels from the outer periphery of the positive binary image shown in FIG. 8. FIG. [Figure 10] FIG. 10 is a diagram showing an example of an image obtained by performing an opening process of 1px width on the image shown in FIG. [Figure 11] 11 is a diagram showing an example of an image obtained by performing small area element removal processing on the image shown in FIG. 10. FIG. [Figure 12] FIG. 12 is a diagram showing the center of gravity of each unpainted frame for the image shown in FIG. 11. [Figure 13] 13 is a diagram showing a state in which the variation in the coordinates of the center of gravity shown in FIG. 12 has been corrected. FIG. [Figure 14] 6 is a flowchart of the frame identification process shown in FIG. 5. [Figure 15] 15 is a diagram showing an example of a negative binary image generated by the frame identification process shown in FIG. 14. [Figure 16] 15 is a diagram showing an example of an image including a box generated by the frame identification process shown in FIG. 14. [Figure 17] 15 is a diagram showing an example of the center of gravity of a box generated by the frame identification process shown in FIG. 14. [Figure 18] 15 is a diagram showing an example of the center of gravity of a box corresponding to a frame, generated by the frame identification process shown in FIG. 14. [Figure 19] 6 is a flowchart of the painted frame identification process shown in FIG. 5 . [Figure 20] 20 is a diagram showing an example of the center of gravity of a painted frame identified in the painted frame identification process shown in FIG. 19 . FIG. [Figure 21] 6 is a flowchart of a selection item specification process shown in FIG. 5. [Figure 22] 4 is a flowchart of the post-processing shown in FIG. 3. [Figure 23] 23 is a flowchart of the output information generation process shown in FIG. 22. [Figure 24]20 is a diagram showing an example of a mark sheet area image on which an image for highlighting the center of gravity of a painted frame identified by the painted frame identification process shown in FIG. 19 is superimposed. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] First, the configuration of the image processing system according to the present embodiment will be described.

[0016] FIG. 1 is a block diagram of an image processing system 10 according to the present embodiment, which is configured using one computer.

[0017] As shown in Fig. 1, the image processing system 10 includes an operation unit 11, which is an operation device such as a keyboard or a mouse through which various operations are input, a display unit 12, which is a display device such as an LCD (Liquid Crystal Display) that displays various information, a communication unit 13, which is a communication device that communicates with an external device via a network such as a LAN (Local Area Network) or the Internet, or directly by wire or wirelessly without using a network, a storage unit 14, which is a non-volatile storage device such as a semiconductor memory or an HDD (Hard Disk Drive) that stores various information, and a control unit 15 that controls the entire image processing system 10. The image processing system 10 may be configured by, for example, a PC (Personal Computer). The image processing system 10 may be arranged on a cloud.

[0018] FIG. 2 is a diagram showing an example of a mark sheet that the image processing system 10 handles.

[0019] The mark sheet 20 shown in FIG. 2 has a plurality of areas (hereinafter referred to as "mark sheet areas") in which an item is selected by filling in the frame, including a first attribute mark sheet area 21 in which an item of a first attribute is selected, and a second attribute mark sheet area 22 in which an item of a second attribute is selected.

[0020] The mark sheet area 21 for the first attribute has a plurality of sets of character strings 21a representing the items of the first attribute and frames 21b arranged immediately below the character strings 21a and filled in when the item represented by the character string 21a immediately above is selected, arranged in the left-right direction. The mark sheet area 21 for the first attribute has "ABC", "DEF", and "GHI" as the items of the first attribute.

[0021] The mark sheet area 22 for the second attribute has a plurality of sets of character strings 22a representing items of the second attribute and frames 22b arranged immediately to the right of the character strings 22a and whose insides are filled when the item represented by the character string 22a immediately to the left is selected, arranged vertically. The mark sheet area 22 for the second attribute has "abcdef", "ghijkl", "mnopqr" and "stuvwx" as items of the second attribute.

[0022] 1, the storage unit 14 can store an image processing program 14a for executing a specific process on an input image and outputting the execution result. The image processing program 14a may be installed in the image processing system 10 during the manufacturing stage of the image processing system 10, or may be additionally installed in the image processing system 10 from an external storage medium such as a USB (Universal Serial Bus) memory, or may be additionally installed in the image processing system 10 from a network.

[0023] The memory unit 14 is capable of storing mark sheet information 14b that indicates, for each mark sheet area, the position of each mark sheet area on the mark sheet 20 (see Figure 2), the arrangement direction of the boxes in each mark sheet area, a list indicating which box in each mark sheet area corresponds to which item (hereinafter referred to as the "box item correspondence list"), and the range of the number of boxes with painted insides (hereinafter referred to as "painted boxes") permitted in each mark sheet area.

[0024] The arrangement direction of the boxes in the mark sheet area is left-right when the mark sheet area is the first attribute mark sheet area 21 (see Figure 2), and is up-down when the mark sheet area is the second attribute mark sheet area 22 (see Figure 2).

[0025] The box item correspondence list is a list showing the correspondence between the positions of the boxes in the arrangement direction of the boxes and the items among the multiple boxes in the mark sheet area. When the mark sheet area is the first attribute mark sheet area 21, the box item correspondence list is a list in which the first box from the left corresponds to the item "ABC", the second box from the left corresponds to the item "DEF", and the third box from the left corresponds to the item "GHI", and when the mark sheet area is the second attribute mark sheet area 22, the box item correspondence list is a list in which the first box from the top corresponds to the item "abcdef", the second box from the top corresponds to the item "ghijkl", the third box from the top corresponds to the item "mnopqr", and the fourth box from the top corresponds to the item "stuvwx".

[0026] The control unit 15 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores programs and various data, and a RAM (Random Access Memory) that serves as a memory used as a working area for the CPU of the control unit 15. The CPU of the control unit 15 executes programs stored in the storage unit 14 or the ROM of the control unit 15.

[0027] The control unit 15 executes the image processing program 14a to realize an image processing unit 15a that performs a specific process on an input image and outputs the result of the process.

[0028] Methods of inputting an image into the image processing system 10 include, for example, a method in which image data is input into a specific folder monitored by the image processing system 10, and the image processing system 10 acquires the image data from this folder; a method in which image data is read from a document by an image reading device such as an MFP (Multifunction Peripheral) or a dedicated scanner and is transmitted from the image reading device to the image processing system 10; a method in which image processing system 10 acquires the image data from an email containing image data when the email is received to an email address monitored by the image processing system 10; a method in which image data is transmitted to the image processing system 10 from a web browser on a computer such as a PC to a website provided by the image processing system 10; and a method in which image data is transmitted to the image processing system 10 from a dedicated application installed on a mobile device such as a smartphone.

[0029] The specific processing performed on an image by the image processing system 10 includes, for example, processing for identifying an item corresponding to a filled-in frame from an image read from a mark sheet by an image reading device (hereinafter referred to as a "mark sheet image").

[0030] Methods for transmitting execution results from the image processing system 10 include, for example, inputting the execution results into a specific cloud service, inputting the execution results into a specific folder, and sending an email including the execution results to a specific email address.

[0031] Next, the operation of the image processing system 10 when identifying an item corresponding to a filled-in frame from a mark sheet image will be described.

[0032] The mark sheet image from which the image processing system 10 identifies the items corresponding to the filled-in frames is input to the image processing system 10 by, for example, any of the input methods described above.

[0033] In the following, a mark sheet image read by an image reading device from a mark sheet 20 (see FIG. 2) will be described as an example of a mark sheet image for which the image processing system 10 identifies items corresponding to filled-in frames. The mark sheet image read by an image reading device from the mark sheet 20 may be either a color image or a monochrome image.

[0034] FIG. 3 is a flowchart showing the operation of the image processing system 10 when identifying an item corresponding to a filled-in frame from a mark sheet image.

[0035] When a mark sheet image is input to the image processing system 10, the image processing unit 15a executes the operation shown in FIG.

[0036] As shown in FIG. 3, the image processing unit 15a performs various pre-processing steps on the mark sheet image input to the image processing system 10 (hereinafter referred to as the "input mark sheet image") to improve the accuracy of identifying the item corresponding to the filled-in frame from the mark sheet image (S101).

[0037] In S101, the image processing unit 15a performs a process of “noise removal” for removing noise from the input mark sheet image that may hinder the identification of an item corresponding to a filled-in frame from the mark sheet image, “alpha channel removal” for removing the alpha channel from the input mark sheet image or re-rasterizing the image if the input mark sheet image contains an alpha channel, “grayscale conversion (achromatization)” for converting the input mark sheet image to achromatic if the input mark sheet image is a color image, “light and dark correction” for correcting the range and distribution of lightness in the input mark sheet image if the input mark sheet image is a multi-value image input and if the range of lightness of the entire input mark sheet image is too narrow or too biased, and “light and dark correction” for correcting the input mark sheet image if it is slightly tilted, rotated, or deformed. The following preprocessing steps are performed as necessary: ​​"Frontal alignment and horizontal (rotation and distortion) correction", which corrects the shape of the input mark-sheet image so that it is a rectangle captured from directly in front; "blur correction", which corrects any blurring in the input mark-sheet image; "search and extraction process for target area (mark-sheet area block)" which extracts an image of the mark-sheet area (hereafter referred to as the "mark-sheet area image") from within the page of the input mark-sheet image; and "image size normalization", which corrects and normalizes this and standardizes or suppresses the amount of fluctuation in pixel size, since even when scanning the same type of mark-sheet, there may be a difference of several percent in the pixel size of the image compared to the width of the original document due to differences in the printing width of the printer or printer driver settings used at the time the mark-sheet was printed, or differences in the scanning width of the scanner or scanner driver settings.

[0038] For example, if the input mark sheet image is an RGB 3ch image, the "grayscale conversion (achromatic conversion)" may be calculated as Y=0.3R+0.5G+0.1B.

[0039] The "search and extraction process for target area (mark sheet area block)" may, for example, locate a rectangular area and extract it as a mark sheet area image, or extract the mark sheet area image by masking if the range of the mark sheet area image is known in advance in the mark sheet information 14b.

[0040] Of the pre-processing in S101, processing other than the "search and extraction processing of target area (mark sheet area block)" may be performed at the page size of the input mark sheet image before the mark sheet area image is cut out from the input mark sheet image by the "search and extraction processing of target area (mark sheet area block)", or may be performed on the mark sheet area image after the mark sheet area image is cut out from the input mark sheet image by the "search and extraction processing of target area (mark sheet area block)".

[0041] FIG. 4 is a diagram showing an example of the mark sheet area image generated by the pre-processing in S101.

[0042] The mark sheet area image 30 shown in Fig. 4 is an image of the topmost second attribute mark sheet area 22 of the multiple second attribute mark sheet areas 22 in the mark sheet 20 shown in Fig. 2. The mark sheet area image 30 shown in Fig. 4 has four character strings 22a and four boxes 22b. Of the four boxes 22b, the first, third and fourth boxes 22b from the top are boxes 30a whose insides are not painted (hereinafter referred to as "unpainted boxes"). Of the four boxes 22b, the second box 22b from the top is a painted box 30b.

[0043] The image processing unit 15a generates, by pre-processing in S101, not only an image of the topmost second attribute mark sheet area 22 among the multiple second attribute mark sheet areas 22 on the mark sheet 20, but also images of each of the second attribute mark sheet areas 22 other than the topmost second attribute mark sheet area 22 among the multiple second attribute mark sheet areas 22 on the mark sheet 20, and each image of the multiple first attribute mark sheet areas 21 on the mark sheet 20, in a similar manner to the mark sheet area images shown in FIG. 4.

[0044] For ease of explanation, hereinafter, an image in which the background is substantially white and the characters and frames are substantially black will be referred to as a positive image.

[0045] As shown in FIG. 3, when the pre-processing in S101 is completed, the image processor 15a executes main processing, which is the main part of the processing for identifying an item corresponding to a filled-in frame from the mark sheet image (S102).

[0046] FIG. 5 is a flowchart of the main processing shown in FIG.

[0047] As shown in FIG. 5, the image processor 15a targets only one marking sheet area that has not yet been targeted in the current main process among a plurality of marking sheet areas on the marking sheet 20 (S121).

[0048] When the process of S121 is completed, the image processor 15a executes an unpainted frame identification process for identifying an unpainted frame in a direction perpendicular to the arrangement direction of the frames on the marking sheet area image of the current target marking sheet area (S122).

[0049] FIG. 6 is a flowchart of the unpainted frame identification process shown in FIG.

[0050] As shown in FIG. 6, the image processor 15a executes an unpainted frame only process for generating an image of only an unpainted frame from the mark-sheet area image (S141).

[0051] FIG. 7 is a flowchart of the unpainted frame only processing shown in FIG.

[0052] As shown in FIG. 7, the image processing unit 15a generates a binarized positive image (hereinafter referred to as a "positive binary image") of the mark sheet area image (S161). If the mark sheet area image generated by the pre-processing of S101 is a monochrome multi-value image, the image processing unit 15a may obtain a threshold value using, for example, the Otsu model and perform binarization in S161. If the mark sheet area image generated by the pre-processing of S101 has already been binarized and is a binary image with a white background and a black foreground, i.e., a character string and a frame, the image processing unit 15a does not need to do anything in particular in S161. If the mark sheet area image generated by the pre-processing of S101 has already been binarized and is a binary image with a black background and a black foreground, i.e., a character string and a frame, the image processing unit 15a performs a negative-positive inversion process in S161 so that the background becomes white, and generates a positive binary image with a white background.

[0053] FIG. 8 is a diagram showing an example of a positive binary image of the mark sheet area image generated by the uncolored frame only processing shown in FIG.

[0054] A positive binary image 31 shown in FIG. 8 is a positive binary image of the mark sheet area image 30 shown in FIG.

[0055] As shown in FIG. 7, when the process of S161 is completed, the image processing unit 15a executes a boundary element removal process to remove elements that are in contact with the boundary of the positive binary image generated in S161 from the positive binary image (S162). Specifically, the image processing unit 15a searches for adjacent pixels whose pixel values ​​are white (hereinafter referred to as "white pixels") for each pixel on the outer periphery of the positive binary image generated in S161. When an adjacent white pixel is found, the image processing unit 15a further searches for a white pixel adjacent to the found white pixel. Thereafter, the image processing unit 15a repeats such a process until no adjacent white pixel is found. Note that, as a method for searching for an adjacent white pixel, either a 4-connection method that searches in a total of four directions, including the horizontal and vertical directions of the pixel of interest, or an 8-connection method that searches in a diagonal direction in addition to the horizontal and vertical directions may be adopted. In this manner, once the image processing unit 15a has extracted all consecutive white pixels from the white pixels on the periphery of the positive binary image of the mark sheet area image, it removes all consecutive white pixels from the white pixels on the periphery of the positive binary image, i.e., replaces them with black pixels.

[0056] FIG. 9 is a diagram showing an example of an image obtained by removing all continuous white pixels from the outer periphery of the positive binary image shown in FIG.

[0057] In the image 32 shown in FIG. 9, an inner portion 32a of the frame 22b in the portion of the unpainted frame 30a in the positive binary image 31 shown in FIG. 8 remains white.

[0058] If the inside of the filled frame 30b in the mark-sensing area is filled in completely, the inside of the frame is blackened at the time of binarization in S161, and therefore when the background is removed, i.e., replaced with black, by the process of S162, the filled frame 30b and the inside of this filled frame 30b blend in with the background and disappear in appearance. On the other hand, if there is an unfilled area inside the filled frame 30b in the mark-sensing area, only the unfilled area 32a remains white after the process of S162, as shown in Figure 9. However, the area of ​​each of the multiple unfilled areas 32a is far smaller than the area of ​​the area 32a inside the unfilled frame 30a, as shown in Figure 9.

[0059] In addition, in the portion of the character string 22a of the item name that is paired with the frame 22b, only the portion 32c inside the closed space of each character in the character string 22a remains white as shown in Fig. 9. However, the area of ​​each of the multiple portions 32c is far smaller than the area of ​​the portion 32a inside the uncolored frame 30a as shown in Fig. 9.

[0060] As shown in Fig. 7, when the boundary element removal process of S162 is completed, the image processing unit 15a performs an opening process (S163) to reduce and then expand the image that has been subjected to the boundary element removal process in S162. The opening process can remove minute elements and very fine elements in an image. For example, if the resolution of the mark sheet area image is 300 dpi and the size of the characters in the mark sheet area image is 10 points, for example, an opening process of 1px width to 2px width may be performed.

[0061] FIG. 10 is a diagram showing an example of an image obtained by performing an opening process of 1px width on the image shown in FIG.

[0062] Image 33 shown in Fig. 10 has had the minute elements and extremely fine elements removed from image 32 shown in Fig. 9. That is, image 33 shown in Fig. 10 has had all of unpainted portion 32a inside painted frame 30b (see Fig. 8) removed. Also, image 33 shown in Fig. 10 has had part of portion 32c inside closed space within each character in character string 22a (see Fig. 8) removed.

[0063] 7, when the opening process of S163 is completed, the image processing unit 15a executes an element coordinate area acquisition process (S164) for acquiring the coordinates and area of ​​an in-image element (an independent island-like cluster of white pixels) for the image for which the opening process has been executed in S163. Specifically, the image processing unit 15a focuses on white pixels in the image, extracts the coordinate positions of the white pixels, and then extracts clusters of white pixels by determining the continuity of the pixels adjacent to the white pixels whose coordinate positions have been extracted, and further calculates the area of ​​each cluster of white pixels from the number of pixels that constitute it.

[0064] When the element coordinate area acquisition process of S164 is completed, the image processing unit 15a executes a small area element removal process for removing elements with small areas from the image on which the opening process has been executed in S163 (S165). Specifically, the image processing unit 15a identifies the maximum area among the areas of each element in the image, and removes all elements with an area equal to or less than a specific ratio less than 1 to this maximum area from the image on which the opening process has been executed in S163. The specific ratio in the small area element removal process may be, for example, 0.8. The "removal" referred to here is a process that essentially paints the white part of the part in question black to blend it into the black background and make its presence invisible from the outside, and in terms of image processing, it corresponds to replacing the pixel value of each element in question from a white color value to a black (background color) color value.

[0065] FIG. 11 is a diagram showing an example of an image obtained by performing small area element removal processing on the image shown in FIG.

[0066] Image 34 shown in Fig. 11 has been removed of the small area elements in image 33 shown in Fig. 10. That is, image 34 shown in Fig. 11 has been removed of all of inner portions 32c of closed spaces in each character in character string 22a (see Fig. 8).

[0067] In the image 33 shown in Fig. 10, it is clear that the element with the largest area is the portion 32a inside the unpainted frame 30a (see Fig. 4) in the mark-marking area image 30 (see Fig. 4). The portions 32a inside the unpainted frame 30a in the mark-marking area image 30 have approximately the same area no matter how many there are in the mark-marking area image 30, unless there is an extremely extreme aberration or distortion in the mark-marking area image 30. Therefore, even if a relatively high value is set as the specific ratio in the small area element removal process in S165, the portions 32a inside the unpainted frame 30a in the mark-marking area image 30 are not removed, and as a result, the image processing unit 15a does not miss the portions 32a inside the unpainted frame 30a.

[0068] Since a relatively high numerical value can be set as the specific ratio in the small area element removal process of S165, elements other than the inner portion 32a of the unpainted frame 30a in the mark sheet area image 30 are removed, and as a result, there is almost no possibility that the image processing unit 15a will erroneously extract elements other than the inner portion 32a of the unpainted frame 30a as the inner portion 32a of the unpainted frame 30a.

[0069] For example, in the character string 22a (see FIG. 4) of the item name that is paired with the frame 22b (see FIG. 4) and arranged near the frame 22b, small remaining elements may remain after the opening process of S163 is executed, as shown in FIG 10. However, since the area of ​​such small remaining elements is far smaller than the area of ​​the inner portion 32a of the unpainted frame 30a, they are removed by the small area element removal process of S165, that is, replaced with the same black color as the background, as shown in FIG 11.

[0070] Also, in the painted frame 30b (see FIG. 4), if some areas have not been painted carefully and some areas have not been painted, small remaining elements may remain after the opening process of S163 is executed, as shown in FIG. 10. However, the area of ​​such small remaining elements is much smaller than the area of ​​the inner part 32a of the unpainted frame 30a, so they are removed by the small area element removal process of S165, that is, replaced with the same black color as the background color.

[0071] Therefore, in the image 34 after the small area element removal process in S165 has been performed, as shown in FIG. 11, only the portion 32a inside the unpainted frame 30a remains.

[0072] As shown in FIG. 7, when the small area element removal process in S165 ends, the image processing unit 15a ends the unpainted frame only process shown in FIG.

[0073] 6, when the process of S141 for removing only unpainted borders is completed, the image processor 15a calculates the coordinates of each element, i.e., the center of gravity of each unpainted border, for the image in a state in which only unpainted borders remain as a result of the process of S141 for removing only unpainted borders (S142). For example, the image processor 15a may extract the coordinates of white pixels for the image in a state in which only unpainted borders remain as a result of the process of S141 for removing only unpainted borders, and determine whether adjacent pixels are connected with the same color value to extract clusters of white pixels that form islands, i.e., elements, and for each extracted cluster, calculate the arithmetic average of the x coordinates and the arithmetic average of the y coordinates of all pixels that make up the cluster as the x coordinate and y coordinate of the center of gravity, respectively.

[0074] FIG. 12 is a diagram showing the centers of gravity 34a of the unpainted frames 30a (see FIG. 4) for the image 34 shown in FIG.

[0075] In FIG. 12, the x coordinates of the centers of gravity 34a of the unpainted frames 30a are different.

[0076] As shown in FIG. 6, when the process of S142 is completed, the image processing unit 15a performs MeanShift processing on each of the x coordinates and y coordinates of all the centers of gravity whose coordinates have been calculated in S142 (S143).

[0077] The MeanShift process in S143 is a process in which, for each element in the sequence list, if there are other elements within a specified distance in the sequence list, the element is replaced with the average value of those elements.

[0078] For example, when MeanShift is applied to a sequence list of {1,10,0,0,2,5,6,7,6,2,1} with a specified distance of 2, the element "0" is replaced with "1", which is the average value between the elements "0", "1", "1", "2", and "2" in the sequence list, since the other elements within the specified distance are the elements "0", "0", "1", "2", and "2" in the sequence list. The element "1" is replaced with "1", which is the average value between the elements "0", "0", "1", "2", and "2" in the sequence list, since the other elements within the specified distance are the elements "0", "0", "1", "1", and "2" in the sequence list, since the other elements within the specified distance are the elements "2", "0", "0", "1", "1", and "2" in the sequence list, since the other elements within the specified distance are the elements "2", "0", "0", "1", "1", and "2" in the sequence list, since the other elements within the specified distance are the elements "5", "6", "6", and "7" in the sequence list, since the other elements within the specified distance are the elements "5", "6", which is the average value between the elements "5" and "6". For the element "6", since the elements "5", "6", and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "6", which is the average value of these elements. For the element "7", since the elements "5", "6", and "6" exist as other elements within the specified distance in the sequence list, it is replaced with "6", which is the average value of these elements. Therefore, when MeanShift processing is performed on a sequence list of {1,10,0,0,2,5,6,7,6,2,1} with a specified distance of 2, the sequence list becomes {1,10,1,1,1,6,6,6,6,1,1}.

[0079] When performing MeanShift processing on a sequence list of {1,10,0,0,2,5,6,7,6,2,1} with a specified distance of 1, the element "0" is replaced with "1 / 2", which is the average value between the elements "0", "1", and "1" in the sequence list, since the elements "0", "0", "1", "2", and "2" are other elements within the specified distance in the sequence list, the element "1" is replaced with "1", which is the average value between the elements "0", "0", "1", "2", and "2" in the sequence list, since the elements "2" are other elements within the specified distance in the sequence list, since the elements "1", "1", and "2" are other elements within the specified distance in the sequence list, the element "2" is replaced with "3 / 2", which is the average value between the elements "2", "1", "1", and "2" in the sequence list, since the elements "5" are other elements within the specified distance in the sequence list, since the elements "6" and "6" are other elements within the specified distance in the sequence list, the element "5" is replaced with "17 / 3", which is the average value between the elements "5" and "6". For element "6", since the elements "5", "6", and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "6", which is the average value of these elements. For element "7", since the elements "6" and "6" exist as other elements within the specified distance in the sequence list, it is replaced with "19 / 3", which is the average value of these elements. Therefore, when MeanShift processing is performed on a sequence list of {1,10,0,0,2,5,6,7,6,2,1} with a specified distance of 1, the sequence list becomes {1,10,1 / 2,1 / 2,3 / 2,17 / 3,6,19 / 3,6,3 / 2,1}.

[0080] When performing MeanShift processing on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 1, the element "1" is replaced with "3 / 2", which is the average value between the element "1" and the elements "2", because the other elements within the specified distance in the sequence list are the elements "1" and "3", and the element "2" is replaced with "2", which is the average value between the elements "1" and "3". The element "3" is replaced with "3", which is the average value between the elements "2" and "4", because the other elements within the specified distance in the sequence list are the elements "2" and "4". The element "4" is replaced with "4", which is the average value between the elements "3" and "5", because the other elements within the specified distance in the sequence list are the elements "3" and "5". The element "5" is replaced with "5", which is the average value between the elements "4" and "6", because the other elements within the specified distance in the sequence list are the elements "4" and "6". For the element "6," since there are elements "5" and "7" within the specified distance in the sequence list, it is replaced with "6," which is the average value of these elements. For the element "7," since there is an element "6" within the specified distance in the sequence list, it is replaced with "13 / 2," which is the average value of these elements. Therefore, when MeanShift processing is performed on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 1, the sequence list becomes {3 / 2,2,3,4,5,6,13 / 2}.

[0081] When performing MeanShift processing on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 2, the element "1" is replaced with "2", which is the average value between the elements "2" and "3" since the elements "2" and "3" exist as other elements within the specified distance in the sequence list. The element "2" is replaced with "5 / 2", which is the average value between the elements "1", "3", and "4" since the elements "2" and "3" exist as other elements within the specified distance in the sequence list. The element "3" is replaced with "3", which is the average value between the elements "1", "2", "4", and "5" since the elements "4" and "2" exist as other elements within the specified distance in the sequence list. For the element "5," since the elements "3," "4," "6," and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "5," which is the average value of these elements. For the element "6," since the elements "4," "5," and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "11 / 2," which is the average value of these elements. For the element "7," since the elements "5" and "6" exist as other elements within the specified distance in the sequence list, it is replaced with "6," which is the average value of these elements. Therefore, when a MeanShift operation is performed on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 2, the sequence list becomes {2,5 / 2,3,4,5,11 / 2,6}.

[0082] When performing MeanShift processing on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 3, the element "1" is replaced with "5 / 2", which is the average value between the elements "2", "3", and "4" in the sequence list, since the other elements within the specified distance are the elements "1", "3", "4", and "5" in the sequence list, and the element "2" is replaced with "3", which is the average value between the elements "2", "3", and the element "3" is replaced with "7 / 2", which is the average value between the elements "3", since the other elements within the specified distance are the elements "1", "2", "4", "5", and "6" in the sequence list, and the element "4" is replaced with "4", which is the average value between the elements "2", "3", "4", and the element "5" in the sequence list, since the other elements within the specified distance are the elements "1", "2", "4", and "5" in the sequence list, and the element "4" is replaced with "4", which is the average value between the elements "2", "3", "4", and the element "2" is replaced with "3", which is the average value between the elements "3", since the other elements within the specified distance are the elements "1", "2", "4", "5", and "6" in the sequence list, and the element "4" is replaced with "4", which is the average value between the elements "2", "3", "4", and the element "2" is replaced with "3", which is the average value between the elements "3", since the other elements within the specified distance are the elements "1", "2", "4", "5", "6", and "7" in the sequence list. For the element "5," since the elements "2," "3," "4," "6," and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "9 / 2," which is the average value of these elements. For the element "6," since the elements "3," "4," "5," and "7" exist as other elements within the specified distance in the sequence list, it is replaced with "5," which is the average value of these elements. For the element "7," since the elements "4," "5," and "6" exist as other elements within the specified distance in the sequence list, it is replaced with "11 / 2," which is the average value of these elements. Therefore, when a MeanShift operation is performed on a sequence list of {1,2,3,4,5,6,7} with a specified distance of 3, the sequence list becomes {5 / 2,3,7 / 2,4,9 / 2,5,11 / 2}.

[0083] The MeanShift process replaces each element in a sequence list with the average value of other elements within a specified distance, so that neighboring elements in the sequence list are replaced with the same value. Therefore, the MeanShift process has the effect of absorbing or suppressing variations in the sequence list. By repeating the MeanShift process multiple times, the elements in the sequence list converge to an invariant value.

[0084] If the degree of the remaining amount of distortion correction and horizontal and vertical adjustment by the pre-processing in S101 is known, a distance slightly exceeding the remaining amount may be set as the specified distance in the MeanShift process, i.e., the correction distance. For example, if the resolution of the mark sheet area image is 300 dpi and the size of the characters in the mark sheet area image is 10 points, a distance of about 5px may be set as the initial value of the correction distance in the MeanShift process.

[0085] FIG. 13 is a diagram showing a state in which the variation in coordinates of the center of gravity 34a shown in FIG. 12 has been corrected.

[0086] In FIG. 13, the x coordinates of the centers of gravity 34a of all the unpainted frames 30a (see FIG. 4) are the same.

[0087] As shown in Fig. 6, when the process of S143 is completed, the image processor 15a judges whether or not the coordinates in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area are aligned among the x and y coordinates of the multiple centroids subjected to the MeanShift process in S143 (S144). For example, when the currently targeted mark sheet area is the first attribute mark sheet area 21 (see Fig. 2), the image processor 15a judges in S144 whether or not the coordinates in the up-down direction perpendicular to the left-right direction, which is the arrangement direction of the frames shown in the mark sheet information 14b for the first attribute mark sheet area 21, i.e., the y coordinates, are aligned among the x and y coordinates of the multiple centroids subjected to the MeanShift process in S143. Furthermore, when the currently targeted mark sheet area is the second attribute mark sheet area 22 (see FIG. 2), the image processing unit 15a determines in S144 whether or not the x and y coordinates of the multiple centers of gravity subjected to MeanShift processing in S143 are aligned in the left-right direction perpendicular to the up-down direction, which is the arrangement direction of the frames shown in the mark sheet information 14b, for the second attribute mark sheet area 22, i.e., the x coordinates.

[0088] When the image processing unit 15a determines in S144 that, among the x and y coordinates of the multiple centroids on which MeanShift processing was performed in S143, the coordinates in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area are not aligned, the image processing unit 15a performs MeanShift processing again on the x and y coordinates of all the centroids on which MeanShift processing was performed in S143 (S145).

[0089] When the processing of S145 is completed, the image processing unit 15a determines whether or not the x-coordinates and y-coordinates of the multiple centers of gravity that have been subjected to MeanShift processing in S145 are aligned in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area (S146).

[0090] When the image processor 15a determines in S146 that the coordinates in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area are not aligned among the x and y coordinates of the multiple centers of gravity that have been subjected to the MeanShift process in S145, the image processor 15a determines whether the prescribed distance in the immediately preceding MeanShift process is equal to or greater than a specific distance (S147). For example, the specific distance in S147 may be half the width of the frames in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area.

[0091] If the image processing unit 15a determines in S147 that the specified distance in the immediately preceding MeanShift process is not equal to or greater than the specific distance, it increases the specified distance in the MeanShift process by a specific value from the current value (S148) and executes the process of S145.

[0092] If the image processing unit 15a determines in S147 that the specified distance in the immediately preceding MeanShift processing is equal to or greater than the specific distance, it determines that it has failed to identify the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames (S149), and terminates the unpainted frame identification processing shown in FIG. 6.

[0093] When the image processing unit 15a determines in S144 that the x and y coordinates of the multiple centers of gravity after MeanShift processing in S143 are aligned in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area, it identifies the x and y coordinates of the multiple centers of gravity after correction for variation in S143 in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area as the representative value of the coordinates of the unpainted frames in the direction perpendicular to the arrangement direction of the frames (S150), and terminates the unpainted frame identification process shown in Figure 6.

[0094] Similarly, when the image processing unit 15a determines in S146 that the x and y coordinates of the multiple centers of gravity that have been subjected to MeanShift processing in S145 are aligned in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area, the image processing unit 15a identifies the x and y coordinates of the multiple centers of gravity that have been corrected for variation in S145 in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area as the representative value of the coordinates of the unpainted frames in the direction perpendicular to the arrangement direction of the frames (S150), and terminates the unpainted frame identification process shown in Figure 6.

[0095] As shown in FIG. 5, when the unpainted frame identification process of S122 is completed, the image processing unit 15a determines whether or not it has determined that identification of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed (S123).

[0096] When the image processor 15a determines in S123 that it has failed to specify the representative value of the coordinates of the unpainted frames in the direction perpendicular to the arrangement direction of the frames, it ends the main process shown in FIG.

[0097] If the image processing unit 15a determines in S123 that it has not failed to identify the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames, it executes a frame identification process to identify the frame on the mark sheet area image of the current target mark sheet area (S124).

[0098] FIG. 14 is a flowchart of the frame identification process shown in FIG.

[0099] As shown in FIG. 14, the image processing unit 15a generates an image (hereinafter referred to as a "negative binary image") by performing negative-positive inversion on the positive binary image generated in S161 (see FIG. 7) (S181).

[0100] FIG. 15 is a diagram showing an example of a negative binary image generated by the frame identification process shown in FIG.

[0101] A negative binary image 40 shown in FIG. 15 is an image obtained by negative-positive inversion of the positive binary image 31 shown in FIG.

[0102] As shown in FIG. 14, the image processing unit 15a surrounds elements in the image of the negative binary image generated in S181 with a bounding box (S182), and generates a box by filling the inside of the bounding box generated in S182 with white (S183).

[0103] FIG. 16 is a diagram showing an example of an image including a box generated by the frame identification process shown in FIG.

[0104] An image 41 shown in Fig. 16 is provided with boxes surrounding elements in the image, namely, the unpainted frame 30a, the painted frame 30b, and all or part of each character of the character string 22a, as compared to the negative binary image 40 shown in Fig. 15. Fig. 16 shows a box 41a surrounding the unpainted frame 30a, a box 41b surrounding the painted frame 30b, and a box 41c surrounding all or part of each character of the character string 22a.

[0105] As shown in Fig. 14, when the process of S183 is completed, the image processing unit 15a calculates the coordinates of the center of gravity of the box generated in S183 (S184). For example, the image processing unit 15a may extract the coordinates of white pixels from the image in which the box was generated in S183, determine whether adjacent pixels are connected with the same color value, extract clusters of white pixels that form islands, i.e., elements, and obtain the arithmetic average of the x coordinates and the arithmetic average of the y coordinates of all pixels constituting the cluster as the x coordinate and the y coordinate of the center of gravity, respectively, for each cluster extracted. In addition, when the bounding box generated in S182 is defined as a rectangle by the coordinates of two diagonal vertices of the bounding box, the image processing unit 15a may obtain the coordinates of the center of gravity by averaging the coordinates of the two diagonal vertices of the bounding box. In addition, when the bounding box generated in S182 is defined by the coordinates of one vertex of the bounding box, the width, and the height, the image processing unit 15a may shift the coordinates of one vertex of the bounding box by half the width of the bounding box in the width direction of the bounding box and half the height of the bounding box in the height direction of the bounding box, as the coordinates of the center of gravity.

[0106] FIG. 17 is a diagram showing an example of the centers of gravity of the boxes 41a, 41b, and 41c generated by the frame specification process shown in FIG.

[0107] The center of gravity 42a shown in Fig. 17 is the center of gravity of the box 41a shown in Fig. 16. The center of gravity 42b shown in Fig. 17 is the center of gravity of the box 41b shown in Fig. 16. The center of gravity 42c shown in Fig. 17 is the center of gravity of the box 41c shown in Fig. 16.

[0108] 14, when the process of S184 is completed, the image processor 15a specifies, among the centroids whose coordinates were calculated in S184, the centroid whose coordinate in the direction perpendicular to the arrangement direction of the frames shown in the mark-sheet information 14b for the currently targeted mark-sheet area is within a specific distance from the representative value of the coordinate of the unpainted frame in the direction perpendicular to the arrangement direction of the frames specified in S150 (see FIG. 6) as the centroid of the frame (S185). For example, the specific distance in S185 may be a distance of 1 / 2 the width of the frame in the direction perpendicular to the arrangement direction of the frames shown in the mark-sheet information 14b for the currently targeted mark-sheet area.

[0109] FIG. 18 is a diagram showing an example of the center of gravity of a box corresponding to a frame, which is generated by the frame identification process shown in FIG.

[0110] The centers of gravity shown in Fig. 18 are the centers of gravity 42a and 42b of the boxes 41a and 41b corresponding to the frame 22b (see Fig. 15) among the centers of gravity 42a to 42c shown in Fig. 17. The centers of gravity shown in Fig. 18 do not include the center of gravity 42c of the box 41c corresponding to all or part of the characters in the character string 22a (see Fig. 15) among the centers of gravity 42a to 42c shown in Fig. 17. In other words, the centers of gravity shown in Fig. 18 are the centers of gravity 42a to 42c shown in Fig. 17 excluding the center of gravity 42c of the box 41c corresponding to all or part of the characters in the character string 22a.

[0111] As shown in FIG. 14, when the process of S185 ends, the image processor 15a ends the frame identification process shown in FIG.

[0112] As shown in FIG. 5, when the frame identifying process of S124 ends, the image processor 15a executes a painted frame identifying process for identifying a painted frame on the mark-sheet area image of the currently targeted mark-sheet area (S125).

[0113] FIG. 19 is a flowchart of the painted frame identification process shown in FIG.

[0114] As shown in FIG. 19, the image processing unit 15a targets only one of the centroids of the frames identified in S185 that has not yet been targeted in the current painted frame identification process (S201).

[0115] When the processing of S201 is completed, the image processing unit 15a determines whether or not there is an unpainted frame whose center of gravity calculated in the unpainted frame identification processing of S122 is located at a distance equal to or less than a specific distance from the center of gravity of the current target frame (S202).

[0116] When the image processing unit 15a determines in S202 that the distance between the center of gravity of the current target frame and the center of gravity of an unpainted frame calculated in the unpainted frame identification process of S122 is less than a specific distance, the image processing unit 15a identifies the center of gravity of the current target frame as the center of gravity of a painted frame (S203).

[0117] When the image processing unit 15a determines in S202 that there is an unpainted frame center of gravity calculated in the unpainted frame identification process of S122 whose distance from the center of gravity of the currently targeted frame is less than a specific distance, or when the image processing unit 15a ends the process of S203, it determines whether or not there is a center of gravity of the frame identified in S185 that has not yet been targeted in the current painted frame identification process (S204).

[0118] When the image processing unit 15a determines in S204 that there is a center of gravity that has not yet been targeted in the current painted frame identification process among the center of gravity of the frame identified in S185, the image processing unit 15a executes the process of S201.

[0119] When the image processing unit 15a determines in S204 that there are no centroids among the centroids of the frames identified in S185 that have not yet been targeted in the current painted frame identification process, the image processing unit 15a terminates the painted frame identification process shown in Figure 19.

[0120] FIG. 20 is a diagram showing an example of the center of gravity of a painted frame identified in the painted frame identification process shown in FIG.

[0121] When the center of gravity of the unpainted frame calculated in the unpainted frame identification process of S122 is the center of gravity 34a shown in FIG. 13, and the center of gravity of the frame identified in S185 is the center of gravity 42a, 42b shown in FIG. 18, the image processing unit 15a identifies the center of gravity 42b shown in FIG. 20 in the painted frame identification process shown in FIG. 19.

[0122] As shown in FIG. 5, when the painted frame identification process of S125 ends, the image processing unit 15a executes a selected item identification process for identifying a selected item on the mark sheet area image of the currently targeted mark sheet area (S126).

[0123] FIG. 21 is a flowchart of the selection item specification process shown in FIG.

[0124] As shown in Figure 21, the image processing unit 15a determines the order of the centers of gravity of the painted frames identified in the painted frame identification process of S125 among the centers of gravity of all frames identified in S185 in the arrangement direction of the frames shown in the mark sheet information 14b for the current target mark sheet area (S221).

[0125] For example, if the center of gravity of the frame identified in S185 is center of gravity 42a, 42b shown in Figure 18, and the center of gravity of the painted frame identified in the painted frame identification process of S125 is center of gravity 42b shown in Figure 20, the image processing unit 15a will determine that in the arrangement direction of the frames shown in the mark sheet information 14b relative to the current target mark sheet area, i.e., in the y coordinate direction, the coordinate of the center of gravity of the painted frame identified in the painted frame identification process of S125 is the second from the top among the coordinates of the centers of gravity of all the frames identified in S185.

[0126] When the processing of S221 is completed, the image processing unit 15a identifies (S222) the item corresponding to the painted frame whose center of gravity was identified in the painted frame identification processing of S125, based on the order identified in S221 and the frame item correspondence list stored in the mark sheet information 14b for the current target mark sheet area.

[0127] When the process of S222 ends, the image processing unit 15a ends the selection item specification process shown in FIG.

[0128] As shown in FIG. 5, when the selection item identification process of S126 is completed, the image processing unit 15a determines whether or not there is any mark-sheet area among the multiple mark-sheet areas on the mark-sheet 20 that has not yet been targeted in the current main process (S127).

[0129] When the image processor 15a determines in S127 that there is a mark-sheet area that has not yet been the target of the current main process among the multiple mark-sheet areas on the mark-sheet 20, the image processor 15a executes the process of S121.

[0130] When the image processor 15a determines in S127 that there is no mark-sheet area that has not yet been the target of the current main process among the multiple mark-sheet areas on the mark-sheet 20, the image processor 15a ends the main process shown in FIG.

[0131] As shown in FIG. 3, when the main process of S102 ends, the image processing unit 15a executes post-processing of the main process (S103).

[0132] FIG. 22 is a flowchart of the post-processing shown in FIG.

[0133] As shown in FIG. 22, the image processor 15a determines whether or not it has been determined in the main process of S102 that identification of a representative value of the coordinates of an unpainted frame in a direction perpendicular to the arrangement direction of the frames has failed (S241).

[0134] When the image processing unit 15a determines in S241 that the main processing of S102 has failed to identify the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames, the image processing unit 15a displays an error on the display unit 12 (S242) and terminates the post-processing shown in Figure 22.

[0135] When the image processing unit 15a determines in S241 that the main processing of S102 has not failed to identify the representative value of the coordinates of the unpainted frame in the direction perpendicular to the frame arrangement direction, it executes an output information generation process (S243) to generate information to be output to subsequent processing (hereinafter referred to as "output information").

[0136] FIG. 23 is a flowchart of the output information generating process shown in FIG.

[0137] As shown in FIG. 23, the image processor 15a targets only one marking sheet area that has not yet been targeted in the current output information generating process among a plurality of marking sheet areas on the marking sheet 20 (S261).

[0138] When the processing of S261 is completed, the image processing unit 15a determines whether or not the number of filled-in frames identified by the main processing of S102 for the currently targeted mark sheet area is within the permitted range based on the mark sheet information 14b (S262).

[0139] When the image processor 15a determines in S262 that the number of filled-in boxes identified by the main process in S102 for the currently targeted mark-sheet area is within the permitted range, the image processor 15a includes in the output information (S263) the selected items identified in the selected item identification process in S126 for the currently targeted mark-sheet area and information for identifying the filled-in boxes corresponding to the selected items. For example, the information for identifying the filled-in boxes may be the coordinates of the center of gravity of the filled-in boxes, the coordinates of two diagonal vertices of the bounding box of the filled-in boxes, or the coordinates of one vertex, the width, and the height of the bounding box of the filled-in boxes.

[0140] When the image processing unit 15a determines in S262 that the number of filled-in frames identified by the main processing of S102 for the currently targeted mark sheet area is not within the permitted range, it determines whether the number of filled-in frames identified by the main processing of S102 for the currently targeted mark sheet area is less than the permitted range (S264).

[0141] When the image processing unit 15a determines in S264 that the number of filled-in frames identified by the main processing of S102 for the currently targeted mark-sheet area is less than the permitted range, it includes in the output information (S265) for the currently targeted mark-sheet area the selected item identified in the selection item identification processing of S126, information for identifying the filled-in frames corresponding to this item, and information that the number of filled-in frames is less than the permitted range.

[0142] When the image processing unit 15a determines in S264 that the number of filled-in boxes identified by the main processing of S102 for the currently targeted mark-sheet area is not less than the permitted range, it includes in the output information for the currently targeted mark-sheet area the selected item identified in the selection item identification processing of S126, information for identifying the filled-in box corresponding to this item, and information that the number of filled-in boxes is greater than the permitted range (S266).

[0143] When the process of S263, S265, or S266 is completed, the image processing unit 15a determines whether or not there is any mark-sheet area among the multiple mark-sheet areas on the mark-sheet 20 that has not yet been targeted in the current output information generation process (S267).

[0144] When the image processor 15a determines in S267 that there is a mark-sheet area that has not yet been targeted in the current output information generating process among the multiple mark-sheet areas on the mark-sheet 20, the image processor 15a executes the process of S261.

[0145] When the image processing unit 15a determines in S267 that there are no mark sheet areas among the multiple mark sheet areas on the mark sheet 20 that have not yet been targeted in this output information generation process, it terminates the output information generation process shown in Figure 23.

[0146] As shown in Fig. 22, when the output information generating process of S243 is completed, the image processing unit 15a generates data in a specific output format based on the output information generated in S243 (S244). For example, when the output information includes multiple overlapping items that can be combined into one, the image processing unit 15a may combine these items into one. Furthermore, when the output information includes multiple items whose order should be changed, the image processing unit 15a may change the order of these items to an appropriate order. Furthermore, the image processing unit 15a may set the file format of the data to be output to a specific format, such as comma-delimited text, tab-delimited text, XML format, or JSON format.

[0147] When the process of S244 is completed, the image processing unit 15a outputs the data generated in S244 to a specific output destination (S245). For example, the image processing unit 15a may write the data generated in S244 to the storage unit 14, or may cause the display unit 12 to display the data based on the data generated in S244.

[0148] When the process of S245 ends, the image processing unit 15a ends the post-processing shown in FIG.

[0149] As shown in FIG. 3, when the post-processing in S103 is completed, the image processing unit 15a ends the operation shown in FIG.

[0150] The image processing unit 15a may fail to identify the selected item. Therefore, a human may check whether the processing contents by the image processing unit 15a are correct. In order to facilitate the checking operation of the checker of the processing contents by the image processing unit 15a, the image processing unit 15a may cut out the part of the painted frame from the mark sheet image or various images generated in the operation shown in FIG. 3 and display it on the display unit 12, or may superimpose an image for highlighting the center of gravity or the bounding box of the painted frame on the mark sheet image or various images generated in the operation shown in FIG. 3 and display it on the display unit 12, or may superimpose an image for highlighting the character string of the item corresponding to the painted frame on the mark sheet image or various images generated in the operation shown in FIG. 3 and display it on the display unit 12.

[0151] FIG. 24 is a diagram showing an example of a mark sheet area image on which an image for making the center of gravity of the filled-in frame stand out is superimposed.

[0152] In the mark sheet area image 30 shown in Fig. 24, an image 50 for making the center of gravity of the painted frame 30b stand out is superimposed on the painted frame 30b. Therefore, a person checking the processing contents by the image processing unit 15a can easily check that the painted frame 30b specified by the image processing unit 15a is correct by looking at the mark sheet area image 30 shown in Fig. 24.

[0153] As described above, the image processing system 10 generates an image 33 (see FIG. 10) as a first image by removing at least elements that are in contact with the boundary of the mark sheet area image 30 (see FIG. 4) from the mark sheet area image 30 (S162 to S163), generates an image 34 (see FIG. 11) as a second image by removing at least a portion 32c (see FIG. 10) that is an element whose area is equal to or less than a specific ratio less than 1 to a portion 32a (see FIG. 10) that is the element with the largest area in the image 33 (S165), and generates an image 34 (see FIG. 11) as a second image by removing the portion 32a that is an element in the image 34 with its inside painted. The boxes 41a, 41b (see FIG. 17), which are elements within the mark sheet area image 30 whose positions in a direction perpendicular to the arrangement direction of the frames 22b are within a specific distance from the unpainted frame 30a, are identified as boxes surrounding the frame 22b (S124). Frames 22b that do not have a corresponding unpainted frame 30a in terms of distance are identified as painted frames 30b (see FIG. 4), which are frames whose insides are painted (S125). This makes it possible to identify frames whose insides are painted more appropriately than before.

[0154] The image processing system 10 identifies the item corresponding to the filled-in frame based on the frame position in the frame arrangement direction of the multiple frames in the mark sheet area and the frame item correspondence list indicating the correspondence between the frame and the item, and the frame position in the frame arrangement direction of the multiple frames in the mark sheet area (S222), thereby enabling the item selected in the mark sheet area to be identified more appropriately than before.

[0155] In this embodiment, when the image processing system 10 determines in S147 that the specified distance in the MeanShift processing in S143 is equal to or greater than the specific distance, it displays an error (S242) and ends the operation shown in Fig. 3. However, when the image processing system 10 determines in S147 that the specified distance in the MeanShift processing in S143 is equal to or greater than the specific distance, it may correct the parameters of the preprocessing in S101 and start over from the preprocessing in S101, rather than displaying an error and ending the operation shown in Fig. 3. Note that when the image processing system 10 determines in S147 that the specified distance in the MeanShift processing in S143 is equal to or greater than the specific distance, it may correct the parameters of the preprocessing in S101 and start over from the preprocessing in S101, and warn that the preprocessing in S101 may be insufficient.

[0156] In this embodiment, when it is determined in S147 that the specified distance in the MeanShift process in S143 is equal to or greater than the specific distance, the image processing system 10 determines that the identification of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed (S149). However, when it is determined in S147 that the specified distance in the MeanShift process in S143 is equal to or greater than the specific distance, the image processing system 10 may not determine that the identification of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed, but may identify the specific coordinates as the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames, and may end the unpainted frame identification process shown in Fig. 6. Here, the image processing system 10 may determine as the specific coordinates any one of the coordinates of the multiple centroids, which have been subjected to the MeanShift process in S145, in the direction perpendicular to the arrangement direction of the frames shown in the mark sheet information 14b for the currently targeted mark sheet area. For example, the image processing system 10 may determine as the specific coordinate the most frequently occurring coordinate among the coordinates of the multiple centroids that have been subjected to the MeanShift process in S145 in the direction perpendicular to the arrangement direction of the boxes shown in the mark sheet information 14b for the current target mark sheet area. Note that, when the image processing system 10 determines in S147 that the specified distance in the MeanShift process in S143 is equal to or greater than the specific distance, it may issue a warning when determining the specific coordinate as the representative value of the coordinates of the unpainted boxes in the direction perpendicular to the arrangement direction of the boxes.

[0157] In this embodiment, the image processing system 10 executes the opening process of S163 after executing the boundary element removal process of S162, and then executes the element coordinate area acquisition process of S164. However, if omitting the opening process of S163 does not have a significant effect on the final processing result, the image processing system 10 may execute the element coordinate area acquisition process of S164 after executing the boundary element removal process of S162 without executing the opening process of S163.

[0158] In this embodiment, the image processing system 10 identifies an item corresponding to a filled-in frame from an image of a first attribute mark sheet area 21 that includes multiple sets of a character string 21a representing an item and a frame 21b that is arranged directly below the character string 21a and is filled in when the item represented by the character string directly above is selected, arranged in the left-right direction. However, the image processing system 10 can also identify an item corresponding to a filled-in frame by a similar method from an image of a mark sheet area that includes multiple sets of a character string representing an item and a frame that is arranged directly above the character string and is filled in when the item represented by the character string directly below is selected, arranged in the left-right direction.

[0159] In this embodiment, the image processing system 10 identifies an item corresponding to a filled-in frame from an image of the second attribute mark sheet area 22, which includes multiple sets of character strings 22a representing an item and frames 22b arranged in the vertical direction, the frames being arranged immediately to the right of the character strings 22a and the inside of which is filled in when the item represented by the character string 22a to the immediately left is selected. However, the image processing system 10 can also identify an item corresponding to a filled-in frame from an image of a mark sheet area, which includes multiple sets of character strings representing an item and frames being arranged immediately to the left of the character strings and the inside of which is filled in when the item represented by the character string to the immediately right is selected, arranged in the vertical direction.

[0160] In this embodiment, the image processing system 10 employs an elliptical frame as the frame of the mark sheet. However, the image processing system 10 can also accommodate frames of shapes other than an ellipse, so long as the frame is a closed frame. For example, the image processing system 10 can accommodate a rectangular frame as the frame of the mark sheet.

[0161] In this embodiment, the image processing system 10 is configured with one computer. However, the image processing system 10 may be configured with multiple computers. [Explanation of symbols]

[0162] 10 Image processing system (computer) 14a Image processing program 20 Mark Sheet 21 First attribute mark sheet area (area) 21a string Frame 21b 22 Second attribute mark sheet area (area) 22a string 22b frame 30 Mark sheet area image (area image) 30a unpainted frame 30b painted frame 32a, 32b, 32c parts (elements) 33 Image (1st image) 34 Image (2nd image) 41a Box Box 41b 41a, 41b, 41c Box (element)

Claims

1. An image processing system for identifying a painted frame, which is a frame with a painted inside, from a region image as an image of the region in an image read by an image reading device from a mark sheet having a region including a character string representing an item and a frame that is arranged in a first direction with respect to the character string and whose inside is painted when the item represented by the character string is selected, the region image being an image of the region, the image processing system comprising: generating a first image by removing from the region image at least elements that are in contact with the boundary of the region image; generating a second image by removing from the first image at least elements whose area is equal to or smaller than a specific ratio (less than 1) of the area of ​​the largest element in the first image; and identifying the element in the second image as an unpainted frame, which is the frame whose inside is not painted; Among the elements in the region image, an element whose position in the first direction is within a specific distance from the unpainted frame is identified as the frame; An image processing system comprising: a processing unit for processing a plurality of frames each having a corresponding unpainted frame;

2. The image processing system according to claim 1, characterized in that the item corresponding to the painted frame is identified based on a list showing the correspondence between the position of the frame in the second direction among the multiple frames in the area and the item, and the position of the painted frame in the second direction among the multiple frames in the area.

3. An image processing program for identifying a painted frame, which is a frame with a painted inside, from an area image as an image of the area in an image read by an image reading device from a mark sheet having an area including a character string representing an item and a frame that is arranged in a first direction with respect to the character string and whose inside is painted when the item represented by the character string is selected, the image processing program comprising: generating a first image by a computer, the first image being at least one region image removed from the region image that is in contact with a boundary of the region image; generating a second image by removing from the first image at least elements whose area is equal to or smaller than a specific ratio (less than 1) of the area of ​​the largest element in the first image; causing the computer to identify an element in the second image as an unpainted frame, the frame having no interior paint; causing the computer to identify, as the frame, an element within the region image whose position in the first direction is within a specific distance from the unpainted frame; An image processing program that causes the computer to identify a frame for which there is no unpainted frame that corresponds in distance as a painted frame.

Images