Image processing system

Abstract

The invention provides an image processing system, and aims to more appropriately determine a frame of which the inner side is coated than in the prior art. The image processing system generates a first image in which at least an element in contact with the boundary of the marking paper region image is removed from the marking paper region image, and generates a second image in which at least an element whose area is less than or equal to a specified ratio of less than 1 with respect to the element having the largest area in the first image is removed from the first image. Among the elements in the mark paper region image, the elements in the second image are identified as uncoated frames which are frames of which the inner sides are not coated, and the elements in the elements in the mark paper region image are identified as frames which are within a specified distance from the uncoated frames at the position in the direction in which the frames are arranged with respect to the character string representing the item. A frame in which there is no corresponding uncoated frame in the distance is determined as a coated frame in which the inner side is coated.

Description

Technical Field

[0001] The present invention relates to an image processing system for determining whether or not a portion of a frame in an image of marked paper is coated. Background Art

[0002] In the past, as an image processing system for determining whether the inside of a frame is coated in an image of a marksheet for selecting an item by coating the inside of the frame, it is known that the inside of the frame is determined to be coated when the difference between the concentration value of the inside of the frame coated and the concentration value of the inside of the frame not coated is greater than a certain concentration value.

[0003] However, in conventional image processing systems, the average concentration of each pixel row in a frame is calculated and its maximum value is used as the concentration value in the frame. Therefore, in cases where, for example, only a thin line extending in a direction perpendicular to the extension direction of the pixel row is drawn in the frame, or the frame is marked with a √ mark or an × mark, if there is a large amount of unpainted portion remaining in the frame, the concentration value in the frame becomes low. As a result, there is a problem that, although the person who paints the frame intends to paint the frame, the image processing system cannot properly determine the frame whose inside is painted. Summary of the invention

[0004] Therefore, an object of the present invention is to provide an image processing system capable of identifying a frame whose inner side is painted more appropriately than before.

[0005] The image processing system of the present invention determines a painted frame from a regional image, wherein the painted frame is a frame with painted inside, and the regional image is an image of the region in an image read by an image reading device from a marking paper having a region, wherein a combination of a character string representing an item in the region and a frame arranged in a first direction relative to the character string and painted inside when the item represented by the character string is selected is arranged in a second direction orthogonal to the first direction. The image processing system generates a first image by removing at least elements touching a boundary of the regional image from the regional image, generates a second image by removing at least elements whose area is less than a determined ratio relative to the largest element in the first image from the first image, determines elements in the second image as unpainted frames, wherein the unpainted frame is the frame with unpainted inside, and the determined ratio is less than 1, and among the elements in the regional image, determines elements whose positions in the first direction are within a determined distance from the unpainted frame as the frames, and determines frames for which there is no corresponding unpainted frame in terms of distance as the painted frames.

[0006] According to this configuration, the image processing system of the present invention generates a first image by removing at least the elements touching the boundary of the regional image from the regional image, generates a second image by removing at least the elements whose areas are less than a certain ratio less than 1 relative to the element with the largest area in the first image from the first image, determines the elements in the second image as frames with no inner coating, i.e., uncoated frames, determines among the elements in the regional image the elements whose positions in a direction orthogonal to the arrangement direction of the frames are within a certain distance from the uncoated frames as frames, determines the frames for which there is no corresponding uncoated frame at the distance as frames with inner coating, i.e., painted frames, thereby being able to determine the inner coating frames more appropriately than before.

[0007] The image processing system of the present invention can also determine the item corresponding to the painted frame based on a list representing the correspondence between the position of the frame in the second direction among the multiple frames in the area and the item, and the position of the painted frame in the second direction among the multiple frames in the area.

[0008] According to this configuration, the image processing system of the present invention determines the item corresponding to the painted frame based on a list representing the correspondence between the position of the frame in the arrangement direction of the multiple frames in the area and the item, and the position of the painted frame in the arrangement direction of the multiple frames in the area, thereby being able to determine the item selected in the area more appropriately than before.

[0009] The image processing system of the present invention can identify the frame whose inside is painted more appropriately than before.

[0010] This specification is presented in a simplified manner, with appropriate reference to the accompanying drawings, to summarize the concepts described in the following detailed description. This specification is not intended to limit the important features and essential features of the subject matter described in the claims, nor is it intended to limit the scope of the subject matter described in the claims. In addition, the subject matter described in the claims is not limited to implementations that solve some or all of the disadvantages described in any part of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram of an image processing system according to one embodiment of the present invention which is composed of one computer.

[0012] Figure 2 Yes means Figure 1 FIG. 4 is a diagram of an example of a marked paper processed by the image processing system shown in FIG.

[0013] Figure 3 This indicates the case where the item corresponding to the painted frame is determined from the marked paper image. Figure 1 Flowchart of the operation of the image processing system shown in FIG.

[0014] Figure 4 Yes means through Figure 3 FIG. 1 is a diagram showing an example of a marked paper area image generated by preprocessing.

[0015] Figure 5 yes Figure 3 A flowchart of the main processing is shown in FIG.

[0016] Figure 6 yes Figure 5 Flowchart of the unpainted frame determination process shown in .

[0017] Figure 7 yes Figure 6 Flowchart of the unpainted frame extraction process shown in .

[0018] Figure 8 Yes means through Figure 7 FIG. 1 is a diagram showing an example of a positive binary image of a marked paper area image generated by the unpainted frame extraction process shown in FIG.

[0019] Fig. 9 It means removing Figure 8 ] is a diagram showing an example of an image in which all white pixels are continuous on the outer periphery of a positive binary image.

[0020] Fig.10 It means for Fig. 9 The image shown in FIG. 1 is a diagram showing an example of an image on which an opening operation (Opening) with a width of 1 pixel has been performed.

[0021] Fig.11 It means for Fig.10 The image shown in FIG. 1 is a diagram showing an example of an image on which the small area element removal process has been performed.

[0022] Fig.12 It is for Fig.11 The image shown in represents a diagram of the center of gravity of each unpainted frame.

[0023] Fig.13 It means it has been corrected Fig.12 The diagram shows the state of the deviation of the center of gravity coordinates.

[0024] Fig.14 yes Figure 5 Flowchart of the frame determination process shown in .

[0025] Fig.15 Yes means through Fig.14 FIG. 4 is a diagram showing an example of a negative binary image generated by the frame determination processing shown in FIG.

[0026] Fig.16 Yes means including Fig.14FIG. 4 is a diagram showing an example of an image of a box generated by the box determination processing shown in FIG.

[0027] Fig.17 Yes means through Fig.14 FIG. 4 is a diagram showing an example of the center of gravity of a box generated by the box determination processing shown in FIG.

[0028] Fig.18 Yes means through Fig.14 0 is a diagram showing an example of the center of gravity of a box corresponding to a box generated by the box determination processing shown in .

[0029] Fig.19 yes Figure 5 Flowchart of the painted frame determination process shown in .

[0030] Fig. 20 It means in Fig.19 ] A diagram showing an example of the center of gravity of a painted frame determined in the painted frame determination process shown in FIG.

[0031] Fig.21 yes Figure 5 Flowchart of the selection item determination process shown in .

[0032] Fig. 22 yes Figure 3 The flowchart of post-processing is shown in FIG.

[0033] Fig.23 yes Fig. 22 Flowchart of the output information generation process shown in .

[0034] Fig.24 It is used to make Fig.19 ] A diagram showing an example of a marked paper area image after superimposing an image in which the center of gravity of a painted frame determined by the painted frame determination processing is highlighted. DETAILED DESCRIPTION

[0035] Hereinafter, one embodiment of the present invention will be described with reference to the drawings.

[0036] First, the configuration of the image processing system according to the present embodiment will be described.

[0037] Figure 1 This is a block diagram of an image processing system 10 according to the present embodiment which is composed of one computer.

[0038] like Figure 1As shown, the image processing system 10 includes: an operation unit 11, which is an operation device such as a keyboard and a mouse for inputting various operations; a display unit 12, which is a display device such as an LCD (liquid crystal display) for displaying various information; a communication unit 13, which is a communication device that communicates with an external device directly through a network such as a LAN (local area network) and the Internet or through a wired or wireless connection without a network; a storage unit 14, which is a non-volatile storage device such as a semiconductor memory and an HDD (hard disk drive) for storing various information; and a control unit 15, which controls the entire image processing system 10. The image processing system 10 can be composed of a PC (personal computer), for example. The image processing system 10 can also be configured on the cloud.

[0039] Figure 2 FIG. 1 is a diagram showing an example of marked paper processed by the image processing system 10 .

[0040] exist Figure 2 In the marking paper 20 shown in the figure, the area for selecting items by applying within the frame (hereinafter referred to as the "marking paper area") includes a plurality of first attribute marking paper areas 21 for selecting items of the first attribute and a second attribute marking paper area 22 for selecting items of the second attribute.

[0041] The first attribute marking paper area 21 includes a plurality of combinations of character strings 21a and frames 21b arranged in the left-right direction, the character strings 21a representing items of the first attribute, and the frames 21b are arranged directly below the character strings 21a, and the inner side of the frames 21b is coated when the item represented by the character string 21a directly above is selected. The first attribute marking paper area 21 includes "ABC", "DEF" and "GHI" as items of the first attribute.

[0042] The second attribute marking paper area 22 includes a plurality of combinations of character strings 22a and frames 22b arranged in the vertical direction, the character string 22a indicating items of the second attribute, and the frame 22b being arranged immediately to the right of the character string 22a, and the inner side of the frame 22b being painted when the item indicated by the character string 22a immediately to the left is selected. The second attribute marking paper area 22 includes "abcdef", "ghijkl", "mnopqr", and "stuvwx" as items of the second attribute.

[0043] like Figure 1As shown, the storage unit 14 can store an image processing program 14a for executing a certain process on an input image and outputting the execution result. The image processing program 14a can be installed in the image processing system 10 at the manufacturing stage of the image processing system 10, or can be additionally installed in the image processing system 10 from an external storage medium such as a USB (Universal Serial Bus) memory, or can be additionally installed in the image processing system 10 from a network.

[0044] The storage unit 14 can store the marked paper information 14b which indicates the marked paper 20 for each marked paper area (see Figure 2 ), the arrangement direction of the boxes in each marking paper area, a list indicating which box in each marking paper area corresponds to which item (hereinafter referred to as the "box item correspondence list"), and the range of the number of boxes that are allowed to be painted on the inside in each marking paper area (hereinafter referred to as the "painted boxes").

[0045] The marked paper area is the first attribute marked paper area 21 (refer to Figure 2 ), the arrangement direction of the frames in the marking paper area is the left-right direction, and the marking paper area is the second attribute marking paper area 22 (refer to Figure 2 ), the arrangement direction of the boxes in the marking paper area is the up and down direction.

[0046] The frame item correspondence list is a list indicating the correspondence between the positions of the frames in the arrangement direction of the frames in the plurality of frames in the marked paper area and the items. When the marked paper area is the marked paper area 21 for the first attribute, the frame item correspondence list is a list in which the first frame from the left corresponds to the item of "ABC", the second frame from the left corresponds to the item of "DEF", and the third frame from the left corresponds to the item of "GHI". When the marked paper area is the marked paper area 22 for the second attribute, the frame item correspondence list is a list in which the first frame from the top corresponds to the item of "abcdef", the second frame from the top corresponds to the item of "ghijkl", the third frame from the top corresponds to the item of "mnopqr", and the fourth frame from the top corresponds to the item of "stuvwx".

[0047] The control unit 15 includes, for example, a CPU (central processing unit), a ROM (read only memory) storing programs and various data, and a RAM (random access memory) as a memory used as a work area of ​​the CPU of the control unit 15. The CPU of the control unit 15 executes the program stored in the storage unit 14 or the ROM of the control unit 15.

[0048] The control unit 15 executes the image processing program 14 a to realize an image processing unit 15 a that performs a predetermined process on an input image and outputs the result of the process.

[0049] As input methods for inputting images into the image processing system 10, there are, for example: a method in which, when image data is input into a certain folder that is monitored by the image processing system 10, the image processing system 10 obtains the image data from the folder; a method in which image data read from an original by an image reading device such as an MFP (multifunction printer) or a dedicated scanner is sent from the image reading device to the image processing system 10; a method in which, when an email containing image data is sent to an email address that is monitored by the image processing system 10, the image processing system 10 obtains the image data from the email; a method in which image data is sent to the image processing system 10 from a web browser of a computer such as a PC to a web site provided to the image processing system 10; and a method in which image data is sent to the image processing system 10 from a dedicated application installed on a mobile device such as a smartphone.

[0050] As a process of specifying an image performed by the image processing system 10 , for example, there is a process of specifying an item corresponding to a painted frame from an image read from marked paper by an image reading device (hereinafter referred to as “marked paper image”).

[0051] Methods for transmitting the execution result of the image processing system 10 include, for example, a method of inputting the execution result to a specified cloud service, a method of inputting the execution result to a specified folder, and a method of transmitting an email including the execution result to a specified email address.

[0052] Next, the operation of the image processing system 10 when specifying an item corresponding to a painted frame from a marked paper image will be described.

[0053] The marked paper image of the object for which the image processing system 10 specifies the item corresponding to the filled frame is input to the image processing system 10 by, for example, any of the input methods described above.

[0054] Next, the image processing system 10 determines the marked paper image of the object corresponding to the painted frame, and obtains the marked paper image from the marked paper 20 (refer to Figure 2 ) is used as an example for explanation. The marked paper image read from the marked paper 20 by the image reading device may be any one of a color image and a black and white image.

[0055] Figure 3 This is a flowchart of the operation of the image processing system 10 when specifying an item corresponding to a painted frame from a marked paper image.

[0056] If the marked paper image is input to the image processing system 10, the image processing unit 15a executes Figure 3 The actions shown in .

[0057] like Figure 3 As shown, the image processing unit 15a performs various preprocessing (S101) on the marked paper image input to the image processing system 10 (hereinafter referred to as "input marked paper image") for improving the accuracy of identifying the items corresponding to the painted frame from the marked paper image.

[0058] In S101, the image processing unit 15a performs, as needed: "noise removal" for removing noise from the input marked paper image that may hinder the determination of the item corresponding to the painted frame from the marked paper image; "alpha channel removal" for removing the alpha channel from the input marked paper image and then rasterizing it when the input marked paper image contains an alpha channel; "grayscale conversion (decolorization)" for decolorizing the input marked paper image when the input marked paper image is a color image; "brightness correction" for correcting the range and distribution of the brightness of the input marked paper image when the input marked paper image is a multi-valued image input and the overall brightness range of the input marked paper image is too narrow or too biased; and "light and dark correction" for correcting the input marked paper image if it is slightly tilted, rotated, or deformed. The shape of the paper image is corrected into a rectangular shape captured from the front through "frontalization and leveling (rotation, deformation) correction"; the blur of the input marked paper image is corrected through "blur correction"; the image of the marked paper area (hereinafter referred to as "marked paper area image") is extracted from the page of the input marked paper image through "retrieval and extraction processing of the object area (marked paper area block)"; even when scanning the same kind of marked paper, the pixel size of the image may differ by several percent relative to the width of the original due to differences in the printing width of the printer used when printing the marked paper, the settings of the printer driver, or the scanning width of the scanner, the settings of the scanner driver, etc., so the image is corrected and standardized, unified, or the offset of the variation is suppressed to suppress the difference, etc., as preprocessing.

[0059] For example, if the input marked paper image is an RGB 3ch image, the “grayscale conversion (achromatic conversion)” can be calculated using Y=0.3R+0.5G+0.1B.

[0060] "Retrieval and extraction processing of object area (marked paper area block)" can, for example, find a square area and extract it as a marked paper area image. When the range of the marked paper area image is known in advance in the marked paper information 14b, the marked paper area image is extracted through a mask.

[0061] In the preprocessing of S101, processing other than "retrieval and extraction processing of object area (marked paper area block)" can be performed with the page size of the input marked paper image before the marked paper area image is cut out from the input marked paper image through "retrieval and extraction processing of object area (marked paper area block)", or can be performed on the marked paper area image after the marked paper area image is cut out from the input marked paper image through "retrieval and extraction processing of object area (marked paper area block)".

[0062] Figure 4 This is a diagram showing an example of a marked paper area image generated by the pre-processing of S101.

[0063] Figure 4 The marked paper area image 30 shown in FIG. Figure 2 1 and 2 show an image of the uppermost second attribute marking paper area 22 among the plurality of second attribute marking paper areas 22 in the marking paper 20. FIG. Figure 4 The marked paper area image 30 shown in FIG. 1 has four character strings 22a and four frames 22b. Among the four frames 22b, the first, third, and fourth frames 22b from the top are frames whose inner sides are not coated (hereinafter referred to as "uncoated frames") 30a. Among the four frames 22b, the second frame 22b from the top is a coated frame 30b.

[0064] The image processing unit 15a performs the pre-processing of S101 and Figure 4 The marking paper area image shown in the marking paper 20 also generates not only the image of the topmost second attribute marking paper area 22 among the multiple second attribute marking paper areas 22 in the marking paper 20 through the preprocessing of S101, but also generates respective images of the second attribute marking paper areas 22 other than the topmost second attribute marking paper area 22 among the multiple second attribute marking paper areas 22 in the marking paper 20, and respective images of the multiple first attribute marking paper areas 21 in the marking paper 20.

[0065] For convenience of explanation, an image in which a background is drawn in substantially white and characters and a frame are drawn in substantially black is hereinafter referred to as a positive image.

[0066] like Figure 3 As shown, when the pre-processing of S101 is completed, the image processing unit 15a executes the main processing which is the main part of the processing of determining the item corresponding to the painted frame from the marked paper image (S102).

[0067] Figure 5 yes Figure 3 A flowchart of the main processing is shown in FIG.

[0068] like Figure 5As shown, the image processing unit 15a selects a target area from a plurality of marking paper areas in the marking paper 20 (S121). The target area is one of the plurality of marking paper areas. The image processing unit 15a selects one of the areas that has not been processed in the main body processing this time as the target area.

[0069] When the process of S121 is completed, the image processing unit 15a executes an unpainted frame determination process for determining an unpainted frame in a direction orthogonal to the arrangement direction of the frames for the marked paper area image of the current target area (S122).

[0070] Figure 6 yes Figure 5 Flowchart of the unpainted frame determination process shown in .

[0071] like Figure 6 As shown, the image processing unit 15a performs an unpainted frame extraction process (S141) for generating an image of only an unpainted frame from the marked paper area image.

[0072] Figure 7 yes Figure 6 Flowchart of the unpainted frame extraction process shown in .

[0073] like Figure 7 As shown, the image processing unit 15a generates a positive image (hereinafter referred to as a "positive binary image") after binarization of the marked paper area image (S161). In the case where the marked paper area image generated by the preprocessing of S101 is a black and white multi-value image, the image processing unit 15a can use, for example, the Otsu model to find a threshold value and perform binarization in S161. In the case where the marked paper area image generated by the preprocessing of S101 has been binarized and has become a binary image with a white background and a black foreground, i.e., a character string and a frame, the image processing unit 15a does not need to perform any special processing in S161. In the case where the marked paper area image generated by the preprocessing of S101 has been binarized and has become a binary image with a black background and a black foreground, i.e., a character string and a frame, the image processing unit 15a performs a negative-positive inversion process in S161 to make the background white, thereby generating a positive binary image with a white background.

[0074] Figure 8 Yes means through Figure 7 FIG. 1 is a diagram showing an example of a positive binary image of a marked paper area image generated by the unpainted frame extraction process shown in FIG.

[0075] Figure 8 The positive binary image 31 shown in Figure 4 A positive binary image of the marked paper area image 30 is shown in FIG.

[0076] like Figure 7As shown, if the processing of S161 is completed, the image processing unit 15a performs a boundary element removal process (S162) to remove elements that touch the boundary of the positive binary image generated in S161. Specifically, the image processing unit 15a searches for pixels whose adjacent pixel values ​​are white (hereinafter referred to as "white pixels") among the pixels on the outer periphery of the positive binary image generated in S161. When the adjacent white pixels are found, the image processing unit 15a further searches for white pixels adjacent to the found white pixels. Thereafter, the image processing unit 15a repeats such a process until no adjacent white pixels are found. In addition, as a method for finding adjacent white pixels, any one of the 4-connection method of searching in a total of 4 directions of the horizontal direction and the vertical direction of the pixel of interest and the 8-connection method of searching in an oblique direction in addition to the horizontal and vertical directions can be used. By doing the above, if the image processing unit 15a extracts all the white pixels continuous from the outer periphery of the positive binary image of the marked paper area image, then all the white pixels continuous from the outer periphery of the positive binary image are removed, that is, replaced with black pixels.

[0077] Fig. 9 Yes means Figure 8 ] is a diagram showing an example of an image in which all white pixels that are continuous from the outer peripheral white pixels are removed from the positive binary image.

[0078] exist Fig. 9 In the image 32 shown in FIG. Figure 8 In the positive binary image 31 shown in FIG. 1 , a portion 32 a inside the frame 22 b of the portion where the frame 30 a is not painted remains in white.

[0079] In the case where the inside of the painted frame 30b in the marking paper area is fully painted and there is no unpainted part, the inside of the frame is blackened at the time of binarization in S161, so if the background is removed by the processing of S162, that is, replaced with black, the painted frame 30b and the inside of the painted frame 30b are assimilated with the background and disappear in appearance. However, in the case where there is an unpainted part inside the painted frame 30b in the marking paper area, after the processing of S162, as shown in FIG. Fig. 9 As shown, the unpainted portion remains in white. Fig. 9 As shown, the areas of the plurality of unpainted portions are each much smaller than the area of ​​the portion 32a inside the unpainted frame 30a.

[0080] In addition, if Fig. 9 As shown, in the part of the character string 22a of the item name paired with the frame 22b, the part 32c inside the closed space within each character of the character string 22a remains in white. Fig. 9As shown, the area of ​​each of the plurality of portions 32c is much smaller than the area of ​​the portion 32a on the inner side of the unpainted frame 30a.

[0081] like Figure 7 As shown, if the boundary element removal process of S162 is completed, the image processing unit 15a performs an opening operation process (S163) for the image that has been subjected to the boundary element removal process in S162, which is followed by an expansion process. The opening operation process can remove tiny elements and extremely fine elements in the image. For example, when the resolution of the marked paper area image is 300 dpi and the size of the text in the marked paper area image is 10 pounds, for example, an opening operation process of 1 pixel width to 2 pixel width is sufficient.

[0082] Fig.10 It means for Fig. 9 The image shown in FIG. 1 is a diagram showing an example of an image on which an opening operation of 1 pixel width is performed.

[0083] Fig.10 The image 33 shown in FIG. 3 has been removed. Fig. 9 The tiny elements and extremely fine elements in the image 32 shown in FIG. Fig.10 The image 33 shown in FIG. 1 has the painted frame 30b removed (see FIG. 1 ). Figure 8 ) all unpainted parts on the inside. Fig.10 The image 33 shown in FIG. 1 has the character string 22a removed (see Figure 8 ) is a part of portion 32c inside the closed space within each character.

[0084] like Figure 7 As shown, if the opening operation processing of S163 is completed, the image processing unit 15a performs coordinate and area acquisition processing (S164) for the image that has completed the opening operation processing in S163. The coordinate and area acquisition processing is a process for obtaining the coordinates and area of ​​the blocks of independent island-shaped white pixels, that is, the elements in the image. Specifically, the image processing unit 15a focuses on the white pixels in the image, and after extracting the coordinate positions of the white pixels, determines the continuity with the pixels adjacent to the white pixels whose coordinate positions are extracted, and extracts the blocks of white pixels. In addition, the image processing unit 15a calculates the area of ​​each block of white pixels based on the number of pixels constituting the block.

[0085] If the element coordinate area acquisition process of S164 is completed, the image processing unit 15a performs a small-area element removal process (S165) for removing elements with small areas on the image that has completed the opening operation process in S163. Specifically, the image processing unit 15a determines the largest area among the areas of the elements in the image, and removes all elements whose areas are less than a certain ratio of less than 1 relative to the largest area from the image that has completed the opening operation process in S163. The determined ratio in the small-area element removal process can be, for example, 0.8. The "removal" mentioned here is essentially a process in which the white part of the part is coated with black so that it is assimilated with the black background, thereby making its existence unknown in appearance. In image processing, it is equivalent to replacing the pixel value of each element from the color value of white to the color value of black (background color).

[0086] Fig.11 It means for Fig.10 The image shown in FIG. 1 is a diagram showing an example of an image on which the small area element removal process has been performed.

[0087] Fig.11 The image 34 shown in FIG. 34 has been removed. Fig.10 The elements with small area in the image 33 shown in FIG. Fig.11 The image 34 shown in FIG. 1 has the character string 22a removed (see Figure 8 )'s entire inner portion 32c of the enclosed space within each character.

[0088] exist Fig.10 It is obvious that the element with the largest area in the image 33 shown in FIG. 3 is the marked paper area image 30 (see FIG. Figure 4 ) in the unpainted frame 30a (refer to Figure 4 ). As long as there is no extreme aberration or deformation in the marked paper area image 30, the areas of the parts 32a inside the unpainted frame 30a in the marked paper area image 30 are roughly the same regardless of how many parts 32a inside the unpainted frame 30a in the marked paper area image 30 exist in the marked paper area image 30. Therefore, even if a relatively high value is set as the determined ratio in the small area element removal process in S165, the parts 32a inside the unpainted frame 30a in the marked paper area image 30 will not be removed, and as a result, the image processing unit 15a will not omit the parts 32a inside the unpainted frame 30a.

[0089] Since a relatively high numerical value can be set as the determined ratio in the small area element removal process of S165, elements other than the inner part 32a of the unpainted frame 30a in the marked paper area image 30 are removed. As a result, there is almost no possibility that the image processing unit 15a mistakenly extracts elements other than the inner part 32a of the unpainted frame 30a as the inner part 32a of the unpainted frame 30a.

[0090] For example, there is the following situation: Figure 4 ) and is arranged near the frame 22b (refer to Figure 4 ), such as Fig.10 As shown in FIG. 1 , there are relatively small residual elements after the execution of the opening operation processing of S163. However, the area of ​​such relatively small residual elements is much smaller than the area of ​​the inner part 32a of the unpainted frame 30a, so as shown in FIG. Fig.11 As shown, the small-area element removal process in S165 is removed, that is, replaced with black, which is the same color as the background.

[0091] In addition, even in the painted frame 30b (refer to Figure 4 ) part, if the coating is not done carefully and there are uncoated parts, such as Fig.10 As shown, after the execution of the opening operation processing of S163, there are sometimes small residual elements. However, the area of ​​such small residual elements is much smaller than the area of ​​the inner part 32a of the unpainted frame 30a, so they are removed by the small area element removal processing of S165, that is, replaced with the same black color as the background color.

[0092] Therefore, in the image 34 after the small area element removal process of S165 is executed, Fig.11 As shown, only the inner portion 32a of the frame 30a remains unpainted.

[0093] like Figure 7 As shown in FIG. 1 , if the small area element removal process of S165 is completed, the image processing unit 15a ends. Figure 7 The unpainted frame extraction process shown in FIG.

[0094] like Figure 6 As shown, if the unpainted frame extraction process of S141 is completed, the image processing unit 15a calculates the centroid coordinates of each element, i.e., each unpainted frame, for the image that has become a state of only unpainted frames through the unpainted frame extraction process of S141 (S142). For example, the image processing unit 15a can extract the coordinates of white pixels for the image that has become a state of only unpainted frames through the unpainted frame extraction process of S141, determine whether adjacent pixels in color value are connected with the same value, extract blocks of white pixels that have become islands, i.e., elements, and for each extracted block, calculate the added average of the x coordinates and the added average of the y coordinates of all pixels constituting the block, and use them as the x coordinate and y coordinate of the centroid, respectively.

[0095] Fig.12 It is for Fig.11 The image 34 shown in FIG. 34 shows the unpainted frame 30a (see FIG. Figure 4 ) of their respective center of gravity 34a.

[0096] exist Fig.12 In the figure, the x coordinates of the centers of gravity 34a of the uncoated frames 30a are different.

[0097] like Figure 6 As shown, when the process of S142 is completed, the image processing unit 15a performs mean shift processing on the x-coordinate and the y-coordinate for all the centroids whose coordinates are calculated in S142 (S143).

[0098] The mean shift processing in S143 refers to a process of replacing each element in the sequence list with the average value of other elements within a predetermined distance in the sequence list.

[0099] For example, when the mean shift process is performed on a sequence list such as {1, 10, 0, 0, 2, 5, 6, 7, 6, 2, 1} with a predetermined distance of 2, for the element "0", there are elements "0", "1", "1", "2", and "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "1" which is the average value of these elements. For the element "1", there are elements "0", "0", "1", "2", and "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "1" which is the average value of these elements. For the element "2", there are elements "0", "0", "1", "1", and "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "1" which is the average value of these elements. For the element "5", there are elements "6", "6", and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. For the element "6", there are elements "5", "6" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. For the element "7", there are elements "5", "6" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. Therefore, if the mean shift process is performed on the sequence list of {1, 10, 0, 0, 2, 5, 6, 7, 6, 2, 1} with 2 as the predetermined distance, it becomes a sequence list of {1, 10, 1, 1, 1, 6, 6, 6, 1, 1}.

[0100] When the mean shift process is performed on a sequence list such as {1, 10, 0, 0, 2, 5, 6, 7, 6, 2, 1} with 1 as the predetermined distance, for the element "0", there are elements "0", "1", and "1" as other elements within the predetermined distance in the sequence list, so it is replaced with "1 / 2" which is the average value of these elements. For the element "1", there are elements "0", "0", "1", "2", and "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "1" which is the average value of these elements. For the element "2", there are elements "1", "1", and "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "3 / 2" which is the average value of these elements. For the element "5", there are elements "6" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "17 / 3" which is the average value of these elements. For the element "6", there are elements "5", "6" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. For the element "7", there are elements "6" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "19 / 3" which is the average value of these elements. Therefore, if the mean shift process is performed on the sequence list of {1, 10, 0, 0, 2, 5, 6, 7, 6, 2, 1} with 1 as the predetermined distance, it becomes a sequence list of {1, 10, 1 / 2, 1 / 2, 3 / 2, 17 / 3, 6, 19 / 3, 6, 3 / 2, 1}.

[0101] When the mean shift process is performed on a sequence list such as {1, 2, 3, 4, 5, 6, 7} with 1 as the predetermined distance, for element "1", there is element "2" as other elements within the predetermined distance in the sequence list, so it is replaced with "3 / 2" which is the average value of these elements. For element "2", there are elements "1" and "3" as other elements within the predetermined distance in the sequence list, so it is replaced with "2" which is the average value of these elements. For element "3", there are elements "2" and "4" as other elements within the predetermined distance in the sequence list, so it is replaced with "3" which is the average value of these elements. For element "4", there are elements "3" and "5" as other elements within the predetermined distance in the sequence list, so it is replaced with "4" which is the average value of these elements. For element "5", there are elements "4" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "5" which is the average value of these elements. For the element "6", there are elements "5" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. For the element "7", there is element "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "13 / 2" which is the average value of these elements. Therefore, if the mean shift process is performed on the sequence list of {1, 2, 3, 4, 5, 6, 7} with 1 as the predetermined distance, it becomes a sequence list of {3 / 2, 2, 3, 4, 5, 6, 13 / 2}.

[0102] When the mean shift process is performed on a sequence list such as {1, 2, 3, 4, 5, 6, 7} with a predetermined distance of 2, for element "1", there are elements "2" and "3" as other elements within the predetermined distance in the sequence list, so it is replaced with "2" which is the average value of these elements. For element "2", there are elements "1", "3" and "4" as other elements within the predetermined distance in the sequence list, so it is replaced with "5 / 2" which is the average value of these elements. For element "3", there are elements "1", "2", "4" and "5" as other elements within the predetermined distance in the sequence list, so it is replaced with "3" which is the average value of these elements. For element "4", there are elements "2", "3", "5" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "4" which is the average value of these elements. For element "5", there are elements "3", "4", "6" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "5" which is the average value of these elements. For the element "6", there are elements "4", "5" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "11 / 2" which is the average value of these elements. For the element "7", there are elements "5" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "6" which is the average value of these elements. Therefore, if the mean shift process is performed on the sequence list of {1, 2, 3, 4, 5, 6, 7} with 2 as the predetermined distance, it becomes a sequence list of {2, 5 / 2, 3, 4, 5, 11 / 2, 6}.

[0103] When the mean shift process is performed on a sequence list such as {1, 2, 3, 4, 5, 6, 7} with a predetermined distance of 3, for the element "1", there are elements "2", "3" and "4" as other elements within the predetermined distance in the sequence list, so it is replaced with "5 / 2" which is the average value of these elements. For the element "2", there are elements "1", "3", "4" and "5" as other elements within the predetermined distance in the sequence list, so it is replaced with "3" which is the average value of these elements. For the element "3", there are elements "1", "2", "4", "5" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "7 / 2" which is the average value of these elements. For the element "4", there are elements "1", "2", "3", "5", "6" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "4" which is the average value of these elements. For the element "5", there are elements "2", "3", "4", "6" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "9 / 2" which is the average value of these elements. For the element "6", there are elements "3", "4", "5" and "7" as other elements within the predetermined distance in the sequence list, so it is replaced with "5" which is the average value of these elements. For the element "7", there are elements "4", "5" and "6" as other elements within the predetermined distance in the sequence list, so it is replaced with "11 / 2" which is the average value of these elements. Therefore, if the mean shift processing is performed on the sequence list of {1, 2, 3, 4, 5, 6, 7} with 3 as the predetermined distance, it becomes a sequence list of {5 / 2, 3, 7 / 2, 4, 9 / 2, 5, 11 / 2}.

[0104] Mean shift processing is a process for replacing each element in the sequence list with the average value of other elements within a predetermined distance in the sequence list, thereby replacing nearby elements in the sequence list with the same value. Therefore, mean shift processing has the effect of absorbing and suppressing deviations in the sequence list. By repeating the mean shift processing multiple times, the elements in the sequence list converge to a constant value.

[0105] If the residual amount of deformation correction, leveling, and vertical adjustment in the preprocessing of S101 is known, then the predetermined distance in the mean shift process, i.e., the correction distance, can be set to a distance slightly larger than the residual amount. For example, when the resolution of the marked paper area image is 300 dpi and the size of the text in the marked paper area image is 10 points, for example, a distance of about 5 pixels can be set as the initial value of the correction distance in the mean shift process.

[0106] Fig.13 Yes means Fig.12 2 is a diagram showing a state in which the deviation of the coordinates of the center of gravity 34a is corrected.

[0107] exist Fig.13 In the whole unpainted frame 30a (refer to Figure 4 )’s x-coordinate is consistent with the center of gravity 34a.

[0108] like Figure 6 As shown in FIG. 14, if the processing of S143 is completed, the image processing unit 15a determines whether the x-coordinates and y-coordinates of the plurality of centroids after the mean shift processing in S143 are consistent with the coordinates in the direction orthogonal to the arrangement direction of the frames indicated by the marking paper information 14b for the current object area (S144). For example, when the current object area is the first attribute marking paper area 21 (refer to Figure 2 ), the image processing unit 15a determines in S144 whether the x-coordinates and y-coordinates of the plurality of centroids after the mean shift processing in S143 are consistent with the y-coordinates in the vertical direction orthogonal to the arrangement direction of the frame indicated by the marked paper information 14b for the first attribute marked paper area 21, i.e., the left-right direction. In addition, when the current target area is the second attribute marked paper area 22 (refer to Figure 2 ), the image processing unit 15a determines in S144 whether the x-coordinates and y-coordinates of the multiple centroids after the mean shift processing is implemented in S143 are consistent with the x-coordinates in the left-right direction orthogonal to the arrangement direction of the box shown in the second attribute marking paper area 22 by the marking paper information 14b, i.e., the up-and-down direction.

[0109] If the image processing unit 15a determines in S144 that the x-coordinates and y-coordinates of the multiple centroids after the mean shift processing is performed in S143 are inconsistent with the coordinates in the direction orthogonal to the arrangement direction of the boxes shown by the marked paper information 14b for the current object area, it will again perform mean shift processing on the x-coordinates and y-coordinates of all the centroids after the mean shift processing is performed in S143 (S145).

[0110] When the process of S145 is completed, the image processing unit 15a determines whether the x-coordinates and y-coordinates of the plurality of centroids after the mean shift process in S145 are consistent with the arrangement direction of the frames indicated by the marked paper information 14b for the current target area (S146).

[0111] If the image processing unit 15a determines in S146 that the x-coordinates and y-coordinates of the plurality of centroids after the mean shift processing in S145 do not match the coordinates in the direction perpendicular to the arrangement direction of the frames indicated by the marked paper information 14b for the current object area, it determines whether the predetermined distance in the most recent mean shift processing is greater than a determined distance (S147). For example, the determined distance in S147 may be a distance of 1 / 2 of the width of the frame in the direction perpendicular to the arrangement direction of the frames indicated by the marked paper information 14b for the current object area.

[0112] If the image processing unit 15a determines in S147 that the predetermined distance in the most recent mean shift process is not greater than the predetermined distance, it increases the predetermined distance in the mean shift process from the current value by the predetermined value (S148) and executes the process of S145.

[0113] If the image processing unit 15a determines in step S147 that the predetermined distance in the most recent mean shift processing is greater than the determined distance, it determines that the determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frames has failed (step S149), and ends the process. Figure 6 The unpainted frame determination process shown in FIG.

[0114] If the image processing unit 15a determines in S144 that the x-coordinates and y-coordinates of the multiple centroids after the mean shift processing in S143 are consistent with the coordinates in the direction orthogonal to the arrangement direction of the frames indicated by the marking paper information 14b for the current object area, then the coordinates in the direction orthogonal to the arrangement direction of the frames indicated by the marking paper information 14b for the current object area among the x-coordinates and y-coordinates of the multiple centroids whose deviations have been corrected in S143 are determined as representative values ​​of the coordinates of the unpainted frames in the direction orthogonal to the arrangement direction of the frames (S150), and the process ends. Figure 6 The unpainted frame determination process shown in FIG.

[0115] Similarly, if the image processing unit 15a determines in S146 that the x-coordinates and y-coordinates of the multiple centroids after the mean shift processing in S145 are consistent with the coordinates in the direction orthogonal to the arrangement direction of the frames indicated by the marking paper information 14b for the current object area, then the coordinates in the direction orthogonal to the arrangement direction of the frames indicated by the marking paper information 14b for the current object area among the x-coordinates and y-coordinates of the multiple centroids whose deviations have been corrected in S145 are determined as representative values ​​of the coordinates of the unpainted frames in the direction orthogonal to the arrangement direction of the frames (S150), and the process ends. Figure 6 The unpainted frame determination process shown in FIG.

[0116] like Figure 5As shown, when the unpainted frame determination process of S122 is completed, the image processing unit 15a determines whether determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frames has failed (S123).

[0117] If the image processing unit 15a determines in S123 that the determination of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed, the process ends. Figure 5 The main processing shown in FIG.

[0118] If the image processing unit 15a determines in S123 that the determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frames has failed, it executes frame determination processing (S124) for the marked paper area image of the current object area.

[0119] Fig.14 yes Figure 5 Flowchart of the frame determination process shown in .

[0120] like Fig.14 As shown, the image processing unit 15a generates a signal that is generated in S161 (refer to Figure 7 ) is a positive-positive inverted image of the positive binary image generated in (hereinafter referred to as a “negative binary image”) (S181).

[0121] Fig.15 Yes means through Fig.14 FIG. 4 is a diagram showing an example of a negative binary image generated by the frame determination processing shown in FIG.

[0122] Fig.15 The negative binary image 40 shown in FIG. Figure 8 The positive binary image 31 shown in FIG. 1 is an image after positive-negative inversion.

[0123] like Fig.14 As shown, the image processing unit 15a performs a bounding box process (S182) on the negative binary image generated in S181. The bounding box process is a process of surrounding the elements in the image with a bounding box. In addition, the image processing unit 15a generates a filled box (S183) by filling the inside of the bounding box generated in S182 with white.

[0124] Fig.16 Yes means including Fig.14 FIG. 1 is a diagram showing an example of an image filled with squares generated by the frame determination processing shown in FIG.

[0125] Fig.16 The image 41 shown in FIG. Fig.15The filled boxes correspond to the boundary boxes respectively surrounded by the unpainted box 30a, the painted box 30b, and all or part of each character of the character string 22a, which are the elements in the image of the negative binary image 40 shown in FIG. Fig.16 Filled boxes 41a, 41b, and 41c are depicted in FIG. Filled box 41a corresponds to a boundary box surrounding the unfilled box 30a. Filled box 41b corresponds to a boundary box surrounding the filled box 30b. Filled box 41c corresponds to a boundary box surrounding all or part of each character of the character string 22a.

[0126] like Fig.14 As shown, if the processing of S183 is completed, the image processing unit 15a calculates the centroid coordinates of the filled box generated in S183 (S184). For example, the image processing unit 15a extracts the coordinates of white pixels for the image of the filled box generated in S183, determines whether adjacent pixels are connected with the same value in color value, extracts blocks of white pixels that become islands, i.e., elements, and calculates the average of the x coordinates and the average of the y coordinates of all pixels constituting the block for each extracted block, as the x coordinate and the y coordinate of the centroid, respectively. In addition, when the bounding box generated in S182 is defined as a rectangle by the coordinates of the two vertices of the diagonal corners of the bounding box, the image processing unit 15a can calculate the centroid coordinates by averaging the coordinates of the two vertices of the diagonal corners of the bounding box. In addition, when the bounding box generated in S182 is defined by the coordinates, width, and height of a vertex of the bounding box, the image processing unit 15a may use as the centroid coordinates the coordinates of a vertex of the bounding box that are offset in the direction of the width of the bounding box by half the width of the bounding box and in the direction of the height of the bounding box by half the height of the bounding box.

[0127] Fig.17 Yes means through Fig.14 2 is a diagram showing an example of the center of gravity of filled-in frames 41a, 41b, and 41c generated by the frame determination processing shown in FIG.

[0128] Fig.17 The center of gravity 42a shown in Fig.16 The center of gravity of the filled box 41a is shown in FIG. Fig.17 The center of gravity 42b shown in Fig.16 The center of gravity of the filled box 41b is shown in FIG. Fig.17 The center of gravity 42c shown in Fig.16 The center of gravity of the filled box 41c is shown in FIG.

[0129] like Fig.14As shown, if the processing of S184 is completed, the image processing unit 15a will mark the coordinates of the direction orthogonal to the arrangement direction of the frame shown by the paper information 14b for the current object area in the center of gravity of the coordinates calculated in S184 and the coordinates of the direction orthogonal to the arrangement direction of the frame shown in S150 (refer to Figure 6 ) is determined as the center of gravity within the determined distance as the center of gravity of the frame (S185). For example, the determined distance in S185 may be a distance of 1 / 2 of the width of the frame in the direction perpendicular to the arrangement direction of the frame indicated by the marked paper information 14b for the current object area.

[0130] Fig.18 Yes means through Fig.14 FIG. 4 is a diagram showing an example of the center of gravity of a filled-in box corresponding to a box generated by the box determination processing shown in FIG.

[0131] Fig.18 The center of gravity shown in Fig.17 The center of gravity 42a to 42c shown in FIG. 2 and the frame 22b (see Fig.15 ) corresponds to the centroid 42a, 42b of the filled boxes 41a, 41b. Fig.18 The center of gravity shown does not include Fig.17 The centroids 42a to 42c shown in FIG. Fig.15 ) corresponds to the center of gravity 42c of the filled box 41c. That is, Fig.18 The center of gravity shown in Fig.17 The centroids 42a to 42c shown in FIG. 2 exclude the centroid 42c of the filled-in box 41c corresponding to all or part of the characters of the character string 22a.

[0132] like Fig.14 As shown, if the processing of S185 is completed, the image processing unit 15a ends Fig.14 The box shown in FIG.

[0133] like Figure 5 As shown, when the frame determination process of S124 is completed, the image processing unit 15a executes a painted frame determination process for determining a painted frame for the marked paper area image of the current target area (S125).

[0134] Fig.19 yes Figure 5 Flowchart of the painted frame determination process shown in .

[0135] like Fig.19As shown, the image processing unit 15a selects the object center of gravity from the center of gravity of the frame determined in S185 (S201). The object center of gravity is one of the centers of gravity of the frame determined in S185. The object center of gravity is selected from the centers of gravity of the frame determined in S185 that have not yet been processed in this painted frame determination process.

[0136] When the process of S201 is completed, the image processing unit 15a determines whether there is a close distance center of gravity in the center of gravity of the unpainted frame calculated in the unpainted frame determination process of S122 (S202). The close distance center of gravity is a center of gravity of the unpainted frame calculated in the unpainted frame determination process of S122 whose distance from the center of gravity of the frame of the current object is less than a predetermined distance.

[0137] If the image processing unit 15a determines in S202 that the near-distance center of gravity does not exist in the centers of gravity of the unpainted frame calculated in the unpainted frame determination process in S122, it determines the center of gravity of the current object frame as the center of gravity of the painted frame (S203).

[0138] If the image processing unit 15a determines in S202 that there is a close distance center of gravity among the centers of gravity of the unpainted frame calculated in the unpainted frame determination process in S122 or the process in S203 is completed, it determines whether there is an unselected center of gravity among the centers of gravity of the frame determined in S185 (S204). The unselected center of gravity is a center of gravity among the centers of gravity of the frame determined in S185 that has not yet been processed in this painted frame determination process.

[0139] If the image processing unit 15 a determines in S204 that there is an unselected center of gravity among the centers of gravity of the frame specified in S185 , it executes the process of S201 .

[0140] If the image processing unit 15a determines in S204 that there is no unselected center of gravity among the centers of gravity of the frame determined in S185, the processing ends. Fig.19 The painted frame determination process shown in .

[0141] Fig. 20 It means in Fig.19 ] A diagram showing an example of the center of gravity of a painted frame determined in the painted frame determination process shown in FIG.

[0142] The center of gravity of the unpainted frame calculated in the unpainted frame determination process of S122 is Fig.13 The center of gravity 34a shown in FIG. 1 and the center of gravity of the frame determined in S185 is Fig.18 In the case of the center of gravity 42a, 42b shown in FIG. Fig.19 The painted frame determination process shown in FIG. Fig. 20 The center of gravity 42b is shown in FIG.

[0143] like Figure 5 As shown, when the painted frame determination process of S125 is completed, the image processing unit 15a executes a selection item determination process (S126) for determining the selection items on the marked paper area image of the current target area.

[0144] Fig.21 yes Figure 5 Flowchart of the selection item determination process shown in .

[0145] like Fig.21 As shown, the image processing unit 15a determines the order of the centers of gravity of the painted frames determined in the painted frame determination process of S125 among the centers of gravity of all the frames determined in S185 in the arrangement direction of the frames indicated by the marking paper information 14b for the current object area (S221).

[0146] For example, when the center of gravity of the frame determined in S185 is Fig.18 The center of gravity 42a, 42b shown in FIG. 4 and the center of gravity of the painted frame determined in the painted frame determination process of S125 are Fig. 20 In the case of the center of gravity 42b shown in the marking paper information 14b, the image processing unit 15a determines that the center of gravity coordinates of the painted frame determined in the painted frame determination processing of S125 are the second from the top among the center of gravity coordinates of all the frames determined in S185 in the arrangement direction of the frames shown in the current object area, that is, the direction of the y coordinate.

[0147] If the processing of S221 is completed, the image processing unit 15a determines the item corresponding to the painted frame whose center of gravity is determined in the painted frame determination processing of S125 as a selected item (S222) according to the order determined in S221 and the corresponding list of frame items stored for the current object area in the marked paper information 14b.

[0148] When the processing of S222 is completed, the image processing unit 15a ends. Fig.21 The selection item determination process shown in .

[0149] like Figure 5 As shown, if the selection item determination processing of S126 is completed, the image processing unit 15a determines whether there is an unselected area in the multiple marking paper areas of the marking paper 20 (S127). The unselected area is an area in the multiple marking paper areas that has not been selected as the object area in this main body processing.

[0150] When the image processing unit 15 a determines in S127 that there is an unselected area among the plurality of marked paper areas of the marked paper 20 , it executes the process of S121 .

[0151] If the image processing unit 15a determines in step S127 that there is no unselected area in the plurality of marked paper areas of the marked paper 20, the image processing unit 15a ends the process. Figure 5 The main processing shown in FIG.

[0152] like Figure 3 As shown, when the main body processing of S102 is completed, the image processing unit 15a performs post-processing of the main body processing (S103).

[0153] Fig. 22 yes Figure 3 The flowchart of post-processing is shown in FIG.

[0154] like Fig. 22 As shown, the image processing unit 15a determines whether the determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frames has failed in the main body processing of S102 (S241).

[0155] If the image processing unit 15a determines in S241 that the determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frames in the main processing of S102 has failed, an error is displayed on the display unit 12 (S242), and the process ends. Fig. 22 The post-processing shown in .

[0156] If the image processing unit 15a determines in S241 that the determination of the representative value of the coordinates of the unpainted frame in the direction orthogonal to the arrangement direction of the frame has failed in the main processing of S102, it executes output information generation processing to generate information for output to subsequent processing (hereinafter referred to as "output information") (S243).

[0157] Fig.23 yes Fig. 22 Flowchart of the output information generation process shown in .

[0158] like Fig.23 As shown, the image processing unit 15a selects a target area from among the plurality of marked paper areas of the marked paper 20, which has not yet been targeted in the current output information generation process (S261).

[0159] When the process of S261 is completed, the image processing unit 15a determines whether the number of painted frames determined by the main body process of S102 is within the allowable range for the current target area based on the marked paper information 14b (S262).

[0160] If the image processing unit 15a determines in S262 that the number of painted frames determined by the main processing of S102 is within the allowable range for the current object area, the selection item and the painted frame determination information are included in the output information for the current object area (S263). The selection item is the item determined in the selection item determination processing of S126. The painted frame determination information is information for determining the painted frame corresponding to the selection item. For example, the so-called painted frame determination information can be the coordinates of the center of gravity of the painted frame, the coordinates of the two vertices of the diagonal corners of the boundary box of the painted frame, or the coordinates, width and height of one vertex of the boundary box of the painted frame.

[0161] If the image processing unit 15a determines in S262 that the number of painted frames determined by the main body processing of S102 for the current object area is not within the allowable range, it determines whether the number of painted frames determined by the main body processing of S102 for the current object area is less than the allowable range (S264).

[0162] If the image processing unit 15a determines in S264 that the number of painted frames determined by the main processing of S102 is less than the allowable range for the current object area, the selection item determined in the selection item determination processing of S126, the painted frame determination information corresponding to the selection item, and the too few information are included in the output information for the current object area (S265). The too few information is information indicating that the number of painted frames is less than the allowable range.

[0163] If the image processing unit 15a determines in S264 that the number of painted frames determined by the main processing in S102 is not less than the allowable range for the current object area, the selection item determined in the selection item determination processing in S126, the painted frame determination information corresponding to the selection item, and the excess information are included in the output information for the current object area (S266). The excess information is information indicating that the number of painted frames is greater than the allowable range.

[0164] When the processing of S263, S265 or S266 is completed, the image processing unit 15a determines whether there is an unselected area in the plurality of marked paper areas of the marked paper 20 (S267). The unselected area is an area in the plurality of marked paper areas that has not been selected as a target area in this output information generation process.

[0165] When the image processing unit 15 a determines in S267 that there is an unselected area among the plurality of marked paper areas of the marked paper 20 , it executes the process of S261 .

[0166] If the image processing unit 15a determines in step S267 that there is no unselected area in the plurality of marked paper areas of the marked paper 20, the image processing unit 15a ends the process. Fig.23The output information shown in the above is generated by processing.

[0167] like Fig. 22 As shown, if the output information generation process of S243 is completed, the image processing unit 15a generates output data in a determined output format based on the output information generated in S243 (S244). For example, in the case where the output information contains multiple repeated selection items that can be summarized into one, the image processing unit 15a can summarize these selection items into one. In addition, in the case where the output information contains multiple selection items whose arrangement order is preferably changed, the image processing unit 15a can change the arrangement order of these selection items to an appropriate order. In addition, the image processing unit 15a can also make the file format of the output data such as comma-delimited text, tab-delimited text, XML format, JSON format, etc. into a determined format.

[0168] When the processing of S244 is completed, the image processing unit 15a outputs the output data generated in S244 to the determined output destination (S245). For example, the image processing unit 15a may write the output data generated in S244 to the storage unit 14, or may display the output data generated in S244 on the display unit 12.

[0169] When the process of S245 is completed, the image processing unit 15a ends. Fig. 22 The post-processing shown in .

[0170] like Figure 3 As shown, if the post-processing of S103 is completed, the image processing unit 15a ends Figure 3 The actions shown in .

[0171] There may be a case where the image processing unit 15a fails to confirm the selected item. Therefore, there may be a case where a person confirms whether the processing content of the image processing unit 15a is correct. Therefore, in order to facilitate the confirmation work of the confirmer on the processing content of the image processing unit 15a, the image processing unit 15a can be selected from the marked paper image or in Figure 3 The various images generated in the actions shown in the figure cut out the painted frame portion and display it on the display unit 12, or the image for making the center of gravity or boundary frame of the painted frame stand out can be superimposed on the marked paper image or on the Figure 3 The various images generated in the actions shown in the figure are displayed on the display unit 12, and an image for making the character string corresponding to the framed item stand out can also be superimposed on the marked paper image or on the Figure 3 The various images generated in the actions shown in FIG. 1 are displayed on the display unit 12 .

[0172] Fig.24 This is a diagram showing an example of a marked paper region image on which an image for making the center of gravity of a painted frame stand out is superimposed.

[0173] exist Fig.24 In the marked paper area image 30 shown in FIG. 1 , an image 50 for making the center of gravity of the painted frame 30b stand out is superimposed on the painted frame 30b. Therefore, the person who confirms the processing content of the image processing unit 15a can confirm the processing content of the image processing unit 15a by Fig.24 The marked paper area image 30 shown in FIG. 1 makes it easy to confirm that the painted frame 30b determined by the image processing unit 15a is correct.

[0174] As described above, the image processing system 10 generates a portion of the marked paper region image 30 (see Figure 4 ) of the first image of the element touching the boundary 33 (refer to Fig.10 )(S162-S163), generating a portion 32a (refer to FIG. 1 ) which is the element with the largest area relative to the area in the image 33 removed from the image 33 Fig.10 ) is less than 1, i.e., part 32c (refer to Fig.10 ) of the second image 34 (refer to Fig.11 )(S165), the element in the image 34, that is, the portion 32a is determined as the frame 22b whose inner side is not coated (refer to Figure 4 ) That is, the unpainted frame 30a (refer to Figure 4 ) part (S142), the elements in the marked paper area image 30 are positioned in a direction orthogonal to the arrangement direction of the frame 22b and are within a certain distance from the unpainted frame 30a, that is, the frames 41a, 41b (refer to Fig.17 ) is determined as a frame surrounding the frame 22b (S124), and the frame 22b that does not have a corresponding unpainted frame 30a at a distance is determined as a frame with an inner side painted, that is, a painted frame 30b (refer to Figure 4 )(S125), it is therefore possible to determine the frame to be coated on the inside more appropriately than before.

[0175] The image processing system 10 determines the selection item (S222) corresponding to the painted frame based on a frame-item correspondence list indicating the correspondence between the position of the frame in the arrangement direction of the multiple frames in the marked paper area and the item, and the position of the painted frame in the arrangement direction of the multiple frames in the marked paper area, thereby being able to determine the selection item selected in the marked paper area more appropriately than before.

[0176] In this embodiment, when the image processing system 10 determines in S147 that the predetermined distance of the mean shift processing in S143 is greater than the determined distance, an error is displayed (S242) and the process ends. Figure 3However, when the image processing system 10 determines in S147 that the predetermined distance of the mean shift processing in S143 is greater than the determined distance, it is also possible to terminate the process instead of displaying an error. Figure 3 Instead of the action shown in S101, the image processing system 10 corrects the parameters of the preprocessing of S101 and re-executes the preprocessing of S101. In addition, when the image processing system 10 determines in S147 that the predetermined distance of the mean shift processing in S143 is greater than the determined distance, the parameters of the preprocessing of S101 may be corrected, and when re-executing the preprocessing of S101, a warning may be issued that the preprocessing of S101 may be incomplete.

[0177] In the present embodiment, when the image processing system 10 determines in S147 that the predetermined distance of the mean shift processing in S143 is greater than the determined distance, it determines that the determination of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed (S149). However, when the image processing system 10 determines in S147 that the predetermined distance of the mean shift processing in S143 is greater than the determined distance, it may not determine that the determination of the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames has failed, but may determine the determined coordinates as the representative value of the coordinates of the unpainted frame in the direction perpendicular to the arrangement direction of the frames, and end the process. Figure 6 The unpainted frame determination process shown in . Here, the image processing system 10 may use any one of the coordinates of the multiple centroid ...

[0178] In the present embodiment, the image processing system 10 performs the opening operation process of S163 and performs the element coordinate area acquisition process of S164 after performing the boundary element removal process of S162. However, if the image processing system 10 does not have a significant impact on the final processing result by omitting the opening operation process of S163, it is also possible to perform the element coordinate area acquisition process of S164 without performing the opening operation process of S163 after performing the boundary element removal process of S162.

[0179] In the present embodiment, the image processing system 10 determines the selection item corresponding to the painted frame from the image of the first attribute marking paper area 21, and a plurality of combinations of a character string 21a representing an item in the first attribute marking paper area 21 and a frame 21b arranged just below the character string 21a and painted inside when the item represented by the character string 21a just above is selected are arranged in the left-right direction. However, the image processing system 10 may determine the selection item corresponding to the painted frame from the image of the marking paper area by the same method, and a plurality of combinations of a character string representing an item in the marking paper area and a frame arranged just above the character string and painted inside when the item represented by the character string just below is selected are arranged in the left-right direction.

[0180] In the present embodiment, the image processing system 10 determines the selection item corresponding to the painted frame from the image of the second attribute marking paper area 22, and a plurality of combinations of a character string 22a representing an item and a frame 22b arranged immediately to the right of the character string 22a and painted inside when the item represented by the character string 22a immediately to the left is selected are arranged in the vertical direction. However, the image processing system 10 may determine the selection item corresponding to the painted frame from the image of the marking paper area by the same method, and a plurality of combinations of a character string representing an item and a frame arranged immediately to the left of the character string and painted inside when the item represented by the character string immediately to the right is selected are arranged in the vertical direction.

[0181] In this embodiment, the image processing system 10 uses an elliptical frame as the frame in the marking paper. However, in the image processing system 10, as long as it is a closed frame, a frame of a shape other than an ellipse can also be used. For example, in the image processing system 10, a rectangular frame can be used as the frame in the marking paper.

[0182] In the present embodiment, the image processing system 10 is constituted by one computer, but the image processing system 10 may be constituted by a plurality of computers.

[0183] The scope of the present invention is not limited to the above content, but is defined by the description of the claims, so it can be considered that the embodiments described in this specification are only illustrative, not limiting. Therefore, all changes that do not depart from the scope and limits of the claims, as well as contents that are equivalent to the scope and limits of the claims, are included in the scope of the claims.

Claims

1. An image processing system, characterized in that: determining a painted frame from an area image, the painted frame being a frame with a painted inner side, the area image being an image of the area in an image read by an image reading device from a marked paper having an area, wherein a plurality of combinations of a character string representing an item in the area and the frame arranged in a first direction relative to the character string and with a painted inner side when the item represented by the character string is selected are arranged in a second direction orthogonal to the first direction, The image processing system generates a first image by removing at least elements touching the boundary of the regional image from the regional image, generates a second image by removing at least elements whose area is less than a predetermined ratio relative to the element with the largest area in the first image from the first image, determines the elements in the second image as unpainted frames, the unpainted frames being the frames whose inner sides are not painted, and the determined ratio is less than 1, Among the elements in the area image, the elements whose positions in the first direction are within a predetermined distance from the positions in the first direction of the unpainted frame are determined as the frame, The frame for which there is no corresponding unpainted frame within the determined distance is determined as the painted frame.

2. The image processing system according to claim 1, characterized in that The image processing system determines the item corresponding to the painted frame based on a list indicating the correspondence between the position of the frame in the second direction among the multiple frames in the area and the items, and the position of the painted frame in the second direction among the multiple frames in the area.

Images