Information processing apparatus, information processing method, and program

Abstract

To accurately extract an area to be masked from an image.SOLUTION: An information processing apparatus has: first extraction means that extracts an area to be masked on the basis of a predetermined format from a document image; second extraction means that extracts an area of character strings to be masked from the document image; third extraction means that, when the positional coordinates of a first area that is the area extracted by the first extraction means and the positional coordinates of a second area that is the area extracted by the second extraction means overlap each other, extracts a third area on the basis of the positional coordinates of the first area and the positional coordinates of the second area; and determination means that determines the area to be masked in the document image on the basis of the area extracted by the first extraction means, the area extracted by the second extraction means, and the area extracted by the third extraction means.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for extracting a region of interest from an image. [Background technology]

[0002] When sharing documents containing personal information or confidential information, such as identification documents, application forms, or drawings, with others, some parts of the document may be masked by filling them in (also known as "redaction" or "masking"). Patent Document 1 discloses a technique for masking fixed areas that should be masked when printing a document and areas where proper nouns appear. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-090997 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, masking is performed on fixed areas depending on the type of form and areas where character strings to be masked appear. The latter areas vary depending on the form. One method for masking fixed areas is to store the area to be masked (position and size) and mask that area. For example, in the case of a driver's license, the fields where the name and address are written are fixed, and the area to be masked can be identified by setting the position and size of those fields. Furthermore, Patent Document 1 discloses a method for masking areas that vary depending on the form, such as billing names and invoice addresses, in which character strings are extracted using OCR and areas where proper nouns appear are extracted and masked.

[0005] However, with methods that extract and mask fixed areas, some areas may not be masked if the amount of text varies so much that it extends beyond the predefined area. Furthermore, with methods that extract and mask areas that vary for each form, accurate masking may not be possible if the form contains noise or if a logo or seal overlaps the text, causing OCR to fail. As described above, when the areas to be masked include a mixture of fixed areas for each form type and areas that vary for each form, neither the method of extracting fixed areas nor the method of extracting variable areas may be able to handle the situation. This poses the problem of being unable to accurately extract the areas to be masked.

[0006] Therefore, an object of the present invention is to accurately extract a region to be masked from an image. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a first extraction means for extracting an area to be masked from a document image based on a predetermined format; a second extraction means for extracting an area of ​​a character string to be masked from the document image; a third extraction means for extracting a third area based on the position coordinates of a first area extracted by the first extraction means and the position coordinates of a second area extracted by the second extraction means when there is overlap between the position coordinates of the first area and the position coordinates of the second area; and a determination means for determining the area to be masked in the document image based on the area extracted by the first extraction means, the area extracted by the second extraction means, and the area extracted by the third extraction means. [Effects of the Invention]

[0008] According to the present invention, it is possible to extract an area to be masked from an image with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating an example of the overall configuration of an image processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an MFP. [Figure 3] FIG. 2 illustrates an example of the hardware configuration of an external storage device. [Figure 4] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Figure 5] 10 is a flowchart showing the overall processing executed by the MFP. [Figure 6] FIG. 10 is a diagram showing an example of a screen displayed on an operation unit of the MFP. [Figure 7] FIG. 10 is a diagram illustrating an example of a correction method. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed on an operation unit of the MFP. [Figure 9] FIG. 10 is a diagram showing an example of a screen displayed on an operation unit of the MFP. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Embodiment 1] <Image processing system> Fig. 1 shows an example of the overall configuration of an image processing system according to this embodiment. The image processing system in Fig. 1 includes an MFP (Multifunction Peripheral) 110 and an external storage 120. The MFP 110 is communicably connected via a LAN (Local Area Network) to a server that provides various services on the Internet.

[0012] The MFP 110 is a multifunction peripheral having multiple functions such as a scanner and a printer, and is an example of an information processing device. The MFP 110 also has a function of transferring scanned image files to a system capable of storing files, such as external storage. Note that the information processing device is not limited to a multifunction peripheral having a scanner and a printer, and may also be a PC (personal computer). The external storage 120 is a server system that provides a service capable of saving a file received via the Internet or transmitting a file in response to a processing request from an external device. The external storage 120 is, for example, a cloud service. The number of external storages present in the image processing system is not limited to one and may be plural.

[0013] The image processing system of this embodiment is configured to include the MFP 110 and the external storage 120, but is not limited thereto. For example, part of the functions and processes of the MFP 110 may be shared by another server arranged on the Internet or on a LAN. Also, the external storage 120 may be arranged on a LAN instead of on the Internet. Further, the external storage 120 may be a mail server, and the MFP 100 may attach and transmit a scanned image file to an email. Also, the MFP 110 may have the storage function of the external storage 120.

[0014] <Hardware Configuration of MFP> FIG. 2 shows an example of the hardware configuration of the MFP 110. The MFP 110 includes a control unit 210, an operation unit 220, a printer 221, a scanner 222, and a modem 223. The control unit 210 includes the following units 211 to 219 and controls the operation of the entire MFP 110. The CPU 211 reads out the programs stored in the ROM 212 and the HDD 214 and executes them using the RAM 213 as a work memory, thereby realizing various functions of the MFP 110 and each process of the flowchart described later. The RAM 213 is used as a temporary storage area such as the main memory and work area of the CPU 211. In this embodiment, it is assumed that one CPU 211 executes each process shown in the flowchart below using one memory (RAM 213 or HDD 214), but it is not limited thereto. For example, a plurality of CPUs, a plurality of RAMs, or HDDs may cooperate to execute each process. The HDD 214 is a large-capacity storage unit that stores image data and various programs.

[0015] The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The operation unit 220 is equipped with a touch panel display, a keyboard, and the like, and accepts operations, inputs, and instructions from a user. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data to be printed is transferred from the control unit 210 to the printer 221 via the printer I / F 216 and printed on a recording medium. The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads an original placed on a platen or an ADF (Auto Document Feeder), not shown, to generate image data, and inputs the image data to the control unit 210 via the scanner I / F 217.

[0016] The MFP 110 has a copy function that prints out image data generated by the scanner 222 from the printer 221, and a function that transmits the image data as a file or by email. The modem I / F 218 is an interface that connects the modem 223 to the control unit 210. The MFP 110 uses the modem 223 to send and receive image data to and from a facsimile machine on a PSTN (Public Switched Telephone Network). The network I / F 219 is an interface that connects the control unit 210 to a LAN. The MFP 110 uses the network I / F 219 to send and receive image data and various information to and from external devices (such as the external storage 120) on the Internet via the LAN.

[0017] <Hardware configuration of external storage> FIG. 3 shows an example of the hardware configuration of the external storage 120. The external storage 120 has a CPU 311, a ROM 312, a RAM 313, a HDD 314, and a network I / F 315. The CPU 311 controls the overall operation of the external storage 120. The CPU 311 reads and executes programs stored in the ROM 312, using the RAM 313 as a work memory, thereby realizing various processes of the external storage 120. The RAM 313 is used as a temporary storage area such as the main memory and work area of ​​the CPU 311. The HDD 314 is a mass storage unit that stores image data and various programs. The network I / F 315 is an interface that connects the external storage 120 to the Internet. The external storage 120 receives processing requests from an external device (such as the MFP 110) using the network I / F 315, and transmits and receives various information.

[0018] <Image processing system functional configuration> FIG. 4 shows an example of the functional configuration of the image processing system according to this embodiment. The functions of the units included in the native function unit 410 are functions that are standardly provided in the MFP 110. On the other hand, the functions of the units included in the additional function unit 420 are functions of an application (hereinafter referred to as a masking app) that masks a target area in a scanned image and transmits the masked image obtained to the external storage 120. The functions of the native function unit 410 and the additional function unit 420 are realized by the CPU 211 reading a program including the masking app stored in the ROM 212 or HDD 214 of the MFP 110 into the RAM 213 and executing the program. The masking app is based on Java (registered trademark), and functions can be easily added to the MFP 110. Note that other additional applications (not shown) may be installed in the MFP 110.

[0019] The native function unit 410 has a scan execution unit 411, an internal data storage unit 412, a print execution unit 413, and a UI (user interface) display unit 414. The additional function unit 420 has a main processing unit 421, an image processing unit 422, a first area extraction unit 423, a second area extraction unit 424, a correction area extraction unit 425, an internet access unit 426, a scan instruction unit 427, a display control unit 428, and a print instruction unit 429.

[0020] The main processing unit 421 controls the overall processing for the additional function unit 420, and requests each unit included in the additional function unit 420 to execute processing. The image processing unit 422 performs analysis and processing on the image data. The image processing unit 422 performs recognition processing on the image data, such as BS, OCR, and image rotation and tilt correction. BS stands for Block Selection, and is a process for extracting a rectangular area indicating the location of a character string from an image. OCR stands for Optical Character Recognition, and is a process for extracting a character string from an image.

[0021] Furthermore, the image processing unit 422 generates a masking image by masking the masking area on the image data. The masking area is the area to be masked. The position and size of the masking area are expressed by the position coordinates of the start point and end point of a rectangular area, such as "(441,957), (1369,1057)". In this embodiment, the masking area is a rectangular area, but it may be any shape, such as an ellipse or a triangle. The masking image is an image in which the masking area on the image data is masked. Masking may be a process of filling the masking area with a chromatic color such as black, or a process of hiding it with a background color. There are no particular limitations on the process, as long as it makes the character string in the masking area invisible.

[0022] The first region extraction unit 423 extracts a first region from the image data. The first region is a fixed region (position and size) predetermined as a masking target, and is set for each form type. For example, it is the address and name section of a standard form such as a driver's license. The first region extraction unit 423 extracts the first region by referring to a predetermined format in which the masking target region is defined. A list of formats is stored in the HDD 214 or the like, and the first region extraction unit 423 selects a predetermined format from the format list. The first region extraction unit 423 may select a format according to the form type, a format specified by the user, or a format based on similarity with the image data. The first region extraction unit 423 may also extract a region specified by the user as the first region. The first region extracted by the first region extraction unit 423 becomes a candidate for a masking region.

[0023] The second region extraction unit 424 extracts a second region from the image data. The second region is the region (position and size) of the character string with the attribute to be masked. For example, it could be a personal name written in a free-form text field in a questionnaire document or a company name with line breaks. The second region varies for each image data. The second region extraction unit 424 estimates the attribute of the character string from the character string obtained by OCR and extracts the region of the character string with the attribute corresponding to the processing target. Here, the attribute can be, for example, "company name," "personal name," "address," or "telephone number." Any other attribute may also be included. One method for estimating the attribute of a character string is to use a model trained by providing known attributes for the character string as training data. Another method is to estimate the attribute of the key character string as the attribute of the value character string when the character string is positioned as a key and a value on the form layout. For example, if "Total Amount" is written to the left of "120 Yen," the attribute of the character string "120 Yen" is estimated to be "Total Amount." Here, the key character string may be normalized and used as an attribute. For example, if the key character string is "total," it is normalized to "total amount" and used as an attribute. There are no particular limitations on the method for estimating the attribute of a character string. The second region extracted by the second region extraction unit 424 becomes a candidate for a masking region.

[0024] If the first region extracted by the first region extraction unit 423 and the second region extracted by the second region extraction unit 424 overlap, the correction region extraction unit 425 corrects and extracts them. In this embodiment, the correction region extraction unit 425 compares the position coordinates of the first region extracted by the first region extraction unit 423 with the position coordinates of the second region extracted by the second region extraction unit 424. If there is overlap between the two, the overlapping first and second regions are designated as regions to be corrected. The correction region extraction unit 425 corrects the region to be corrected and extracts the corrected region. The correction method may involve performing a logical OR on the position coordinates of the overlapping first and second regions, or may involve obtaining a circumscribing rectangle that encompasses the position coordinates of the overlapping first and second regions. Alternatively, the circumscribing rectangle may be reduced by excluding regions without text. Other correction methods are not particularly limited. The area extracted by the correction area extraction unit 425 becomes a candidate for the masking area.

[0025] The internet access unit 426 sends a processing request to a cloud service or the like that provides a storage function. Cloud services generally use protocols such as REST and SOAP to publish various interfaces for saving files in cloud storage and retrieving saved files from external devices. The internet access unit 426 sends a processing request to the cloud service using the published interface of the cloud service. In this embodiment, the internet access unit 426 sends the masked image generated by the image processing unit 422 to the external storage 120 via the network I / F 219.

[0026] A scan instruction unit 427 requests the scan execution unit 411 to perform scan processing according to the scan settings input via the UI screen. The UI screen is displayed on the touch panel display of the operation unit 220 under the control of a display control unit 428. The scan execution unit 411 receives a scan request including scan settings from the scan instruction unit 427. In accordance with the scan request, the scan execution unit 411 controls the scanner 222 via the scanner I / F 217 to read a document placed on the platen glass and generate scanned image data.

[0027] Display control unit 428 controls the display of a UI screen for accepting user operations on a liquid crystal display unit (touch panel display) having a touch panel function of operation unit 220 of MFP 110. The operations accepted by the UI screen here include, for example, an operation to instruct scan settings or start scanning, an operation to specify a masking area for a preview of a scanned image, and an operation to instruct output settings or start output for a preview of a masked image.

[0028] The print instruction unit 429 transmits a print request and a masking image according to the print settings input via the UI screen to the print execution unit 413 . The print execution unit 413 receives a print request including print settings and a masking image from the print instruction unit 429. The print execution unit 413 generates image data for printing in accordance with the print request. The print execution unit 413 controls the printer 221 via the printer I / F 216 in accordance with the generated image data for printing to print the masking image on a recording medium.

[0029] An internal data storage unit 412 acquires and stores data in the HDD 214 . The UI display unit 414 displays, on the touch panel display of the operation unit 220 of the MFP 110, a menu screen and a UI screen for accepting operations for setting and starting various functions such as a copy function.

[0030] <Overall processing flow> 5 is a flowchart showing the overall process of generating and printing a masked image from an image scanned by the MFP 110. The process of this flowchart is realized by the CPU 211 reading into the RAM 213 and executing a program including a masking application stored in the ROM 212 or HDD 214 of the MFP 110. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted.

[0031] Although this flowchart shows processing for printing a masking image, processing for converting the masking image into a file and transmitting the file to the external storage 120 may also be used. Note that, in this embodiment, an example will be described in which the display control unit 428 displays a UI screen on the touch panel display of the operation unit 220, but a configuration in which the display control unit 428 provides a UI screen to an external device and the UI screen is displayed on the external device may also be used.

[0032] In S501, the main processing unit 421 requests the display control unit 428 to display the main screen of the masking application. The display control unit 428 generates a main screen for accepting masking settings and displays it on the touch panel display of the operation unit 220.

[0033] 6(a) shows an example of a main screen in this embodiment. A main screen 600 includes an attribute specification check box 601, an attribute all selection check box 602, and a scan button 603.

[0034] The attribute specification checkboxes 601 are checkboxes for specifying attributes to be masked when extracted by the second area extraction unit 424. The attribute specification checkboxes 601 accept the user's selection of attributes to be masked. The attribute specification checkboxes 601 correspond to attributes that can be extracted by the second area extraction unit 424. In this embodiment, the attributes that can be selected by the attribute specification checkboxes 601 are "personal name," "address," and "company name," but any other attributes may be included. The attribute all selection checkboxes 602 are checkboxes for selecting or deselecting all checkboxes in the attribute specification checkboxes 601. The main processing unit 421 acquires the attributes selected by the attribute specification checkboxes 601 as masking settings. The scan button 603 is a button for instructing the start of scanning.

[0035] Returning to the description of Fig. 5, in S502, the main processing unit 421 requests the scan execution unit 411 to perform a scan via the scan instruction unit 427, and acquires image data. In this embodiment, the main processing unit 421 acquires image data read by scanning, but the main processing unit 421 may also acquire image data from the HDD 214 via the internal data storage unit 412. The main processing unit 421 may also acquire image data from the external storage 120 via the internet access unit 426. Any other method may be used to acquire image data.

[0036] In S503, the main processing unit 421 requests the image processing unit 422 to perform skew rotation correction on the image data acquired in S502. The image processing unit 422 performs skew rotation on the image data and generates skew-rotation-corrected image data. In S504, the main processing unit 421 requests the image processing unit 422 to perform BS / OCR on the image data generated in S503. The image processing unit 422 performs BS / OCR on the image data generated in S503 and extracts character strings and character string areas. Table 1 below shows an example of an extracted character string and character string area as an example of a BS / OCR result. The position and size of the character string area are represented by the position coordinates of the start point and end point of the rectangular area.

[0037] [Table 1]

[0038] In S505, the main processing unit 421 requests the first area extraction unit 423 to extract a first area using the character string area extracted in S504. The first area extraction unit 423 extracts the first area using the character string area. First, the first area extraction unit 423 obtains a list of formats from the external storage 120 via the Internet access unit 426. Note that the first area extraction unit 423 may obtain the list of formats from the HDD 214 via the internal data storage unit 412, or may obtain it by any other method. The format is made up of a character string area on the form and an area to be masked. The positions and sizes of the character string area on the form and the area to be masked are represented by the position coordinates of the start and end points of the rectangular area.

[0039] Next, the first area extraction unit 423 compares the position coordinates of the character string area of ​​each format in the acquired format list with the position coordinates of the character string area extracted in S504, and selects the format with the highest similarity. Here, the similarity may be based on the calculation result of the overlapping area of ​​the character string areas, and is not particularly limited. Table 2 below shows an example of a format selected using the character string areas shown in Table 1. The first area extraction unit 423 extracts the area to be masked in the selected format as the first area.

[0040] [Table 2]

[0041] In S506, the main processing unit 421 requests the second region extraction unit 424 to extract a second region using the masking settings acquired in S501 and the character string and character string region extracted in S504. The second region extraction unit 424 extracts a second region using the masking settings, character string, and character string region. First, the second region extraction unit 424 estimates the attribute of the character string. For example, a character string representing a company name such as "Iroha Co., Ltd." is estimated to have the attribute "company name." Table 3 below shows an example of attributes estimated for the character strings and character string regions shown in Table 1.

[0042] [Table 3]

[0043] Next, the second region extraction unit 424 extracts, as the second region, a character string region that corresponds to the attribute specified in the masking setting. For example, if the attribute specified in the masking setting is "company name," the second region extraction unit 424 extracts, as the second region, a character string region of "Iroha Co., Ltd.", whose attribute is the company name.

[0044] In this embodiment, the second region is extracted after the first region is extracted, but the first region may be extracted after the second region is extracted, or the extraction of the first region and the extraction of the second region may be performed in parallel.

[0045] In S507, the main processing unit 421 requests the correction area extraction unit 425 to integrate and extract the first area extracted in S505 and the second area extracted in S506 if there is any overlap between the first area and the second area. The correction area extraction unit 425 compares the position coordinates of the first area with the position coordinates of the second area, and if there is any overlap between the first area and the second area, performs correction so as to integrate the overlapping first area and second area.

[0046] An example of a correction method will be described with reference to Fig. 7. Fig. 7(a) shows regions 700 and 701 extracted from an image 710 showing an enlarged portion. Region 700 is an example of a first region. Region 701 is an example of a second region. Parts of region 700 and region 701 overlap.

[0047] Area 700 includes "K.K. Co., Ltd." on the first line, but does not include "Yanon" on the second line, which is created by a line break. It also includes an area with no text. Area 700 was defined assuming that the company name would be written on a single line, so the text area on the second line, which is created by a line break, extends beyond area 700 and cannot be extracted. Area 701 includes "Yanon Inc.", but does not include the "Ki" written to the right of the space. Because the "Ki" on the right end of the first line was recognized as noise by OCR and was not included in the character string area, "Yanon Inc.", which is estimated to have the attribute "company name," is extracted. In other words, area 701 does not include the "Ki" on the right end of the first line, but does include "Yanon" on the second line.

[0048] FIG. 7B shows an example in which correction is performed on the area 700 and the area 701. Area 702 indicates an area corrected by taking the logical sum of area 700 and area 701. In the logical sum method, the areas that could not be extracted in the first area extraction and the second area extraction are complemented, and all character string areas are included in area 702.

[0049] Area 703 indicates an area corrected by a method of taking a circumscribing rectangle that encompasses area 700 and area 701. In the method of taking a circumscribing rectangle, as in the method of taking a logical sum, the areas that could not be extracted by the first area extraction and the second area extraction are complemented, and all character string areas are included in area 703. In addition, area 700 and area 701 are contained within area 703, and the positions and sizes of each area are hidden, which has the effect of making it difficult to guess the number of character strings from the size of area 703.

[0050] Area 704 indicates the area corrected by shrinking the circumscribing rectangle that encompasses area 700 and area 701. In the shrinking method, as in the case of the logical sum method, the areas that could not be extracted by the first area extraction and the second area extraction are complemented, and all character string areas are included in area 704. In addition, since areas without character strings are excluded, there is an effect of suppressing unnecessary masking areas.

[0051] Table 4 shows examples of areas corrected using the correction methods (logical sum method, circumscribed rectangle method, and reduction method) shown in Fig. 7(b) for the first area (area to be masked) shown in Table 2 and the second area (character string area with the attribute "company name") shown in Table 3. As shown in Table 2, the correction area extraction unit 425 uses the position coordinates of the first area and the second position coordinates to determine the position coordinates of the area obtained by integrating the first area and the second area.

[0052] [Table 4]

[0053] Returning to the description of Figure 5, in S508, the main processing unit 421 requests the display control unit 428 to generate a preview screen using the image data generated in S503, the first region extracted in S505, the second region extracted in S506, and the region corrected in S507. The display control unit 428 generates a preview screen that superimposes and displays rectangular items according to the position coordinates of the first region, the second region, and the corrected region on the image data generated in S503, and displays the preview screen on the touch panel display of the operation unit 220.

[0054] 6B is a diagram illustrating a preview screen according to this embodiment. The preview screen 610 has a preview display area 611, a preview masking area 612, a previous page button 613, a page number display 614, and a next page button 615. The preview screen 610 also has an unmasking button 616, an area selection button 617, an area instruction button 618, a display reduction button 619, a display fit button 620, and a display enlargement button 621. The preview screen 610 also has a format specification form 622, an attribute specification form 623, a correction specification form 624, and a print button 625.

[0055] The preview display area 611 is an area that displays the image data generated in S503. The preview masking area 612 is an area that displays, as candidates for the masking area, the first area extracted in S505, the second area extracted in S506, the area corrected in S507, and the area specified by an area instruction button 618 (described later). Note that the display control unit 428 superimposes a rectangular item (for example, black) on each area of ​​the preview masking area 612 in the preview display area 611 so that the character strings on the image data in the preview masking area 612 are not visible. Because the preview masking area 612 is superimposed on the image data in the preview display area 611, the masked image can be presented to the user for confirmation before printing.

[0056] The previous page button 613 is a button for displaying the image data of the previous page when there are multiple pages of image data. The page number display 614 displays the page of the image data currently being displayed and the total number of pages. The next page button 615 is a button for displaying the image data of the next page when there are multiple pages of image data.

[0057] The masking release button 616 is a button for excluding a selected area from among the areas in the preview masking area 612 as a candidate for the masking area. The area selection button 617 is a button for selecting one of the areas in the preview masking area 612. When the display control unit 428 detects that one of the areas in the preview masking area 612 has been pressed while the area selection button 617 is selected, it recognizes that area as having been selected for the next operation. Subsequently, when the display control unit 428 detects that the masking release button 616 has been pressed, it hides the rectangular item corresponding to the selected area and excludes the selected area from the preview masking area 612.

[0058] The area instruction button 618 is a button for creating a new masking area candidate through a user operation. When the area instruction button 618 is selected, the display control unit 428 detects the point on the preview display area 611 touched with a finger as the start point and the point on the preview display area 611 where the finger is released as the end point, and sets the rectangular area specified by the start point and end point as a new masking area candidate. The display control unit 428 adds the rectangular area specified by the area instruction button 618 to the preview masking area 612, and displays a rectangular item superimposed on the preview masking area 612 according to the position coordinates of the rectangular area on the image data.

[0059] The display reduction button 619 is a button that reduces the display magnification of the preview display area 611 by a fixed amount and displays a reduced image. The display fit button 620 is a button that changes the magnification to the maximum amount at which the image data fits within the preview display area 611. The display enlargement button 621 is a button that increases the display magnification of the preview display area 611 by a fixed amount and displays an enlarged image.

[0060] The format designation form 622 is a form for allowing the user to select any format from the list of formats acquired in S505. The format designation form 622 displays the area to be masked for each format in the list of formats, and if it does not fit in the display area, a scroll bar is automatically displayed. In the list of formats, the format selected in S505 is selected, and other formats are unselected.

[0061] When the display control unit 428 detects pressing of any of the unselected formats in the format specification form 622, it places that format in a pre-selected state. Then, the display control unit 428 hides a rectangular item corresponding to the area to be masked in the selected format and transparently displays the rectangular item according to the position coordinates of the area to be masked in the pre-selected format. When the display control unit 428 detects pressing of a pre-selected format, it excludes the area to be masked in the selected format from the target of the preview masking area 612 and places that format in an unselected state. Then, the display control unit 428 selects the pre-selected format, adds the area to be masked in that format to the target of the preview masking area 612, and superimposes and displays a rectangular item according to the position coordinates of that area on the image data.

[0062] When a preselected format is present and the display control unit 428 detects a press on a format other than the preselected format, the display control unit 428 deselects the format and hides the rectangular item corresponding to the area to be masked in the format. The display control unit 428 then redisplays the rectangular item according to the position coordinates of the area to be masked in the selected format.

[0063] The attribute specification form 623 is a form for re-specifying the attribute specified in S501. The attribute specification form 623 displays an attribute specification checkbox and an attribute selection checkbox, which have the same functions as those shown on the main screen 600. When an attribute is selected by detecting a press on the attribute specification checkbox or the attribute selection checkbox, the display control unit 428 acquires the character string area of ​​the attribute using the character string attribute estimation result and displays the character string area as a target in the preview masking area 612. When the selection of an attribute is deselected, the character string area of ​​the attribute is excluded from the target in the preview masking area 612.

[0064] The correction specification form 624 is a form for allowing a user to specify the correction method used in S507. The correction specification form 624 has a pull-down menu, which displays the correction methods described with reference to FIG. 7B, namely, "logical sum," "circumscribed rectangle," and "reduction," as selectable options. The correction methods displayed in the pull-down menu are not limited to "logical sum," "circumscribed rectangle," and "reduction." Upon detecting the selection of a correction method from the pull-down menu, the display control unit 428 excludes the areas of the preview masking area 612 that were corrected in S507 from the preview masking area 612. The correction area extraction unit 425 then re-corrects the areas that were excluded from the preview masking area 612 using the selected correction method. The display control unit 428 displays the areas re-corrected by the correction area extraction unit 425 as the areas of the preview masking area 612.

[0065] Returning to the explanation of Fig. 5, in S509, when the main processing unit 421 detects that the print button 625 has been pressed, it determines each area in the displayed preview masking area 612 as a masking area. The print button 625 is a button that instructs the start of printing of a masked image in which the determined masking area is masked for the image data generated in S503.

[0066] In S510, the main processing unit 421 requests the image processing unit 422 to generate a masking image using the masking area determined in S509 and the image data generated in S503. The image processing unit 422 masks a rectangular area generated at the same position coordinates as the masking area on the image data generated in S503 to generate a masking image.

[0067] In S511, the main processing unit 421 requests the print instruction unit 429 to print the masking image generated in S509. The print instruction unit 429 executes printing of the masking image via the print execution unit 413. In this embodiment, the main processing unit 421 generates a printed matter of the masking image by printing, but the masking image may also be stored in the HDD 214 via the internal data storage unit 412. The main processing unit 421 may also store the masking image in the external storage 120 via the Internet access unit 426, or may send it by email to any destination. This completes the series of processes in this flowchart.

[0068] According to this embodiment, an image is subjected to extraction of a fixed area designated as a masking target (first extraction) and extraction of an area where a character string of an attribute to be masked appears (second extraction). The overlapping areas between the results of the first and second extractions can be corrected and extracted. In the first extraction, character strings that extend beyond the fixed area due to line breaks when filling in a form entry field may not be extracted. However, by correcting the results of the second extraction, such character string areas can be targeted for masking. In addition, in the second extraction, a portion of a character string of an attribute to be masked, such as a company name, may not be extracted due to being recognized as noise by OCR. However, by correcting the results of the first extraction, such character string areas can be targeted for masking. In this way, even when the area to be masked contains a mixture of fixed areas for each form type and areas that vary for each form, such as in a form that includes an address and name field and a free-text field, the area to be masked can be accurately extracted and masked.

[0069] [Embodiment 2] In the second embodiment, a method for displaying rectangular items that are displayed according to the position coordinates of each candidate masking area on the image data on the preview screen in a manner that allows them to be identified by extraction type will be described. Note that in describing this embodiment, explanations of parts that have the same configuration and processing procedures as the first embodiment will be omitted, and only parts that differ will be described.

[0070] FIG. 8A is a diagram illustrating a preview screen according to this embodiment. In this embodiment, in S508 of the flowchart in FIG. 5, the display control unit 428 generates the preview image shown in FIG. 8A and displays it on the touch panel display of the operation unit 220. The preview screen 800 includes a fixed area layer tab 801, a dynamic area layer tab 802, a correction area layer tab 803, and a manual area layer tab 804 in addition to the preview screen 610 in FIG. 6B. In this embodiment, each area in the preview masking area 612 is classified into layers according to type. That is, the first area extracted in S505 is classified into a fixed area layer, the second area extracted in S506 is classified into a dynamic area layer, the area corrected in S507 is classified into a correction area layer, and the area created by user operation is classified into a manual area layer. The display control unit 428 displays each area in the preview masking area 612 by grouping them by layer.

[0071] The fixed area layer tab 801 is a tab for prominently displaying areas classified as fixed area layers among the areas in the preview masking area 612. When the display control unit 428 detects a press on the fixed area layer tab 801, it recognizes that the fixed area layer is selected. Alternatively, when the display control unit 428 detects a press on any area among the areas in the preview masking area 612 and the area is the first area extracted in S505, it recognizes that the fixed area layer is selected. When the fixed area layer is selected, the display control unit 428 highlights areas classified as fixed area layers or suppresses the display of areas other than fixed area layers. Furthermore, when the fixed area layer is selected, the display control unit 428 displays the format specification form 622 and hides the attribute specification form 623 and the correction specification form 624. Alternatively, the display control unit 428 puts the format designation form 622 in a state where it can accept operations, and puts the attribute designation form 623 and the correction designation form 624 in a state where they cannot accept operations. In this way, when a fixed area layer is selected, the display control unit 428 highlights the areas classified as the fixed area layer, and displays the format designation form 622 for correcting the areas classified as the fixed area layer.

[0072] An example of a method for prominently displaying an area classified as a selected layer in this embodiment will be described using FIG. 8(b). A preview display area 810 in FIG. 8(b) shows a display example when a fixed area layer is selected. The preview display area 810 includes a fixed area 811, a dynamic area 812, and a correction area 813. The fixed area 811 is an example of the first area extracted in S505. The dynamic area 812 is an example of the second area extracted in S506. The correction area 813 is an example of the area corrected in S507. When the fixed area layer is selected, the display control unit 428 displays rectangular items in the fixed area 811 opaquely and rectangular items in the dynamic area 812 and the correction area 813 transparently. The display method is not particularly limited as long as it displays the fixed area 811 more prominently than the dynamic area 812 and the correction area 813.

[0073] Returning to the explanation of FIG. 8( a), the dynamic area layer tab 802 is a tab for prominently displaying areas classified as the dynamic area layer among the areas of the preview masking area 612. When the display control unit 428 detects a press on the dynamic area layer tab 802, it recognizes that the dynamic area layer is selected. Alternatively, when the display control unit 428 detects a press on any area in the preview masking area 612 and the area is the second area extracted in S506, it recognizes that the dynamic area layer is selected. When the dynamic area layer is selected, the display control unit 428 highlights areas classified as the dynamic area layer or suppresses the display of areas other than the dynamic area layer. Furthermore, when the dynamic area layer is selected, the display control unit 428 displays the attribute specification form 623 and hides the format specification form 622 and the correction specification form 624. Alternatively, the display control unit 428 puts the attribute designation form 623 in a state where it can accept operations, and puts the format designation form 622 and the correction designation form 624 in a state where they cannot accept operations. In this way, when a dynamic area layer is selected, the display control unit 428 highlights the areas classified as the dynamic area layer and displays the attribute designation form 623 for correcting the areas classified as the dynamic area layer.

[0074] The correction area layer tab 803 is a tab for prominently displaying areas classified as the correction area layer among the areas of the preview masking area 612. When the display control unit 428 detects a press on the correction area layer tab 803, it recognizes that the correction area layer is selected. Alternatively, when the display control unit 428 detects a press on any area among the areas of the preview masking area 612 and the area is corrected in step S507, it recognizes that the correction area layer is selected. When the correction area layer is selected, the display control unit 428 highlights areas classified as the correction area layer or suppresses the display of areas other than the correction area layer. When the correction area layer is selected, the display control unit 428 displays the correction specification form 624 and hides the format specification form 622 and the attribute specification form 623. Alternatively, the display control unit 428 makes the correction specification form 624 available for operation and makes the format specification form 622 and the attribute specification form 623 unavailable for operation. In this way, when a correction area layer is selected, the display control unit 428 highlights the area classified as the correction area layer and displays a correction specification form 624 for modifying the area classified as the correction area layer.

[0075] FIG. 8(c) shows a display example when the correction area layer is selected in this embodiment. FIG. 8(c) shows a display example of the preview display area when the correction area layer is selected. The preview display area 820 includes a fixed area 821, a dynamic area 822, a correction area 823, a fixed area within the correction area 824, and a dynamic area within the correction area 825. The fixed area 821 is an example of the first area extracted in S505. The dynamic area 822 is an example of the second area extracted in S506. The correction area 823 is an example of the area corrected in S507. The fixed area within the correction area 824 is an area resulting from the first area within the correction area 823. The dynamic area within the correction area 825 is an area resulting from the second area within the correction area 823. The display control unit 428 displays rectangular items in the fixed area 821 and the dynamic area 822 transparently.

[0076] Furthermore, the display control unit 428 displays the fixed area 824 within the correction area and the dynamic area 825 within the correction area in a distinguishable manner. For example, the fixed area 824 within the correction area and the dynamic area 825 within the correction area are displayed in different colors. In this way, the display control unit 428 highlights the correction area 823 and displays the fixed area and dynamic area included in the correction area 823 in a distinguishable manner. When the display control unit 428 detects a press on the fixed area 824 within the correction area or the dynamic area 825 within the correction area, it switches the display to the corresponding layer.

[0077] Returning to the description of FIG. 8A , the manual area layer tab 804 is a tab for prominently displaying areas classified as the manual area layer among the areas in the preview masking area 612. When the display control unit 428 detects a press on the manual area layer tab 804, it recognizes that the manual area layer is selected. Alternatively, when the display control unit 428 detects a press on any area in the preview masking area 612 and the area is created by a user operation, it recognizes that the manual area layer is selected. When the manual area layer is selected, the display control unit 428 highlights areas classified as the manual area layer or suppresses the display of areas other than the manual area layer. Furthermore, when the manual area layer is selected, the display control unit 428 hides the format specification form 622, the attribute specification form 623, and the correction specification form 624. Alternatively, the display control unit 428 disables the format specification form 622, the attribute specification form 623, and the correction specification form 624 from accepting operations.

[0078] The fixed area layer tab 801, the dynamic area layer tab 802, the correction area layer tab 803, and the manual area layer tab 804 may be switches instead of tabs, and each layer may be displayed superimposed.

[0079] According to this embodiment, on a preview screen that displays areas extracted as candidate masking areas superimposed on an image, the areas extracted as candidate masking areas are displayed separately for each extraction method type, thereby reducing the burden on the user of confirmation work. In addition, correction methods appropriate for each extraction method type are presented, making it easy for the user to correct areas extracted as candidate masking areas.

[0080] [Embodiment 3] In the third embodiment, a method for collecting data corresponding to areas extracted as candidates for masking areas for each extraction type will be described. In describing this embodiment, explanations of the same configurations and processing procedures as in the first embodiment will be omitted, and only differences will be described.

[0081] Fig. 9 is a diagram for explaining a preview screen according to this embodiment. In this embodiment, in S508 of the flowchart in Fig. 5, the display control unit 428 generates a preview image shown in Fig. 9 and displays it on the touch panel display of the operation unit 220. The preview screen 900 has a context menu 901 and a registration button 902 in addition to the preview screen 800 in Fig. 8(a).

[0082] The context menu 901 is a UI for duplicating a selected area as an area on a fixed area layer or an area on a dynamic area layer. When the display control unit 428 detects a press on any of the areas in the preview masking area 612, it displays the context menu 901 on the preview screen 900. Note that the display control unit 428 may display the context menu 901 when it detects a predetermined operation such as a long press or two consecutive touches rather than a simple press. The context menu 901 displays selectable items for instructing duplication as an area on a fixed area layer and for instructing duplication as an area on a dynamic area layer.

[0083] When one of the items in the context menu 901 is selected, the display control unit 428 duplicates the selected area according to the selected item. For example, when "Duplicate to Fixed Area Layer" is selected in the context menu 901, the display control unit 428 duplicates the selected area as an area in the fixed area layer. This allows areas that could not be extracted using one extraction method but could be extracted using another extraction method to be additionally collected as data for improving accuracy. For example, if the number of lines increases and an area that could not be extracted by the first area extraction in S505 can be extracted by the second area extraction in S506, information about that area can be collected and used as data for extraction by the first area extraction.

[0084] The register button 902 is a button for registering information about each area in the preview masking area 612. When the display control unit 428 detects pressing of the register button 902, it transmits information about each area in the preview masking area 612 to the external storage 120 via the Internet access unit 426, associating it with the layer type. The display control unit 428 may also store the information in the HDD 214 via the internal data storage unit 412. The transmitted areas are used for subsequent area extraction. For example, if the transmitted area belongs to a fixed area layer, a new format for masking the area can be created and added as a format usable in the first area extraction. This adds a format that includes areas that could not be extracted in the first area extraction, thereby improving extraction accuracy in subsequent areas. For example, if the transmitted area belongs to a dynamic area layer, it is possible to retrain a model used in the second area extraction to extract the area. The information transmitted to the external storage 120 may be used for any other purpose.

[0085] According to this embodiment, information on areas extracted in the second area extraction, which extracts areas of character strings with target attributes, and information on areas corrected using the areas extracted in the second area extraction, can be used as learning data used in the first area extraction, which extracts using a form format. Furthermore, information on areas extracted in the first area extraction and information on areas corrected using the areas extracted in the first area extraction can be used as learning data used in the second area extraction. This allows for improved accuracy of area extraction for each extraction type. Furthermore, when an area that could not be extracted in one extraction type is complemented with an area extracted in the other extraction type, the area can be corrected using this information, and the corrected information can be collected as learning data, which is expected to continuously improve the accuracy of area extraction.

[0086] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative examples of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features.

[0087] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0088] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first extraction means for extracting an area to be masked from a document image based on a predetermined format; a second extraction means for extracting a region of a character string to be masked from the document image; a third extraction means for extracting a third area based on the position coordinates of the first area extracted by the first extraction means and the position coordinates of the second area extracted by the second extraction means when the position coordinates of the first area overlap with the position coordinates of the second area extracted by the second extraction means; a determining means for determining an area to be masked in the document image based on the area extracted by the first extracting means, the area extracted by the second extracting means, and the area extracted by the third extracting means; An information processing device comprising: (Configuration 2) further comprising a display control means for controlling display of the area extracted by the first extraction means, the area extracted by the second extraction means, and the area extracted by the third extraction means in the document image as candidates for an area to be masked; The information processing device according to configuration 1, wherein the display control means makes it possible to distinguish whether the displayed area is an area extracted by the first extraction means, an area extracted by the second extraction means, or an area extracted by the third extraction means. (Configuration 3) the display control means controls to present a correction method for correcting the area displayed as a candidate for the area to be masked, The information processing device according to configuration 2, characterized in that when any of the areas displayed as candidates for areas to be masked is selected, the correction method presented varies depending on whether the selected area is an area extracted by the first extraction means, an area extracted by the second extraction means, or an area extracted by the third extraction means. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the third extraction means extracts, as the third region, a region obtained by performing a logical sum of the first region and the second region. (Configuration 5) The information processing device described in any one of configurations 1 to 3, characterized in that the third extraction means extracts an area obtained by a circumscribing rectangle that encompasses the first area and the second area as the third area. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the third extraction means excludes an area without a character string from the third area. (Configuration 7) The information processing device described in any one of configurations 1 to 6, characterized in that the third extraction means uses a method specified by the user from among a method of extracting an area obtained by the logical sum of the first area and the second area, a method of extracting an area obtained by a circumscribing rectangle that encompasses the first area and the second area, and a method of excluding areas that do not contain text from an area extracted by the logical sum or an area extracted by a circumscribing rectangle. (Configuration 8) The information processing device described in configuration 3, wherein the display control means displays a UI for correcting the format used by the first extraction means when the selected area is an area extracted by the first extraction means. (Configuration 9) The information processing device described in configuration 3 is characterized in that, when the selected area is an area extracted by the second extraction means, the display control means displays a UI for correcting attributes of the character string to be masked extracted by the second extraction means. (Configuration 10) The information processing device described in configuration 3, wherein the display control means displays a UI for correcting the extraction method used by the third extraction means when the selected area is an area extracted by the third extraction means. (Configuration 11) 11. The information processing apparatus according to any one of configurations 1 to 10, further comprising a masking means for generating a masked image by masking the area of ​​the document image to be masked that has been determined by the determining means. (Configuration 12) The information processing device according to configuration 2, wherein the display control means makes it possible to distinguish between areas resulting from the first area and areas resulting from the second area among the areas extracted by the third extraction means. (Configuration 13) further comprising a data collection means for collecting information on the area extracted by the first extraction means, information on the area extracted by the second extraction means, and information on the area extracted by the third extraction means, each of which is associated with a different group; 13. The information processing device according to any one of configurations 1 to 12, wherein the data collection means collects an area selected by a user as information on a group other than the group corresponding to the area. (method) A first extraction step of extracting a region to be masked from a document image based on a predetermined format; a second extraction step of extracting a region of a character string to be masked from the document image; a third extraction step of extracting a third area based on the position coordinates of the first area extracted in the first extraction step and the position coordinates of the second area extracted in the second extraction step when the position coordinates of the first area overlap with the position coordinates of the second area extracted in the second extraction step; a determining means for determining an area to be masked in the document image based on the area extracted in the first extraction step, the area extracted in the second extraction step, and the area extracted in the third extraction step; An information processing method comprising: (program) The computer of the information processing device, a first extraction means for extracting an area to be masked from a document image based on a predetermined format; a second extraction means for extracting a region of a character string to be masked from the document image; a third extraction means for extracting a third area based on the position coordinates of the first area extracted by the first extraction means and the position coordinates of the second area extracted by the second extraction means when the position coordinates of the first area overlap with the position coordinates of the second area extracted by the second extraction means; a determining means for determining an area to be masked in the document image based on the area extracted by the first extracting means, the area extracted by the second extracting means, and the area extracted by the third extracting means; A program characterized by functioning as [Explanation of symbols]

[0089] 110: MFP, 120: External storage

Claims

1. a first extraction means for extracting a region to be masked from a document image based on a predetermined format; a second extraction means for extracting a region of a character string to be masked from the document image; a third extraction means for extracting a third area based on the position coordinates of the first area extracted by the first extraction means and the position coordinates of the second area extracted by the second extraction means when the position coordinates of the first area overlap with the position coordinates of the second area extracted by the second extraction means; a determining means for determining an area to be masked in the document image based on the area extracted by the first extracting means, the area extracted by the second extracting means, and the area extracted by the third extracting means; An information processing device comprising:

2. a display control unit that controls display of the area extracted by the first extraction unit, the area extracted by the second extraction unit, and the area extracted by the third extraction unit in the document image as candidates for an area to be masked; 2. The information processing device according to claim 1, wherein the display control means makes it possible to distinguish whether the displayed area is an area extracted by the first extraction means, an area extracted by the second extraction means, or an area extracted by the third extraction means.

3. the display control means controls to present a correction method for correcting the area displayed as a candidate for the area to be masked, 3. The information processing device according to claim 2, wherein when any of the areas displayed as candidates for areas to be masked is selected, the correction method presented varies depending on whether the selected area is an area extracted by the first extraction means, an area extracted by the second extraction means, or an area extracted by the third extraction means.

4. 2. The information processing apparatus according to claim 1, wherein the third extraction means extracts, as the third region, a region obtained by performing a logical sum of the first region and the second region.

5. 2. The information processing apparatus according to claim 1, wherein the third extraction means extracts, as the third region, a region obtained by a circumscribing rectangle that includes the first region and the second region.

6. 2. The information processing apparatus according to claim 1, wherein the third extraction means excludes an area that does not contain a character string from the third area.

7. The information processing device according to claim 1, characterized in that the third extraction means uses a method specified by the user from among a method of extracting an area obtained by the logical sum of the first area and the second area, a method of extracting an area obtained by a circumscribing rectangle that encompasses the first area and the second area, and a method of excluding areas that do not contain text from the area extracted by the logical sum or the area extracted by the circumscribing rectangle.

8. 4. The information processing apparatus according to claim 3, wherein the display control means displays a UI for correcting a format used by the first extraction means when the selected area is an area extracted by the first extraction means.

9. The information processing device according to claim 3, characterized in that the display control means displays a UI for modifying attributes of the character string to be masked extracted by the second extraction means when the selected area is an area extracted by the second extraction means.

10. 4. The information processing apparatus according to claim 3, wherein the display control means displays a UI for modifying the extraction method used by the third extraction means when the selected area is an area extracted by the third extraction means.

11. 2. The information processing apparatus according to claim 1, further comprising a masking unit that generates a masked image by masking the area of ​​the document image to be masked that has been determined by the determining unit.

12. 3. The information processing device according to claim 2, wherein the display control means makes it possible to distinguish between an area resulting from the first area and an area resulting from the second area among the areas extracted by the third extraction means.

13. further comprising a data collection means for collecting information on the area extracted by the first extraction means, information on the area extracted by the second extraction means, and information on the area extracted by the third extraction means, each of which is associated with a different group; 2. The information processing apparatus according to claim 1, wherein the data collection means collects an area selected by a user as information on a group other than the group corresponding to the area.

14. a first extraction step of extracting a region to be masked from a document image based on a predetermined format; a second extraction step of extracting a region of a character string to be masked from the document image; a third extraction step of extracting a third area based on the position coordinates of the first area extracted in the first extraction step and the position coordinates of the second area extracted in the second extraction step when the position coordinates of the first area overlap with the position coordinates of the second area extracted in the second extraction step; a determining means for determining an area to be masked in the document image based on the area extracted in the first extraction step, the area extracted in the second extraction step, and the area extracted in the third extraction step; An information processing method comprising:

15. The computer of the information processing device, a first extraction means for extracting a region to be masked from a document image based on a predetermined format; a second extraction means for extracting a region of a character string to be masked from the document image; a third extraction means for extracting a third area based on the position coordinates of the first area extracted by the first extraction means and the position coordinates of the second area extracted by the second extraction means when the position coordinates of the first area overlap with the position coordinates of the second area extracted by the second extraction means; a determining means for determining an area to be masked in the document image based on the area extracted by the first extracting means, the area extracted by the second extracting means, and the area extracted by the third extracting means; A program characterized by functioning as

Images