Information processing device, method for controlling information processing device and program

Abstract

To enable masking processing to be performed on character strings corresponding respectively to items each different from the items of other classifications of image data.SOLUTION: An information processing device includes: identification means for identifying a classification of editing target image data from among a plurality of classifications of image data on the basis of recognition processing for identifying the classification of editing target image data; retrieval means for retrieving a character string corresponding to a category associated with the identified classification from among a plurality of character strings included in the editing target image data; and masking means for performing masking processing on the retrieved character string in the editing target image data. Therein each of the classifications of the plurality of image data is associated with each of different categories.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for performing a masking process on image data. [Background technology]

[0002] When sharing a document including information to be concealed, such as personal information or confidential information, with others, a technique for masking the area of ​​the document to be concealed is known. Examples of masking techniques include a method of generating a composite image by superimposing a mask image on the area to be concealed.

[0003] Patent Document 1 discloses a technology in which multiple character strings corresponding to multiple items set as targets for masking processing are searched for around the items that have been character-recognized on a form image, and masking processing is performed on the circumscribing rectangles of the multiple character strings. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-234344 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, it is not possible to perform masking processing on character strings that correspond to items that are set as targets for masking processing and that differ for each type of image data.

[0006] An object of the present invention is to make it possible to perform masking processing on character strings that correspond to different items for each type of image data. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device of the present invention comprises an identification means for identifying the type of image data to be edited from among multiple types of image data based on a recognition process for identifying the type of image data to be edited, a search means for searching for a character string corresponding to a category associated with the identified type from among multiple character strings contained in the image data to be edited, and a masking means for performing a masking process on the searched character string in the image data to be edited, and is characterized in that a different category is associated with each of the multiple types of image data. Effect of the Invention

[0008] According to the information processing device of the present invention, it is possible to provide a mechanism that enables masking processing to be performed on character strings corresponding to different items for each type of image data. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device that constitutes the present system. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an external storage that constitutes the present system. [Figure 4] FIG. 2 is a diagram illustrating an example of the software configuration of an information processing device that constitutes the present system. [Diagram 5] 2 is a flowchart illustrating the overall processing executed by the present system. [Figure 6] 10 is a flowchart illustrating a scan setting screen display process executed in the present system. [Figure 7] 11 is a flowchart illustrating a masking region specification process executed by the present system. [Figure 8] 1 is a flowchart illustrating a masking editing process executed by the present system. [Figure 9]11 is a flowchart illustrating a preset registration process executed in the present system. [Figure 10] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 11] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 12] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 13] 11 is a flowchart illustrating an information type acquisition process executed in the present system. [Figure 14] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 15] 11 is a flowchart illustrating an information type acquisition process executed in the present system. [Figure 16] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 17] FIG. 2 is a diagram showing an example of a screen configuration of the present system. [Figure 18] 11 is a flowchart illustrating a masking region specification process executed by the present system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention.

[0011] <First embodiment> <Overall composition> Fig. 1 is a diagram showing the overall configuration of this system. The image processing system in Fig. 1 includes an MFP (Multifunction Peripheral) 110 and an external storage 120. The MFP 110 is communicably connected to a server that provides various services on the Internet via a LAN (Local Area Network).

[0012] The MFP110 is a multifunction peripheral having a plurality of functions such as a scanner and a printer, and is an example of the information processing apparatus of the present invention. The MFP110 also has a function of transferring a scanned image file to a service capable of file storage such as an external storage service. Note that the information processing apparatus of the present invention is not limited to a multifunction peripheral having a scanner and a printer, and may be a PC.

[0013] The external storage (service) 120 is a service capable of storing a file received via the Internet or acquiring a file from an external device via a web browser. The external storage 120 is, for example, a cloud service. There are not only one external storage 120 but also a plurality of them.

[0014] The image processing system of the present embodiment is configured to include the MFP110 and the external storage 120, but is not limited thereto. For example, a part of the functions and processes of the MFP110 may be performed by another server arranged on the Internet or on a LAN. Also, the external storage 120 may be arranged on a LAN instead of on the Internet. Further, the external storage 120 may be replaced with a mail server or the like, and a scanned image may be attached to and transmitted by mail. The MFP110 may also have the storage function of the external storage 120.

[0015] <Hardware Configuration of MFP> FIG. 2 is a hardware configuration diagram of the MFP 110. The MFP 110 is composed of a control unit 210, an operation unit 220, a printer 221, a scanner 222, and a modem 223. The control unit 210 is composed of the following units 211 to 219, and controls the operation of the entire MFP 110. The CPU 211 reads out a control program stored in the ROM 212 or the HDD 214, and executes and controls various functions of the MFP 110, such as reading, printing, and communication. The RAM 213 is used as a temporary storage area such as the main memory and work area of ​​the CPU 211. Note that in this embodiment, one CPU 211 executes each process shown in a flowchart described later using one memory (RAM 213 or HDD 214), but is not limited to this. For example, each process may be executed by cooperating with multiple CPUs and multiple RAMs or HDDs. The HDD 214 is a large-capacity storage unit that stores image data and various programs.

[0016] The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The operation unit 220 is provided with a touch panel, a keyboard, and the like, and accepts operations / inputs / instructions from a user. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data for printing is transferred from the control unit 210 to the printer 221 via the printer I / F 216, and is printed on a recording medium. The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads an original placed on a platen or an ADF (Auto Document Feeder), not shown, to generate image data, and inputs the image data to the control unit 210 via the scanner I / F 217. The MFP 110 can print (copy) the image data generated by the scanner 222 from the printer 221, as well as transmit the image data as a file or by e-mail. The modem I / F 218 is an interface that connects the modem 223 and the control unit 210. A modem 223 communicates image data by facsimile with a facsimile device on the PSTN. A network I / F 219 is an interface that connects the control unit 210 (MFP 110) to a LAN. The MFP 110 uses the network I / F 219 to transmit image data and information to each service on the Internet and to receive various information.

[0017] <Hardware configuration of external storage> Figure 3 is a hardware configuration diagram of the external storage 120. The external storage 120 is composed of a CPU 311, a ROM 312, a RAM 313, an HDD 314, and a network I / F 315. The CPU 311 reads out the control program stored in the ROM 312 and executes various processes to control the overall operation. The RAM 313 is used as a temporary storage area such as the main memory and work area of the CPU 311. The HDD 314 is a large-capacity storage unit that stores image data and various programs. The network I / F 315 is an interface that connects the external storage 120 to the Internet. The external storage 120 receives a processing request from another device (such as the MFP 110) via the network I / F 315 and transmits and receives various information.

[0018] <Software Configuration of MFP> Figure 4 is a software configuration diagram of the image processing system according to this embodiment. The MFP 110 is roughly divided into two parts: a native function unit 410 and an additional function unit 420. Each function unit is realized by the CPU 211 reading out the program stored in the ROM 212 and HDD 214 of the MFP 110 into the RAM 213 and executing it. Each part included in the native function unit 410 is standardly provided in the MFP 110, while the additional function unit 420 is an application additionally installed in the MFP 110. The additional function unit 420 is an application based on Java (registered trademark), and can easily realize the addition of functions to the MFP 110. Note that other additional applications (not shown) may be installed in the MFP 110.

[0019] The native function unit 410 has a scan execution unit 411, an internal data storage unit 412, a print execution unit 413, and a UI (user interface) display unit 414. The additional function unit 420 has software modules 421 to 430. These are a main processing unit 421, a scan instruction unit 422, an image processing unit 423, a data management unit 424, a print instruction unit 425, an Internet access unit 426, a display control unit 427, an information detection unit 428, a document type determination unit 429, and a masking information type identification processing unit 430.

[0020] The main processing unit 421 has a function of performing overall processing for the additional function unit 420. Specifically, the main processing unit 421 controls the overall processing of the additional function unit 420, and requests each unit included in the additional function unit 420 to perform processing.

[0021] The scan instruction unit 422 requests the scan execution unit 411 to perform a scan process according to the scan settings input via the UI screen. The scan execution unit 411 receives a scan request including the scan settings from the scan instruction unit 422. In accordance with the scan request, the scan execution unit 411 generates scan image data by reading a document placed on a platen glass with the scanner 222 via the scanner I / F 217. The generated scan image data is sent to the internal data storage unit 412. The scan execution unit 411 sends an image identifier that uniquely identifies the stored scan image data to the scan instruction unit 422. The image identifier is a number, symbol, alphabet, or the like (not shown) for uniquely identifying an image scanned in the MFP 110. The internal data storage unit 412 stores the scan image data received from the scan execution unit 411 in the HDD 214.

[0022] The image processing unit 423 performs analysis and processing on the scanned image. The image processing unit 423 receives an image identifier from the scan instruction unit 422, and acquires scanned image data corresponding to the image identifier from the internal data storage unit 412. The image processing unit 423 performs image recognition processing such as character area analysis in the acquired image data, OCR (Optical Character Recognition), and image rotation and tilt correction. The image processing unit 423 also superimposes a mask image on a part of the scanned image (hereinafter referred to as a masking area) to synthesize the scanned image and the mask image, and instructs the internal data storage unit 412 to store the generated masked synthetic image data. In this embodiment, the masking area is a rectangular area to be masked in the scanned image, and is an area represented by information indicating the coordinates of the start point and end point of the rectangular area. For example, the masking area is represented by information indicating the coordinates of the start point and end point of the rectangular area, such as "(441, 957), (1369, 1057)". In this embodiment, the masking area is a rectangular area, but it may be any shape, such as an ellipse or a triangle. The mask image is an image that is synthesized to perform a masking process on the masking area on the scanned image data. The mask image may be an image filled with black paint, or an image of the background color of the scanned image. Any other image may be used as long as it can mask the character string included in the masking area in the scanned image. The image processing unit 423 sends an image identifier to the print instruction unit 425, the data conversion unit 426, and the Internet access unit 426 according to the output settings set for the image after the mask synthesis.

[0023] The data management unit 424 holds information such as the information type, document type, file name, and save destination set in the masking area identification process and the masking editing process described later in association with an image identifier. Also, the information type, document type, file name, and save destination set in the masking area identification process and the masking editing process are saved as presets in the HDD 214. A preset is a template in which the name of the preset, coordinates representing the masking area in the scanned image, character strings included in the masking area, information type, and information indicating the file name and save destination of the file including the scanned image are saved in association with the type of document. By selecting an item indicating the type of document displayed as a preset button on a screen 1000 described later, it is possible to perform processing using the settings saved in the template without making individual settings again.

[0024] The print instruction unit 425 sends a print processing request according to the print settings input via the UI screen and the image identifier received from the image processing unit 423 to the print execution unit 413. The print execution unit 413 receives the print request including the print settings and the image identifier from the print instruction unit 425. The print execution unit 413 obtains data from the internal data storage unit 412 corresponding to the image identifier and generates image data for printing according to the print request. The print execution unit 413 prints a mask composite image on a recording medium by the printer 221 via the printer I / F 216 according to the generated image data for printing.

[0025] The internet access unit 426 transmits a processing request to a cloud service or the like that provides a storage function (storage service). Cloud services generally use protocols such as REST and SOAP to publish various interfaces for saving files in cloud storage and retrieving the saved files from an external device. The internet access unit 426 operates the cloud service using the published interface of the cloud service. The internet access unit 426 acquires a file and transmission information corresponding to the image identifier received from the image processing unit 423 from the data management unit 424. The internet access unit 426 uses the transmission information acquired from the data management unit 424 to transmit the file acquired from the data management unit 424 to the external storage 120 via the network I / F.

[0026] The display control unit 427 displays a UI screen for accepting operations by a user on a liquid crystal display unit having a touch panel function of the operation unit 220 of the MFP 110. For example, it displays an operation screen for accepting operations for scan setting and scan start, a preview of a scanned image and a designation operation of a mask area described later, a preview of a mask composite image, and an operation for output setting and output start.

[0027] The information detection unit 428 classifies the character strings received from the image processing unit 423 into types such as "name" and "email address", detects character strings corresponding to types related to personal information and confidential information, and requests the data management unit 424 to store them in the HDD 214. In this embodiment, a category of character strings such as "name" and "email address" is called an information type. The information type is a type of concept expressed by a character string. The information type is used to specify a character string to be masked. In this embodiment, the detectable information types are items related to personal information and confidential information such as "name", "credit card number", and "email address", but are not limited thereto. As a result of the information detection process by the information detection unit 428, the information that can be obtained is the information type and a character string corresponding to the information type. For example, the character string corresponding to the information type "name" is "Yamada Taro". For example, the information detection unit 428 determines the information type using machine learning that learns the feature amount of the information type using sample character strings of multiple information types as teacher data. Alternatively, a table in which information types are associated with regular expressions of character strings corresponding to the information types is stored, and character strings that fit the associated regular expressions are detected as character strings corresponding to the corresponding information types. Alternatively, information types to be detected are stored in advance, and character strings located near the information types extracted in the image data to be detected are detected as character strings corresponding to the information types. In this embodiment, the information detection unit 428 holds items of predetermined detectable information types, but the location where the items are held is not limited to the information detection unit 428. For example, the above machine learning models, tables, etc. may be stored in the data management unit 424 or the external storage 120, and the information detection unit 428 may acquire the information from the data management unit 424 or the external storage 120 when detecting the information types.

[0028] The document type determination unit 429 infers the document type based on image processing of the scanned image. The document type is a classification of image data into several types based on the purpose of the document, such as "contract," "delivery note," and "bill." In this embodiment, the document type is determined by machine learning that learns the feature quantities of the document types using sample documents of multiple document types as training data. In this embodiment, a machine learning model based on Transformer is used, but is not limited to this. For example, a bidirectional Long Short-Term Memory (LSTM) neural network, a Sequence to Sequence model, a recurrent neural network (RNN), or the like may be used. Note that the method of determining the document type is not limited to this method. For example, a table that associates each document type with the layout of the ruled lines, character string blocks, etc. of the image data is stored in advance. The document type that has the highest degree of agreement between the layout of the ruled lines, character string blocks, etc. of the image data to be determined, analyzed by the image processing unit 423, and the stored layout may be determined as the type of the image data. Also, a table in which each document type is associated with a predetermined character string may be stored in advance, and the document type may be determined using the table. Specifically, a document type in which a stored predetermined character string exists in a character string contained in image data to be determined, the character string being recognized by the image processing unit 423, may be determined as the type of the image data. The machine learning model and table for determining the document type may be stored in the data management unit 424, or may be stored in an external storage, and the document type determination unit 429 may obtain them from the external storage as necessary.

[0029] The masking information type specification processing unit 430 determines the type of information to be masked based on the preset, the processing result of the information detection processing unit 428, and the processing result of the document type determination unit 429.

[0030] <Overall process flow> In the processing described below, the CPU 211 of the MFP 110 reads out a control program stored in the ROM 212 or HDD 214, and executes and controls various functions of the MFP 110 and functions of additional applications.

[0031] 5 is a flowchart for generating and outputting a mask composite image from an image scanned by the MFP 110. Examples of outputting the mask composite image include printing the mask composite image, converting the mask composite image into a file and sending it to cloud storage (external storage 120), or saving it in the internal data storage unit 412 of the MFP 110. In addition, both printing and sending, or both printing and saving may be performed. Note that, in this embodiment, an example will be described in which the display control unit 421 displays a screen on the touch panel of the operation unit 220, but a configuration in which the display control unit 421 provides each screen in this embodiment to another device, and the operation unit of the other device displays each screen, may also be used.

[0032] An additional application (hereinafter, referred to as a masking application) for generating a masked composite image in which a partial area of ​​an image of the present invention is masked can be used by installing it on the MFP 110. The masking application can transmit the generated masked composite image to a cloud service, store it on the MFP 110, or print it. When the masking application is installed on the MFP 110, a button for using the function of the application is displayed on the main screen of the MFP 110.

[0033] The flow shown in FIG. 5 is started when the display control unit 427 detects a user operation on a button for using a function of the masking application. In S501, the main processing unit 421 requests the display control unit 427 to display a scan setting screen of the masking application, and the display control unit 427 displays the scan setting screen 1400 on the touch panel of the operation unit 220. Then, when the main processing unit 421 receives a scan execution instruction on the scan setting screen 1400, it creates job information based on the selection received from the user. The job information includes the masking mode, masking type, masking character string, and masking area selected by the user. Any other information may also be included in the job information. Details will be described with reference to FIG. 6.

[0034] In S502, the main processing unit 421 requests the scan execution unit 411 to scan via the scan instruction unit 422, and acquires image data by scanning a document with the scanner 222, and stores the image data in the RAM 213. In this embodiment, the main processing unit 421 acquires image data by scanning, but is not limited to this. The main processing unit 421 may request the data management unit 424 to acquire image data from the HDD 214 via the internal data storage unit 412. The main processing unit 421 may also request the data management unit 424 to acquire image data from the external storage 120 via the Internet access unit 426. Image data may be acquired using any other means. When image data is acquired by a method other than scanning, the scan setting screen 1400 is not displayed. For example, the display control unit 427 displays a screen (not shown) for the user to select a method for acquiring an image before S501. The display control unit 427 accepts a selection from among acquiring image data by scanning, acquiring image data from the HDD 214, and acquiring image data from the external storage via the screen. If a selection to acquire image data by scanning has been accepted, the process proceeds to S501. If a selection to acquire image data from the HDD 214 or an external storage has been accepted, the display control unit 427 displays a screen for selecting a file of the image data to be acquired, instead of the scan setting screen 1400. For example, a list of folders in the HDD 214 or the external storage and the files contained in the folders is displayed, and a file selection is accepted from the user. Then, in S502, the main processing unit 421 acquires the file selected by the user.

[0035] In S503, the main processing unit 421 requests the image processing unit 423 to generate masking information from the job information acquired in S501 and the image data acquired in S502. The image processing unit 423 performs skew rotation correction on the image data acquired in S502 to generate corrected image data. Furthermore, the image processing unit 423 analyzes the corrected image data, compares it with the job information acquired in S501, and generates masking information. Details will be described with reference to FIG. 7.

[0036] In S504, the main processing unit 421 requests the image processing unit 423 to create a preview image by superimposing a mask image on the corrected image data, based on the corrected image data and masking information generated in S503. The image processing unit 423 creates a preview image from the corrected image and masking information. Next, the main processing unit 421 requests the display control unit 427 to generate and display a preview screen 1100 on which the created preview image can be viewed and modified as necessary. The display control unit 427 generates the preview screen, displays it on the touch panel of the operation unit 220, and modifies the masking based on the selection received from the user. Details will be described with reference to FIG. 8.

[0037] In S505, the main processing unit 421 creates a mask composite image and requests printing, saving, or transmission. Specifically, in response to an image output instruction on the preview screen to print, save, or transmit, a mask composite image is created based on the preview image, and the image is output. The mask composite image is an image in which the mask image set on the preview screen is combined with the corrected image data, and is an image that is combined so that it cannot be restored to the original corrected image data. In the case of printing, a request is made to the print execution unit 413 via the print instruction unit 425 to print the masking image created in S504, and the print execution unit 413 executes printing with the printer 221. In the case of saving, a request is made to the data management unit 424 to save the image in the HDD 214 via the internal data saving unit 412. In the case of transmission, the main processing unit 421 may request the data management unit 424 to save the image in the external storage 120 via the Internet access unit 426, or may send the image by email to an arbitrary destination. The image may be saved using any other means. The main processing unit 421 determines whether to print, transmit, or store the mask composite image based on the job information created in S501.

[0038] In S506, the main processing unit 421 requests the display control unit 427 to generate and display a preset registration screen 1200. The display control unit 427 displays the preset registration screen on the touch panel of the operation unit 220, and registers a preset based on a selection received from the user. Details will be described with reference to FIG.

[0039] FIG. 6 is a flowchart for explaining the scan setting screen display process in S501 in this embodiment.

[0040] In S601, the display control unit 427 generates a scan setting screen 1400 in this embodiment and displays it on the operation unit 220 via the UI display unit 414. An example of the scan setting screen is shown in FIG. 14. The scan setting screen 1400 is composed of an output method setting area 1410, a scan setting area 1420, a masking setting area 1430, a scan execution button 1401, and a back button 1402. The output method setting area 1410 is composed of an output method selection button 1411 and an output format selection button 1412. The output format selection button 1412 is an item for specifying a file format for saving that is valid when the setting of 1411 is set to "print and save" or "save only". The output method selection button 1411 is an item for specifying an output method of the masked scanned image. The user can select from print only, save only, or print and save as the output method. The scan setting area 1420 is an item for specifying scan settings, and accepts changes to the settings of "original size," "gray / color," "single-sided / double-sided," and "detailed settings." In "detailed settings," changes to detailed settings such as resolution, image quality, mixed original sizes, and density are accepted on a screen not shown. The scan execution button 1401 is a button for executing a scan using the scan settings set in the scan setting area 1420.

[0041] The masking setting area 1430 is composed of a masking selection button 1431, a masking method selection button 1432, and a masking mode selection button 1433. The masking selection button 1431 is a button for accepting a selection of whether or not to perform masking. The masking method selection button 1432 is a button for selecting a method of performing masking when the masking selection button 1431 is selected. It is possible to select whether to perform masking setting automatically or to perform masking setting manually. Performing masking setting automatically means that the type of the scanned image to be edited is automatically determined, and a character string corresponding to the information type stored in association with the determined type is automatically masked. A specific process will be described later. When the display control unit 427 detects that the masking method selection button 1432 has been selected as automatic, the display control unit 427 sets the mode displayed in the masking mode selection button 1433 to "automatic."

[0042] In this embodiment, the user can also select to manually set the masking settings using the masking selection button 1431. In S602, the display control unit 427 judges whether or not "Manual masking setting" is selected with the masking method selection button 1432. If "Manual" is selected with the masking method selection button 1432, in S603, the masking mode selection screen 1000 shown in FIG. 10(a) is popped up and the selection of the masking mode is accepted. Specifically, the masking mode selection screen 1000 is popped up on the scan setting screen 1400 displayed on the operation unit 220. Here, the masking mode refers to the type of method for specifying the masking area in this embodiment, and includes automatic, character string specification, information type selection, area specification, and preset selection. Character string specification is a mode in which a character string selected by the user among character strings recognized in the scanned image is masked. Information type selection is a mode in which a character string corresponding to the information type selected by the user among character strings recognized in the scanned image is masked. The area designation is a mode in which a rectangular area designated by the user on the preview screen is masked. The preset selection is a mode in which presets in which a masking area is designated are prepared in advance, the user selects one of the prepared presets, and masks according to the contents of the selected preset. The method of designating a masking area includes at least one of character string designation, information type selection, and area designation. The masking mode selection screen 1000 is composed of a masking mode list 1001. In this embodiment, the masking mode list 1001 is composed of a character string designation button 1002, an information type selection button 1003, an area designation button 1004, and a preset selection button 1005, but may include any other buttons.

[0043] In S604, the display control unit 427 determines which mode has been selected by the user. Then, in S605 to S608, the display control unit 427 transitions the masking mode selection screen 1000 displayed in the pop-up to a details screen corresponding to the selected masking mode. When the operation on the details screen is completed, the display control unit 427 closes the pop-up and displays the selected mode in the masking mode selection button 1433 on the scan setting screen 1400. Note that, although a pop-up display is performed in this embodiment, transition to another screen may be made, or all may be displayed in a pull-down menu to accept the selection.

[0044] The processing of S604 to S608 will be specifically described. When the display control unit 427 detects pressing of the character string designation button 1002 in S604, it determines that "character string designation" has been selected, and proceeds to S605 to transition the screen. If the masking mode is character string designation, the area of ​​the selected character string among the character strings detected in the OCR process in S702 described later is masked. In S605, the display control unit 427 generates a character string designation screen 1010 shown in FIG. 10(b), and transitions the pop-up display on the scan setting screen 1400 of the operation unit 220 from the masking mode selection screen 1000 to the character string designation screen 1010. The character string designation screen 1010 is composed of a character string input form 1011 and a completion button 1012. Any other method may also be included. The character string input form 1011 is a form for inputting a character string to be masked. For example, when masking the character string "product code XXX", the character string input form 1011 accepts the user's input of the character string. When the done button 1012 is pressed on the character string specification screen, the display control unit 427 stores the information set on the character string specification screen 1010 in the RAM 213. For example, when "Product Code XXX" is entered in the character string input form 1011, the display control unit 427 stores "character string specification" as the masking mode, "specified character string" as the masking type, and "Product Code XXX" as the masking character string. Note that since the coordinates of the masking area are not determined when a character string is specified, the information of the masking area is stored as empty. Then, the pop-up is closed, and the process returns to S602.

[0045] Also, in S604, when the display control unit 427 detects the pressing of the information type selection button 1003, it determines that "information type selection" has been selected, and proceeds to S606 to transition the screen. When the masking mode is information type selection, information detection in the flowchart of FIG. 7 described later is performed to mask the character string area corresponding to the relevant information type. In S606, the display control unit 427 acquires a list of predetermined detectable information types from the information detection unit 428, and generates an information type selection screen 1020 shown in FIG. 10(c). Next, the display control unit 427 transitions the pop-up display on the scan setting screen 1400 of the operation unit 220 from the masking mode selection screen 1000 to the information type selection screen 1020. The information type selection screen 1020 is composed of an information type list 1021 and a completion button 1012. In addition, any other configuration may be included. The information type list 1021 is an area that lists and displays information type items 1022 in the number of detectable information types. For example, when the detectable information type is "company name, name, telephone number, address", the information type list 1021 displays information type items of "company name, name, telephone number, address". When the information type item 1022 does not fit in the display area, the information type list 1021 may be configured to display a scroll bar so that the information type items can be scrolled. The information type item 1022 is composed of an information type name 1023 and a toggle button 1025. The information type name 1023 is an area for displaying the information type name. The toggle button 1025 is a button that is switched on when the information type displayed in the information type name 1023 is to be masked, and is switched off when the information type is not to be masked. For example, when the information type "company name" is to be masked, the toggle button corresponding to the information type "company name" is switched on when pressed by the user. In this way, the information type selection screen is a screen that accepts the detectable information type to be masked. When the complete button 1012 is pressed on the information type selection screen 1020 , the display control unit 427 stores the information set on the information type selection screen 1020 in the RAM 213 .Specifically, the display control unit 427 saves "information type selection" as the masking mode, and all information types for which the toggle button 1025 is turned on in the information type list 1021 as the masking types. For example, when the toggle button for the information type "company name" is turned on in the information type list 1021, the display control unit 427 saves the information type "company name" as the masking type. Note that when an information type is selected, the masked character string and the coordinates of the masking area are not determined, so the information on the masking character string and the masking area is saved as blank. Then, the pop-up is closed, and the process returns to S602.

[0046] Also, in S604, when the display control unit 427 detects the pressing of the area designation button 1004, it determines that "area designation" has been selected, and proceeds to S607 to transition the screen. When the masking mode is area designation, the display control unit 427 obtains the coordinates of a rectangular area in the flowchart of FIG. 8 described later, and masks the rectangular area selected in response to the user's pressing. In S607, the display control unit 427 generates an area designation screen 1030 shown in FIG. 10(d), and displays the area designation screen 1030 as a pop-up on the scan setting screen 1400 of the operation unit 220. The area designation screen 1030 is composed of a message 1031 and a scan button 1012. It may also include any other configuration. The message 1031 is an area that displays a message prompting the user to scan. When the completion button 1012 is pressed on the area designation screen 1030, the display control unit 427 saves "area designation" as the masking mode in the RAM 213, closes the pop-up, and returns to S602. In the area specification, the user specifies the area to be actually masked after acquiring the image data, so the masking type, masking area, and masking character string are not yet determined in S607. Therefore, the information on the masking type, masking area, and masking character string is saved as blank.

[0047] Also, in S604, when the display control unit 427 detects pressing of the preset selection button 1005, it determines that "preset selection" has been selected, and proceeds to S608 to transition the screen. When the masking mode is preset selection, a masking area is determined and masked according to the contents of the selected preset in the flowchart of FIG. 7 described later. In S608, the display control unit 427 requests the data management unit 424 to acquire a list of registered presets. The data management unit 424 acquires the preset list from the HDD 214 via the internal data storage unit 412 and passes it to the display control unit 427. The data management unit 424 may also acquire the preset list from the external storage 120 via the Internet access unit 426, or may acquire the list using any other means. The display control unit 427 extracts each preset name from the acquired preset list, generates a preset selection screen 1040 shown in FIG. 10(e), and displays the preset selection screen 1040 as a pop-up on the scan setting screen 1400 of the operation unit 220. The preset selection screen is a screen that displays a list of presets and accepts the selection of any preset. The preset selection screen 1040 is composed of a preset list 1041. The preset list 1041 is an area that displays a list of preset buttons 1042 in the same number as the number of presets in the preset list. For example, if the preset list is a preset of "invoice, proposal, contract, license, public release", the preset list 1041 displays the preset buttons of "invoice, proposal, contract, license, public release". The preset list 1041 may be configured to display a scroll bar so that the preset buttons 1042 can be scrolled when they do not fit within the display area. The preset buttons 1042 are buttons for selecting presets. The preset buttons 1042 display the names of the presets.

[0048] When the user presses any of the preset buttons in the preset list 1041, the display control unit 427 generates a preset details screen 1050, 1060, or 1070 shown in FIG. 10(f), (g), or (h) from the contents of the preset. Next, the display control unit 427 pops up the preset details screen 1050, 1060, or 1070 on the scan setting screen 1400 of the operation unit 220. The preset details screen is a screen that displays the contents of a selected preset. For example, when the "contract" preset is selected, a preset details screen that displays the contents of the "contract" preset is generated.

[0049] When the preset content is a masking character string, the display control unit 427 generates a preset details screen 1050 for character string designation shown in Fig. 10(f). When the preset content is a masking type, the display control unit 427 generates a preset details screen 1060 for information type selection shown in Fig. 10(g). When the preset content is a masking area, the display control unit 427 generates a preset details screen 1070 for area designation shown in Fig. 10(h). When the preset content is a composite content including a masking character string, a masking type, and a masking area, each preset details screen may be displayed so as to be switched.

[0050] The preset details screen 1050 is composed of a preset name 1051, a character string input form 1011, and a scan button 1012. The preset name 1051 displays the name of the preset. A masking character string for the preset is input by default into the character string input form 1011. The character string input form 1011 may also receive input from the user and allow the user to freely edit the masking character string. When the complete button 1012 is pressed on the preset details screen 1050, the display control unit 427 holds the settings input on the preset details screen 1050 in the RAM 213, closes the pop-up, and returns to S602.

[0051] The preset details screen 1060 is composed of a preset name 1051, a character string input form 1011, an information type list 1021, and a scan button 1012. In the information type list 1021, the toggle button of an information type item 1022 that corresponds to the masking type of the preset is switched on by default. If not, the toggle button is switched off by default. In addition, the toggle button may be freely edited by the user's press. When the completion button 1012 is pressed on the preset details screen 1060, the display control unit 427 holds the settings input on the preset details screen 1060 in the RAM 213. Specifically, the display control unit 427 saves all information types in the information type list 1021 for which the toggle button 1025 is on as masking types. Then, the popup is closed, and the process returns to S602.

[0052] The preset details screen 1070 is composed of a preset name 1051, a preset preview 1071, and a scan button 1012. The preset preview 1071 is composed of an image outline preview 1072 and a masking area 1073. The image outline preview 1072 displays the outline of the image from the image size included in the preset. The masking area 1073 displays the area to be masked from the masking area included in the preset within the same coordinate position on the image outline preview 1072. The preset preview 1071 displays the area to be masked in the image by displaying the image outline preview 1072 and the masking area 1073. When the complete button 1012 is pressed on the preset details screen 1070, the display control unit 427 holds the masking area of ​​the preset set on the preset details screen 1070 in the RAM 213. For example, if the preset masking area is "(441,957), (1369,1057)", the masking type is saved as "rectangle area" and the masking area is saved as "(441,957), (1369,1057)". Then, the popup is closed and the process returns to S602.

[0053] If the preset contents are complex and the preset details screen is switched and displayed, pressing the scan button 1012 causes the display control unit 427 to generate job information and add the masking mode confirmed in S603 to the job information. The display control unit 427 then stores the contents set in all of the preset details screens in the RAM 213, closes the pop-up display, and returns to S602.

[0054] In S609, it is determined whether or not the scan button 1401 has been pressed. When the display control unit 427 detects the pressing of the scan button 1401 and determines that it has been pressed, the process proceeds to S610 and generates job information. Specifically, the display control unit 427 generates job information based on information set on the scan setting screen 1400 and information set in S605 to S608 and held in the RAM 213. The job information includes information on the masking mode selected by the user, the masking character string, the masking information type, and the masking area.

[0055] FIG. 7 is a flowchart for explaining the masking area specification process of S503 in this embodiment. The flowchart in FIG. 7 shows the process when the masking mode is set to "automatic" in S501. When the masking mode is set to "character string specification", "information type selection", "area specification", or "preset selection", some of the processes are omitted. For example, when the masking mode is selected to "character string specification" and the masking type included in the job information is empty, that is, when the user manually specifies the coordinates of the masking area after acquiring the image data, S702, S703, and S704 may be skipped. Also, when the masking mode is "preset selection" and the masking type included in the job information is only a rectangular area, S702 and S703 may be skipped. Whether or not some of the processes are omitted is determined by the main processing unit 421 based on the job information generated in S501.

[0056] In S701, the image processing unit 423 performs skew correction and rotation correction on the image data acquired in S502, and stores the corrected image data in the RAM 213.

[0057] In S702, the image processing unit 423 converts the corrected image data into a one-channel image to improve OCR accuracy, performs OCR (character recognition processing), and stores the extracted character string and the character string area corresponding to the character string in the RAM 213. Specifically, the character string and coordinates indicating the position of the character string area are associated with each other and stored in the RAM 213. The character string area corresponding to the character string is a circumscribed rectangle of the character string.

[0058] In S703, the information detection unit 428 performs information detection processing from the character string extracted in S702, and stores in the RAM 213 the information types that the information detection unit 428 can detect and the character strings corresponding to the detected information types. An example of the information detection processing result in this embodiment is shown in the information detection processing result information list. In the information detection processing result information list, the information types are "company name", "name", "telephone number", and "address". The character strings corresponding to the detected information types are "XXX Co., Ltd." as the character string corresponding to the company name, "Kiyano" and "Tanaka" as the character string corresponding to the name, and "△△△-XXX-□□□□" as the character string corresponding to the telephone number. "- (none)" in Table 1 indicates that no character string corresponding to the address was detected. In this embodiment, the information types that the information detection unit 428 can detect are "company name", "name", "telephone number", and "address", but other character strings may also be detected. For example, "credit card number", "email address", "date and time", etc. can be given. Note that similar information detection may be performed on an external device or an external cloud service, rather than on the information detection unit of the MFP 110. In step S703, the type of information and the character string detected externally may be acquired.

[0059] [Table 1]

[0060] In S704, the document type determination unit 429 performs recognition processing to identify the document type. For example, the document type is identified from the character string extracted in S702. In this embodiment, the document type is determined using a machine learning model that has been trained in advance, but the document type may be determined by setting a keyword corresponding to the document type in advance. Alternatively, the document type may be determined from a layout based on ruled lines, character string blocks, etc. of the image data, without using the character string extracted in S702.

[0061] In S705, the masking information type identification processing unit 430 judges whether or not the document type was identified in S704. If the document type was identified (YES in S705), the process proceeds to S706, and if the document type was not identified (NO in S705), the process ends.

[0062] In S706, the masking information type identification processing unit 430 requests the data management unit 424 to acquire a preset corresponding to the determined and acquired document type, and stores the information type to be masked in the acquired preset in the RAM 213. The preset may be acquired from the HDD 214 or from the external storage 120. Details will be described with reference to FIG. 13.

[0063] FIG. 13 is a flowchart for explaining the process of acquiring an information type corresponding to a document type in S706 in this embodiment.

[0064] In S1301, the masking information type identification processing unit 430 acquires a preset corresponding to the document type from the external storage 120 via, for example, the Internet access unit 426, and stores it in the RAM 213. Here, a preset is an information type that is often targeted for masking for each document type, and is stored in association with the document type. The information type and document type may be registered in advance by the user on a separately prepared registration screen, or may be set on a preset registration screen (described later) after the mask composite image is output.

[0065] Examples of presets stored in the external storage 120 or HDD 214 are shown in the preset information list of Table 2. As shown in Table 2, different information types are associated with each document type and stored. By preparing presets for multiple document types in this way, masking processing for the information type corresponding to the document type can be automatically performed.

[0066] [Table 2]

[0067] In S1302, the masking information type identification processing unit 430 judges whether or not a preset corresponding to the document type determined in S704 exists among the acquired presets for each document type. If a preset exists (YES in S1302), the process proceeds to S1303, and if no preset exists (NO in S1302), the process ends this flow assuming that there is no information type to be masked.

[0068] In S1303, the masking information type identification processing unit 430 extracts information on the information type to be masked that is linked to the preset corresponding to the document type determined in S704 from the preset information acquired in S1301, stores the information in the RAM 213, and ends this flow. For example, if the information type is determined to be "invoice" in S704, the information types to be masked are "name", "telephone number", and "address". Also, if the information type is determined to be "proposal" in S704, the information types to be masked are "company name", "name", and "date".

[0069] In S707, the image processing unit 423 compares the information type detected in S703 with the information type of the masking target acquired in S706, and identifies the matching information type. Then, among the information types detected in S703, a character string area of ​​a character string corresponding to the matching information type is determined as a masking area. Then, the image processing unit 423 stores the masking information and the document type in the RAM 213. The masking information is composed of the information type and the character string corresponding to the information type, which are the information detection processing results, the necessity of masking indicating whether the information type is the masking target acquired in S706, and the position (coordinates) of the masking area. An example of the masking information in this embodiment is shown in the masking information list of Table 3. Table 3 is a masking information list when it is determined to be "invoice" in S704. The masking necessity of the information types of "name", "telephone number", and "address" to be masked is "necessary". Then, among the multiple character strings included in the image data to be edited obtained by the OCR processing in S702, character strings with the information types of "name", "telephone number", and "address" are searched for. Specifically, the information detection unit 428 searches using the information detection process result detected in S703. In the information detection process, character strings detected as character strings corresponding to "name", "telephone number", and "address" are searched for. That is, "Kiyano", "Tanaka", and "△△△-○○○○-□□□□" are searched for. Then, based on the correspondence between character strings and character string areas obtained by the OCR process in S702, the coordinates of the searched character string area are specified. The positions of the areas included in the masking information list are character string areas of character strings corresponding to the information types of "company name", "name", and "telephone number", respectively. The character string area of ​​the character string corresponding to "address" is not specified because a character string corresponding to "address" is not detected and is "- (none)", and the position of the character string area cannot be obtained. If it is determined to be "proposal" in S704, the masking necessity of "company name" and "name", which are the information types to be masked among the information types detected in S703, is set to "needed".

[0070] [Table 3]

[0071] FIG. 8 is a flowchart for explaining the masking editing process of S504 in this embodiment.

[0072] In S801, the image processing unit 423 acquires the corrected image data acquired in S701 and the masking information created in S707 or S805 from the RAM 213. The image processing unit 423 acquires the position (start point / end point coordinates) of the area that needs to be masked or not from the acquired masking information, and generates a preview image in which a mask image is superimposed on a rectangular area specified by the coordinate position on the corrected image data. The image processing unit 423 holds the generated preview image in the RAM 213. The preview image data generated here is the mask image superimposed on the original corrected image data acquired in S701, and also includes data of the masked area on the corrected image data. In other words, the mask image is generated so that it is possible to cancel the mask image on a preview screen described later and return to the original corrected image data.

[0073] In S802, the display control unit 427 acquires the preview image generated in S801 from the RAM 213, generates the preview screen 1100 of FIG. 11, and displays it on the touch panel via the UI display unit 414. The preview screen 1100 is a diagram showing an example of a display of a preview image related to the present invention. The preview screen 1100 is composed of an image display area 1110, a document type display area 1120, a masking information display area 1130, a print button 1114 for confirming the masking process on the preview image being displayed and specifying printing, and a setting save button 1115. The setting save button 1115 will be described in detail later with reference to FIG. 9. The image display area 1110 is composed of 1101 to 1109 and 1111 to 1113. The preview image display area 1101 is an area for displaying the preview image generated in S801. If the preview image does not fit on the screen, a scroll bar is automatically displayed. The preview display enlargement button 1102 and the preview display reduction button 1104 are buttons for the user to specify the enlargement or reduction of the preview image displayed in the preview image display area 1101. 1105 and 1106 are mask images. The preview display fit button 1103 is a button for accepting an instruction from the user to determine the enlargement or reduction ratio of the preview image so that the preview image fits exactly in the preview image display area 1101 and display the preview image. The page number display 1108 is an indication showing the page number of the preview image displayed in the preview image display area. The previous page button 1107 is a button for accepting an instruction from the user to display the preview image of the previous page. The next page button 1109 is a button for accepting an instruction from the user to display the preview image of the next page. The masking deletion button 1111 is a button for accepting an instruction from the user to delete the mask image. The masking instruction button 1112 is a button for the user to instruct to perform a masking area on the preview image. The masking selection button 1113 is a button for the user to instruct selection of a mask image.

[0074] The enlarge preview display button 1102 is a button for enlarging the display magnification of the preview image displayed in the preview image display area 1101 by a fixed amount. The fit preview display button 1103 is a button for changing the display magnification of the preview image to the maximum magnification at which the preview image fits within the preview image display area 1101. The reduce preview display button 1104 is a button for reducing the display magnification of the preview image display area by a fixed amount.

[0075] A previous page button 1107 is a button for displaying the previous page of the scanned image when there are multiple pages of scanned images. A page number display 1108 displays the page of the scanned image currently being displayed and the total number of pages. A next page button 1109 is a button for displaying the next page of the scanned image when there are multiple pages of scanned images.

[0076] The masking deletion button 1111 is a button for canceling the mask image selected in the preview image display area 1101. When the display control unit 427 detects that the mask image displayed as the masking target is touched with a finger on the preview image while the masking selection button 1113 is selected, the display control unit 427 recognizes that the touched mask image is selected for the next operation. Then, when the display control unit 427 detects the pressing of the masking deletion button 1111 following the selection, the display control unit 427 sets the masking necessity of the masking information for the masking area corresponding to the selected mask image to "no." The image processing unit 423 creates a preview image based on the updated masking information, and updates the preview image display area 1101.

[0077] When the masking instruction button 1112 is selected, the display control unit 427 detects the location on the preview image touched by the finger as the start point. That is, when the display control unit 427 detects a user operation, it acquires coordinate information of the detected location in the preview image. Next, the display control unit 427 detects the end point where the finger is removed after dragging. That is, when the display control unit 427 detects that the user's finger has been removed from the panel, it acquires coordinate information of the location where the removal was detected in the preview image. Then, the display control unit 427 recognizes a rectangular area specified by the coordinates of the start point and the coordinates of the end point as a masking area. Then, the display control unit 427 updates the preview image by overlaying a mask image on the recognized masking area.

[0078] The document type display area 1120 is configured with a document type selection drop-down list 1121, and displays the document type determination result of S704. The document type selection drop-down list 1121 accepts the document type selected by the user. If the document type determination result cannot be obtained, the document type is displayed as blank, indicating "not set" (not shown).

[0079] The masking information display area 1130 is an area that displays information about masking displayed in the preview image display area 1110. After scanning, on the preview screen that is first displayed in S802, the display control unit 427 displays masking information corresponding to the document type displayed in the document type display area 1120 in the masking information display area 1130. If the document type is not set in the document type display area 1120, the masking information display area 1130 does not display masking information because no masking information exists. If the document type is specified by the user in the document type selection drop-down list 1121, the masking information type identification processing unit 430 acquires the information type corresponding to the document type (the same process as in S706) and generates masking information. The masking information display area 1130 displays the information type and the character string corresponding to the information type detected in S703 in a tree structure. It is displayed so that the parent is the information type and the child is the character string corresponding to the information type. That is, for each of a plurality of information types, character strings corresponding to one or more information types classified into the respective information types are displayed in association with each other.

[0080] Furthermore, the masking information display area 1130 has a check box that accepts whether or not masking is required by tapping for each information type and character string corresponding to each information type. When the check box is selected, the masking is required, and when the check box is deselected, the masking is not required. The display control unit 427 accepts a designation for selecting or deselecting each information type or character string corresponding to each information type from the user via the check box. In this embodiment, the check box for the information type for which masking is required and for which the position of the area exists and the character string corresponding to the information type are displayed in a state in which they are selected in advance with a check mark. When a check box is pressed, the masking of the corresponding masking information is changed. When the check box is pressed when the masking is required, the masking is changed to "not required", and when the check box is pressed when the masking is not required, the masking is changed to "required".

[0081] For example, the display control unit 427 accepts an instruction to cancel the selection of "Name" by pressing the check box of "Name" in a state where "Name" is selected as shown in FIG. 11. Then, the display control unit 427 changes the necessity of masking corresponding to "Name" in the masking information from "Needed" to "No". Then, the display control unit 427 displays the character string corresponding to "Name" without masking in the preview image 1101 displayed on the image display area 1110. That is, the mask image displayed superimposed on the character strings "Kiyano" and "Tanaka" corresponding to "Name" is deleted. Also, the display control unit 427 accepts an instruction to select "Company Name" by pressing the check box of "Company Name" in a state where "Company Name" is not selected as shown in FIG. 11. Then, the display control unit 427 changes the necessity of masking corresponding to "Company Name" in the masking information from "No" to "Needed". Then, the display control unit 427 masks and displays the character string corresponding to the "company name" in the preview image 1101 displayed on the image display area 1110. That is, the mask image is displayed so as to be superimposed on the character string "XXX Co., Ltd." corresponding to the "company name".

[0082] The display control unit 427 also displays information types for which masking is "necessary" at the top to improve visibility, but the display order is not important. Furthermore, information types for which masking is "not required" based on the masking information are displayed with the checkboxes unselected. Information types for which masking is "not required" are not included in the presets, but the information types detected by the information detection process are personal information or confidential information, and since there is a high possibility that masking will be performed depending on the user, they are displayed for the purpose of presenting them to the user.

[0083] A warning mark 1131 is a display that warns the user about an item for which masking is "necessary" but no area exists. The area position edit button allows the position of the area to be set in the masking information. Area position edit buttons 1132, 1133, 1134, 1135, and 1136 are buttons for editing the area position. As with the masking instruction button 1112, a rectangular area is accepted and linked to the corresponding item to update the masking information. In the masking information list, the information type "address" is "necessary" for masking but the area position has not been acquired. When it is desired to specify an area for this, the area position edit button 1135 is pressed and a rectangular area is selected. The rectangular area is updated as the area position of the item corresponding to the information type "address" as the masking information list after the masking area is added.

[0084] [Table 4]

[0085] In S803, the display control unit 427 judges whether or not a user operation is detected on the preview screen 1100 via the UI display unit 414. If a user operation on the preview screen 1100 is detected, the process proceeds to S811. If a user operation is not detected, the process repeats S803 and waits for a user operation. In S811, the display control unit 427 judges whether or not the user operation detected in S803 is a user operation to press the print button 1114. If it is judged that the user operation is a user operation to press the print button 1114, the process ends this flow. If it is judged that the user operation is not a user operation to press the print button 1114, the process proceeds to S812. In S812, the display control unit 427 judges whether or not the user operation detected in S803 is a user operation to select a document type. That is, it judges whether or not a document type is reselected by the document type selection dropdown list 1121. If it is judged that the user operation is a user operation to select a document type, the process proceeds to S707. If it is judged that the user operation is not a user operation to select a document type, the process proceeds to S813. S707 is the same as the process of S707 described in FIG. 7. In S813, the display control unit 427 judges whether the user operation detected in S803 is an operation of instructing to add or delete a masking region. The operation of instructing to add a masking region is, for example, an operation of pressing the masking instruction button 1112 or the region position edit buttons 1132 to 1136 to specify a region on the preview image 1101. Also, it is an operation of selecting a check box in the masking information display area 1130. The operation of deleting a masking region is an operation of pressing the masking selection button 1113 to select a mask image displayed on the scan image 1101, and pressing the masking deletion button 1111. If it is judged that the user operation detected in S803 is an operation of adding or deleting a masking region, the process proceeds to S805. If it is judged that the user operation detected in S803 is not an operation of adding or deleting a masking region, the process proceeds to S802, where the preview screen is updated and displayed according to the user operation in S803. For example, if the user operation detected in S803 is a user operation of pressing the preview display enlargement button 1102, the preview image is enlarged and the preview screen is displayed in S802.In S805, the display control unit 427 acquires the coordinates of the masking area instructed to be added or deleted, returns to S801, and adds or deletes the mask image according to the instruction. Specifically, the display control unit 427 detects that the masking instruction button 1112 or the area position editing buttons 1132 to 1136 is pressed and an operation of specifying an area on the preview image 1101 is performed. Then, the display control unit 427 acquires a rectangular area selected on the touch panel via the UI display unit 414, and converts it into a rectangular area on the masking image displayed as a preview image. The display control unit 427 acquires masking information of the image data to be edited, and adds the rectangular area converted in S805 to the masking information. The masking type is "rectangular area", the character string corresponding to the information type is "- (none)", the masking necessity is "necessary", and the rectangular area converted in S805 as the area position (coordinates of the start point and end point) is added to the masking information, and is saved in the RAM 213. An example of a masking information list in which a masking region has been added to the masking information in this way is shown below.

[0086] [Table 5]

[0087] FIG. 9 is a flowchart for explaining the preset registration process of S506 in this embodiment. In S901, the display control unit 427 judges whether a preset has been used in the masking process. Specifically, the display control unit 427 judges whether the masking mode included in the job information is preset selection or automatic. If the masking mode is preset selection or automatic (YES in S901), the process proceeds to S902, and if the masking mode is not preset selection or automatic (NO in S901), the process proceeds to S903. In S902, the display control unit 427 judges whether the preset selected in S607 has been edited in response to a user's input. That is, the display control unit 427 judges whether the user has edited the preset on the preset details screen shown in FIG. 10(f), (g), and (h), and whether the user has edited the masking area on the preview screen 1100. For example, it is sufficient to judge whether the final masking information matches the masking information of the selected preset. If the preset has been edited (S902: YES), the flow proceeds to S903, and if the preset has not been edited (S902: NO), the flow ends. In S903, the display control unit 427 displays on the operation unit 220 a screen (not shown) for confirming with the user via the UI display unit 414 whether or not to register a preset.

[0088] In S904, the display control unit 427 judges whether or not the user has selected to register the mask as a preset. If registration has been selected (YES in S904), the process proceeds to S905, and if registration has not been selected (NO in S904), the process ends this flow. In S905, the display control unit 427 requests the UI display unit 414 to display a preset registration screen on the operation unit 220. The preset registration screen is a screen on which the masking information implemented in this embodiment can be registered as a preset. If the masking type of the masking information is only the information type, the display control unit 427 generates a preset registration screen for selecting the information type. If the masking type of the masking information is only the rectangular area, the display control unit 427 generates a preset registration screen (not shown) for area designation.

[0089] FIG. 12 is an example of a preset registration screen displayed on the operation unit 220 by the display control unit 427 in S905, and is a preset registration screen for selecting an information type. The preset registration screen 1200 is composed of a preset name display area 1201, a document type selection area 1220, a registration information display area 1230, a save button 1203, and a cancel button 1204. The preset name display area 1210 displays the name of the preset to be registered and accepts correction of the preset name. By default, a character string in which the name of the document type selected in the document type selection area 1220 described later and a serial number are connected with an underscore is set, but the user may change it to an arbitrary preset name. The document type selection area 1220 accepts selection of the document type to be registered. By default, the document type displayed in the document type display area 1120 is set. The registration information display area 1230 is an area for displaying the masking information displayed in the masking information display area 1130. The registration information display area 1230 is composed of display areas 1231, 1232, 1233 for displaying the names of the information types to be registered, and toggle buttons 1235, 1236, 1237 for selecting whether or not to include the information type in the preset. When the toggle button is ON, the information type is registered as a target for masking in the preset, and when the toggle button is OFF, the information is not registered in the preset. The initial display of the toggle button in the registration to preset screen 1200 is set according to whether or not masking of the masking information is required, similar to the display of the masking information display area 1130. The save button 1203 is a button for confirming the registration of the information displayed in 1210 to 1230 as a preset. The cancel button 1204 is a button for ending this embodiment without performing the preset registration process.

[0090] In S906, the display control unit 427 requests the data management unit 424 to save the preset confirmed in S905. The data management unit 424 may save the preset in the external storage 120 via the Internet access unit 426, or may save the preset in the HDD 214 via the internal data storage unit 412. Here, if a preset with the same name is saved in the HDD 214 or the external storage 120, the display control unit 427 may request the operation unit 220 to display a warning message inquiring the user whether or not to overwrite the saved preset. Furthermore, if the preset registration has been normally completed, the display control unit 427 may request the operation unit 220 to display a message indicating that the preset registration has been completed.

[0091] The preset registration can be performed beforehand before scanning. The preset registration screen in this case is similar to the preset registration screen 1200 shown in FIG. 12. The document type selection area 1220 displays a list of document types that can be determined by the document type determination unit 429. As described above, the list of document types is included in a machine learning model or table for determining the document type stored in the document type determination unit 429, the data management unit 424, or the external storage 120. When displaying the preset registration screen, the display control unit 427 acquires information on the document type from the data management unit 424 or the external storage 120 and displays the list of document types. The registration information display area 1230 displays information types that can be detected by the information detection unit 428. As described above, the list of information types is included in the information detection unit 428, the data management unit 424, or the external storage 120. When displaying the preset registration screen, the display control unit 427 acquires information on the information type from the data management unit 424 or the external storage 120 and displays the list of information types. On the preset registration screen 1200, the user can set the preset name to be registered as a preset, the document type, and the information type to be associated with the document type. The information type to be registered as a preset is the information type to be subjected to masking processing in image data of the corresponding document type. When the save button is pressed on the preset registration screen 1200, the display control unit 427 requests the data management unit 424 to save the preset that has been confirmed. The data management unit 424 may save the data in the external storage 120 via the Internet access unit 426, or in the HDD 214 via the internal data storage unit 412.

[0092] By carrying out the present invention according to the above processing procedure, the document type can be automatically determined, and a mask image can be superimposed on the character string area of ​​the character string corresponding to the information type corresponding to the document type to generate a mask composite image. In addition, the character string to be masked can be edited by selecting and deselecting an item of the information type using the preview screen. By determining the item according to the information type corresponding to the document type and performing and setting masking, there is an effect of reducing the number of operations and setting load for the user when performing masking. This concludes the explanation of this embodiment.

[0093] <Second embodiment> In this embodiment, it is assumed that there are multiple presets corresponding to the same document type, and the user has not explicitly specified one preset, i.e., the masking mode is automatic. In such a case, a method for extracting all information types to be masked will be described with reference to the drawings. In the description of this embodiment, the description of the same configuration and processing procedure as the first embodiment will be omitted, and only the differences will be described.

[0094] In this embodiment, a list of presets stored in the external storage 120 is shown in the preset list. The preset list is made up of the preset name, document type, and information type to be masked. The preset name is used to identify the preset in the external storage 120. The preset name is the same as the character string specified in the preset name display area 1210. If there are duplicate preset names, they are identified by adding a sequential number or the like to the end of the preset name. In addition, any name may be used as long as it does not overlap within the preset list.

[0095] The preset list shown in Table 6 holds three presets. The first preset has a preset name of "invoice_01", a document type of "invoice", and information types to be masked are "name", "phone number", and "address". The second preset has a preset name of "invoice_02", a document type of "invoice", and information types to be masked are "name" and "email address". The third preset has a preset name of "contract_01", a document type of "contract", and information types to be masked are "document number", "name", and "address". The preset of "invoice_02" is a preset that is newly created when, for example, the preset of "invoice_01" has already been set in the preset registration screen 1200 and saved in the preset list. In the preset registration screen 1200, multiple presets can be created for one document type by changing the preset name.

[0096] [Table 6]

[0097] FIG. 16 is a preview screen in this embodiment. Only differences from the preview screen in the first embodiment (FIG. 11) will be described. The preview screen 1100 has a preset name display area 1610. The preset name display area 1610 has a preset name selection dropdown list 1611 for selecting a preset name. The preset name selection dropdown list 1611 displays all preset names that match the document type displayed in the document type display area 1120 from the preset list list, and accepts the selection of a preset name from the user. For example, when "Invoice" is selected in the information type selection dropdown list 1121, the preset names that can be selected in the preset name selection dropdown list 1611 are "Invoice_01" and "Invoice_02". The initial value of the preset name selection dropdown list 1611 is displayed as a blank space representing "not set".

[0098] When the user selects the preset name "Invoice_02", the information types to be masked can be determined as "Name" and "Email address". On the other hand, if a preset name is not set in the preset name selection dropdown list 1611, the preset cannot be determined, and therefore the information type to be masked cannot be determined either. In order to obtain all the information types to be masked in such a case, the information types to be masked that correspond to the document type "Invoice" are comprehensively obtained and presented to the user. By comprehensively presenting the information types to be masked, even if the information is not subject to masking, the masking process can be performed simply by deselecting the check box.

[0099] 15 is a flowchart for the case where multiple presets exist for a document type in this embodiment and the user has not explicitly selected one from the preset name selection dropdown list 1611. This flowchart is for explaining a method for acquiring all information types to be masked in such a case. Only the differences from the flowchart for acquiring an information type corresponding to a document type in the first embodiment (FIG. 13) will be explained. This flow is executed by the control unit 210.

[0100] In S1501, the masking information type identification processing unit 430 judges whether or not a preset corresponding to the document type determined in S704 exists among the presets acquired in S1301. If a preset corresponding to the document type exists (YES in S1501), the process proceeds to S1502, and if a preset corresponding to the document type exists (NO in S1501), the process ends this flow assuming that there is no information type to be masked. In this embodiment, the presets acquired in S1301 include two presets with the preset names "Invoice_01" and "Invoice_02".

[0101] In S1502, the masking information type identification processing unit 430 judges whether or not there is one preset corresponding to the document type judged in S704. If there is one preset (YES in S1502), the process proceeds to S1303, and if there is not one preset, i.e., if there is one or more presets (NO in S1502), the process proceeds to S1503.

[0102] In S1503, the masking information type identification processing unit 430 acquires an information type from the acquired presets so that the information types to be masked are a union, stores the information type in the RAM 213, and ends this flow.

[0103] In this embodiment, a union is used, but a preset having a large number of items of the information type to be masked may be acquired. Also, a priority may be set for the presets, and the information type may be determined based on the priority.

[0104] In this embodiment, the information types to be masked set by the preset with the preset name "Bill_01" are "Name", "Telephone number", and "Address". The information types to be masked set by the preset with the preset name "Bill_02" are "Name" and "Email address". The union of the information types to be masked set by the preset with the preset name "Bill_01" and the preset with the preset name "Bill_02" is "Name", "Telephone number", "Address", and "Email address". Therefore, the information types to be masked in this flow are "Name", "Telephone number", "Address", and "Email address". The masking information type identification processing unit 430 generates masking information by setting the masking necessity corresponding to these information types to "Needed".

[0105] An example of masking information in this embodiment is shown in the masking information list of Table 7. The masking information list is obtained by adding information of the information type "email address" to the masking information list of Table 3. Specifically, the character string corresponding to the information type is "kiyano@example.com", the masking necessity is "Yes", and the area position is "(707,1555), (877,1629)".

[0106] [Table 7]

[0107] By carrying out the present invention according to the above processing procedure, masking processing can be performed even when there are multiple presets for a document type but one preset has not been finalized. Specifically, all information types for which the masking requirement is "required" are displayed on the preview screen 1100, allowing the user to confirm. Also, by simply deselecting the checkboxes for only the character strings that do not require masking, masking omissions can be prevented. This concludes the explanation of this embodiment.

[0108] <Third embodiment> In this embodiment, the fact that the presets can be used more accurately and easily will be described with reference to the drawings. In the description of this embodiment, the description of the same configuration and processing procedure as the first and second embodiments will be omitted, and only the differences will be described.

[0109] Fig. 17 is a diagram for explaining the scan setting screen in this embodiment. In this embodiment, the scan setting screen is displayed after an authentication server (not shown) identifies the user based on the user information input by the user via the operation unit 220. Only the differences from the scan setting screen (Fig. 14) in the first embodiment will be explained. The output method setting area 1410 has an output destination designation button 1701. The output destination designation button 1701 is an item for designating the destination of the masking image that is valid when the setting of 1411 is set to "print and save" or "save only".

[0110] The output destination designation button 1701 allows the user to select one of "New input", "Address book", "Send to self", and "Designate folder". "New input" is a button for setting the destination of the masking image via the operation unit 220. The destination can be the HDD 214, the external storage 120, or an IP address via the network I / F 219. "Address book" is a button for inputting an email address and sending an email with a masking image attached. "Send to self" is a button that is valid when the user uses the button after an authentication server (not shown) identifies the user based on the user information input by the user via the operation unit 220. The authentication server sets the registered email address based on the authenticated user information. "Designate folder" is a button that can be used by setting a cloud or a file server in advance as the destination of the masking image via the operation unit 220. As with "Send to self", this button allows the user to easily set the output destination by registering destinations frequently used by the authenticated user in advance. For example, in "Designate folder", a "project folder" is preset as the destination. The project folder is a folder that can be accessed by multiple people with access restrictions, and may also include other save destination buttons.

[0111] In this embodiment, a preset list is an example of a preset list stored in the external storage 120. The preset list is composed of a preset name, a document type, an information type to be masked, and an output condition. The preset registration screen 1200 in this embodiment also has an area for accepting designation of the output condition, that is, the destination, although not shown. The designation accepted in this area is associated with the document type and is stored in the external storage 120 or the like as shown in Table 8. In this embodiment, the preset list is stored in association with the user information of the user who uses the MFP 110. When the masking information type identification processing unit 430 acquires a preset in S1301, the preset corresponding to the user information input via the operation unit 220 of the MFP 110 is acquired.

[0112] The preset list is the preset list of Table 6 in the second embodiment with the addition of presets with the preset names "Invoice_03" and "Invoice_04". The preset with the preset name "Invoice_03" is a preset with a document type of "Invoice", an information type to be masked of "Address", and an output destination condition of "Send to self". The preset with the preset name "Invoice_04" is a preset with a document type of "Invoice", an information type to be masked of "Phone number" and "Address", and an output destination condition of "Project folder in specified folder".

[0113] [Table 8]

[0114] When the document type is "invoice", when the information type to be masked is obtained in S706, the corresponding preset names will be four: "invoice_01", "invoice_02", "invoice_03", and "invoice_04". In the second embodiment, the information types to be masked at this time are "name", "telephone number", "address", and "email address". The masking information display area 1130 is displayed with the information types preselected for the check boxes for "name", "telephone number", "address", and "email address".

[0115] On the other hand, in this embodiment, when acquiring the information type to be masked in S706, a preset is specified based on the document type and the output destination condition, and the information type associated with the specified preset is acquired. For example, consider a case where the document type determined in S704 is "invoice" and "send to me" is selected in the output destination designation button 1701 in S501. In this case, a preset with the document type being "invoice" and the output destination condition being "send to me" is identified from the preset list shown in Table 8. Since the only corresponding preset is the preset with the preset name "invoice_03", "address" is acquired as the information type to be masked. In the masking information display area 1130, the check box for the information type "address" is displayed in a preselected state, and the check boxes for the information types "name", "telephone number", and "email address" are displayed in a deselected state.

[0116] Compared to a case where the information type to be masked is determined using only the information type, the number of operations performed by the user regarding the check boxes in the masking information display area 1130 can be reduced.

[0117] Similarly, when a project folder is specified in "Specify Folder", the only preset that is acquired is the preset name "Invoice_04". The information types to be masked are "phone number" and "address". The masking information display area 1130 is displayed with the check boxes for the information types "phone number" and "address" preselected, and the check boxes for the information types "name" and "email address" deselected.

[0118] If you specify a project folder in "Specify Folder", multiple people will be able to access it, so you can set items to be masked that correspond to people with access permissions.

[0119] Furthermore, if you specify "Send to me," you can perform even more minimal masking.

[0120] By carrying out the present invention with the above processing procedure, the number of operations performed by the user on the preview screen 1100 can be reduced by determining the information type to be preset and masked using the output destination designation in addition to the document type. Furthermore, by determining the information type to be masked by designating the output destination, masking can be performed without excess or deficiency.

[0121] This concludes the description of this embodiment.

[0122] <Fourth embodiment> In this embodiment, the processing load of the document type determination process can be reduced and the processing speed can be easily increased, as will be described with reference to the drawings. Note that in describing this embodiment, descriptions of the same configurations and processing procedures as those of the first, second, and third embodiments will be omitted, and only the differences will be described.

[0123] 18 is a flowchart for explaining a method for reducing the processing load of the document type determination process (S704) by the document type determination processing unit 429 in the masking area specification process (S503). This flow is executed by the control unit 210.

[0124] In S1801, the image processing unit 423 acquires a character string area from the RAM 213.

[0125] In S1802, the image processing unit 423 judges whether or not data similar to the character string region acquired in S702 exists in the character string region database. If it exists (YES in S1802), the process proceeds to S1803, and if it does not exist (NO in S1802), the process proceeds to S704.

[0126] The character string area database stores one or more character string areas registered in advance by the user as one document layout in the HDD 214. Furthermore, the document layout and the document type are associated and stored in the HDD 214. Also, a similar form registration button (not shown) may be provided so that the character string area and the document type acquired in S702 at the time of preset registration are associated and saved. The similar form registration button allows the user to select a character string area that the user wishes to register as a document layout from among the character string areas acquired in S702, and registers the selected character string area in the character string area database.

[0127] The number of times the character string region and the information type are used may be stored in the RAM 213, and if the number of times the character string region and the information type are used is equal to or greater than a threshold value, the character string region and the information type may be registered in the character string region database.

[0128] When making a similarity determination, the area of ​​the common area is used when the character string area in the character string area database and the character string area acquired in S702 are overlapped with the same top left coordinate. Similarity is determined when the ratio of the area of ​​the common area to the area of ​​the character string area in the character string area database or the area of ​​the character string area acquired in S702 is equal to or greater than a threshold. The area of ​​the character string area refers to the area of ​​a rectangle formed by the coordinates representing the character string area.

[0129] In S1803 , the image processing unit 423 acquires the document type corresponding to the similar character string region from the character string region database, and stores it in the RAM 213 .

[0130] By carrying out the present invention according to the above processing procedure, it is possible to determine the document type by performing similarity determination based on character string regions, and the processing load of the document type determination processing unit (429) can be reduced.

[0131] This concludes the description of this embodiment.

[0132] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of each of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC or FPGA) that realizes one or more functions. [Explanation of symbols]

[0133] 110 MFP 423 Image Processing Unit 427 Display control section 429 Document Type Determination Unit

Claims

1. A specifying means for specifying a type of image data to be edited from among a plurality of types of image data based on a recognition process for specifying the type of image data to be edited; a search means for searching a character string corresponding to the category associated with the specified type from among a plurality of character strings included in the image data to be edited; a masking unit that performs a masking process on the searched character string in the image data to be edited; having A different category is associated with each of the plurality of types of image data.

23. An information processing apparatus comprising:

2. a first reception means for receiving from a user a designation of a category to be subjected to a masking process for each of the plurality of types of image data; a setting means for setting a category, the designation of which has been accepted, as a category to be masked in association with each of the plurality of types of image data; and The search means searches for a character string corresponding to the specified type and corresponding to the category set by the setting means.

2. The information processing apparatus according to claim 1,

3. a providing means for providing a screen including the image data to be edited, in which the searched character string is masked and displayed, and a plurality of categories including the category associated with the specified type; a second receiving means for receiving, via the screen, an instruction from a user to select or deselect each of the plurality of categories as a target for a masking process in the image data to be edited; and When the instruction is an instruction to select one unselected category among the plurality of categories as a target for masking processing, a character string corresponding to the one unselected category is masked and displayed in the image data displayed on the screen; When the instruction is an instruction to cancel the selection of one of the multiple categories as a target for masking processing, a character string corresponding to the selected category is displayed without being masked in the image data displayed on the screen.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the first receiving means receives a new designation of a second category to be subjected to the masking process for one type of the plurality of image data types for which a first category to be subjected to the masking process has already been set; The setting means further sets the newly designated second category in association with the one type in a state in which the already set first category is set in association with the one type.

3. The information processing apparatus according to claim 2.

5. The search means searches for a character string corresponding to a first category and a character string corresponding to a second category, which are associated with the specified type.

5. The information processing apparatus according to claim 4.

6. the first reception means further receives, from a user, a designation of a destination in addition to a designation of a category to be subjected to a masking process for each of the plurality of types of image data; the setting means sets a transmission destination in association with each of the plurality of types of image data; When a destination of the image data to be edited is set by the setting means in association with the specified type, the search means searches for a character string corresponding to a category set in association with the specified type.

3. The information processing apparatus according to claim 2.

7. Further comprising a scanner; The image data to be edited is obtained by scanning a document with the scanner.

2. The information processing apparatus according to claim 1,

8. Further comprising a printer; a printing means for printing, by the printer, the image data to be edited that has been subjected to the masking process by the masking means; 2. The information processing apparatus according to claim 1, further comprising:

9. The image data to be edited, which has been subjected to the masking process by the masking means, is stored in a storage of the information processing device or a storage of an external device.

2. The information processing apparatus according to claim 1,

10. The search means acquires a character string corresponding to a category associated with the specified type from an external device, and searches for the character string among a plurality of character strings included in the image data to be edited.

2. The information processing apparatus according to claim 1,

11. A method for controlling an information processing device, comprising: a step of identifying a type of image data to be edited from among a plurality of types of image data based on a recognition process for identifying the type of image data to be edited; a search step of searching for a character string corresponding to the category associated with the specified type from among a plurality of character strings included in the image data to be edited; a masking step of performing a masking process on the searched character string in the image data to be edited; having A different category is associated with each of the plurality of types of image data. A control method comprising:

12. On the computer, a step of identifying a type of image data to be edited from among a plurality of types of image data based on a recognition process for identifying the type of image data to be edited; a search step of searching for a character string corresponding to a category associated with the specified type from among a plurality of character strings included in the image data to be edited; a masking step of performing a masking process on the searched character string in the image data to be edited; A program for executing A different category is associated with each of the plurality of types of image data. A program characterized by:

Images