Image processing device, image processing method, and program

Abstract

To provide an image processing device, method, and program that conceal character strings that may include confidential information while retaining information necessary for system improvement in a form image.SOLUTION: An image processing device is configured to perform a step S1201 for acquiring information on a first character string, which is a character string contained in a document image, a step S1202 for detecting a second character string corresponding to a non-confidential string from among the information in the first character string by using information on multiple non-confidential character strings that correspond to non-confidential information, detecting a third character string corresponding to confidential information from among the information in the first character string, performing a first concealing process on the information in the third character string, and performing a second concealing process that is different from a first concealing process for information in a fourth character string, which is the first character string minus the second and third character strings, and a step S1203 for generating a concealing image corresponding to the document image on the basis of the information of the second character string, which is the detection result of first detection means, and a processing result of the first concealing process and the processing result of the second concealing process.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating an image in which confidential information included in a form is concealed. [Background technology]

[0002] There is a system that manages forms by generating form images by scanning and digitizing paper forms and automatically setting file names based on the contents of the forms to files containing the data of the generated form images. In order for developers to improve such systems to better suit the actual usage of users, it is important to collect and utilize form image data corresponding to forms actually used in the user's business (hereinafter referred to as "real forms"). In this case, in order to collect form image data corresponding to real forms, it is necessary to obtain consent from users to collect form image data, and to ensure that confidential information is not leaked by performing masking processing on confidential information contained in the form images of the real forms. On the other hand, in order for developers to use form image data to improve the system, it is necessary to collect form image data in a state where information contained in the real forms is left as much as possible.

[0003] In order to achieve both concealment of confidential information and retention of information necessary for system improvement, there is a method of detecting character strings corresponding to confidential information from a form image and replacing the detected character strings with other character strings that do not contain confidential information. Patent Document 1 discloses a technology for detecting character strings corresponding to confidential information from character strings included in a form image and replacing the detected character strings with other character strings. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-104272 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 has a problem in that a character string that is not detected as being classified as classified information may contain classified information, and in such a case, it is not possible to guarantee that the user's classified information will not be leaked. The present disclosure is intended to solve this problem, and aims to provide a technology that can conceal character strings that may contain classified information in a form image while leaving information necessary for improving the system. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes an acquisition means for acquiring information of a first string which is a string included in a document image, a first detection means for detecting a second string which corresponds to the non-confidential string from the information of the first string using information of a plurality of non-confidential strings which correspond to non-confidential information, a second detection means for detecting a third string which corresponds to confidential information from the first string, a concealment means for performing a first anonymization process on the information of the third string, and performing a second anonymization process different from the first anonymization process on information of a fourth string which is obtained by excluding the second string and the third string from the first string, and a generation means for generating an anonymized image corresponding to the document image based on the information of the second string which is the detection result of the first detection means, a processing result of the first anonymization process, and a processing result of the second anonymization process. Effect of the Invention

[0007] According to the present disclosure, in a form image, character strings that may include confidential information can be concealed while leaving information necessary for system improvement. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of an image processing system. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of an MFP. [Diagram 3] 2 is a block diagram showing an example of the hardware configuration of a client PC and an image processing server that provides an MFP cooperation service. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing system. [Diagram 5] FIG. 2 is a sequence diagram showing an example of a processing flow of the image processing system. [Figure 6] 10 is a flowchart showing an example of the flow of a file name generation process in an image processing unit. [Figure 7] FIG. 11 is a diagram illustrating an example of a data structure of a file name setting rule. [Figure 8] 11A and 11B are diagrams illustrating an example of a result of extraction of a foreground region in an image processing unit. [Figure 9] FIG. 13 illustrates an example of a scanned form list screen. [Figure 10] FIG. 13 illustrates an example of a file name setting screen. [Figure 11] FIG. 13 is a diagram showing an example of a confirmation screen for an improvement image. [Figure 12] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit. [Figure 13] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit. [Figure 14] 11A and 11B are diagrams illustrating an example of a process for detecting character string information corresponding to non-confidential information. [Figure 15] 11A to 11C are diagrams illustrating an example of a detection process of character string information corresponding to confidential information, and a first concealment process. [Figure 16] FIG. 11 is a diagram illustrating an example of a second concealment process. [Figure 17] FIG. 13 is a diagram showing an example of a result of a concealment process performed by an image processing unit. [Figure 18] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit according to the second embodiment. [Figure 19] FIG. 11 is a diagram showing an example of a concealed image generated by an image processing unit according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the disclosed solution.

[0010] First Embodiment [Overall configuration] 1 is a block diagram showing an example of the configuration of an image processing system 100 according to the first embodiment. The image processing system 100 includes an MFP (Multifunction Peripheral) 110, a client PC 111, an MFP cooperation service 120, and cloud storages 130 and 140. The MFP 110 and the client PC 111 are communicably connected to a server that provides various services on the Internet via a LAN (Local Area Network).

[0011] The MFP 110 is a multifunction machine having multiple functions such as a scan function and a print function, and is an example of an image reading device. The client PC 111 is configured with a personal computer or the like, and receives a service requested from the MFP cooperation service 120. The MFP cooperation service 120 is a service provided by one device configured with an image processing server or the like, or by the cooperation of multiple devices. Specifically, the MFP cooperation service 120 has a function of saving an image file obtained by scanning a paper document or the like in the MFP 110 in a storage device within the device, or transferring the image file to a device that provides a service capable of saving files, such as a storage service. The cloud storages 130 and 140 are services provided by one device configured with a storage server or the like, or by the cooperation of multiple devices. Specifically, the cloud storages 130 and 140 are services that save a file received via the Internet in a storage device within the device, or read a file specified using a web browser or the like from the storage device and transmit it to an external device.

[0012] In this embodiment, cloud storage 130 is used to store image files obtained by scanning paper forms and the like by users (hereinafter simply referred to as "users") who use MFP linkage service 120, and to reuse the image files. Cloud storage 140 is also used to accumulate data used by providers or developers (hereinafter simply referred to as "developers") of MFP linkage service 120 when improving the service.

[0013] Hereinafter, image processing system 100 will be described as including MFP 110, client PC 111, MFP collaboration service 120, and cloud storages 130 and 140, but the configuration of image processing system 100 is not limited to this. For example, MFP 110 may serve as at least one of client PC 111 and MFP collaboration service 120. Furthermore, MFP collaboration service 120 may be provided by a device located on a LAN rather than on the Internet. Furthermore, cloud storage 130 may be replaced with a service provided by a mail server or the like, and an image file obtained by scanning may be attached to an email and transmitted to an external device.

[0014] [MFP hardware configuration] 2 is a block diagram showing an example of the hardware configuration of the MFP 110 according to the first embodiment. The MFP 110 has, as its hardware configuration, a control unit 210, an operation unit 220, a printer 221, a scanner 222, and a modem 223. The control unit 210 controls the overall operation of the MFP 110. The control unit 210 has, as its hardware configuration, a CPU 211, a ROM 212, a RAM 213, and a HDD 214. In addition to the above-mentioned hardware configuration, the control unit 210 also has an operation unit I / F 215, a printer I / F 216, a scanner I / F 217, a modem I / F 218, and a network I / F 219.

[0015] The CPU 211 reads and executes a control program stored in the ROM 212, thereby executing and controlling various functions of the MFP 110, such as reading and printing images, and transmitting and receiving data. The RAM 213 is used as a temporary storage area such as the main memory and work area of ​​the CPU 211. Note that in this embodiment, one CPU 211 executes each process shown in the flowcharts described later using one memory (RAM 213 or HDD 214), but is not limited to this. For example, a configuration may be adopted in which multiple CPUs, or multiple RAMs or HDDs work together to execute each process shown in the flowcharts described later. The HDD 214 is a large-capacity storage device that stores image data, various programs, and the like.

[0016] The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The operation unit 220 is provided with a touch panel, a keyboard, or the like, and accepts operations, inputs, or instructions from a user. In the following, the operation unit 220 will be described as being configured with a liquid crystal display device having a touch panel function. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data for printing is transferred from the control unit 210 to the printer 221 via the printer I / F 216, and an image corresponding to the image data is printed on a recording medium by the printer 221.

[0017] The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads an original placed on a platen or an ADF (Auto Document Feeder) not shown in FIG. 2 to generate image data, and outputs the generated image data to the control unit 210 via the scanner I / F 217. The MFP 110 has a copy function of printing an image corresponding to the image data generated by the scanner 222 with the printer 221, as well as a function of transmitting the image data to the outside or attaching it to an e-mail and transmitting it to the outside. The modem I / F 218 is an interface that connects the modem 223 and the control unit 210. The modem 223 communicates image data by facsimile with a facsimile device connected via a PSTN (Public Switched Telephone Network). The network I / F 219 is an interface that connects the control unit 210 (MFP 110) to a computer network such as a LAN. The MFP 110 transmits image data and various information to various services on the Internet via a network I / F 219, and receives various information from various services.

[0018] [Hardware configuration of client PC and device providing MFP linkage service 120] Fig. 3 is a block diagram showing an example of the hardware configuration of a client PC 111 according to the first embodiment, and an image processing server 300 that provides an MFP linkage service 120. Specifically, Fig. 3(a) shows an example of the hardware configuration of the image processing server 300, and Fig. 3(b) shows an example of the hardware configuration of the client PC 111.

[0019] The image processing server 300 includes a control unit 310. The control unit 310 controls the overall operation of the image processing server 300 to provide the MFP cooperation service 120. The control unit 310 includes a CPU 311, a ROM 312, a RAM 313, a HDD 314, and a network I / F 315. The CPU 311 controls the overall operation of the image processing server 300 by reading and executing a control program stored in the ROM 312. The RAM 313 is used as a temporary storage area such as a main memory and a work area of ​​the CPU 311. The HDD 314 is a large-capacity storage device that stores image data and various programs. The network I / F 315 is an interface that connects the control unit 310 (image processing server 300) to a computer network such as the Internet. The image processing server 300 receives a processing request from an external device such as the MFP 110 via the network I / F 315, and transmits and receives various information to and from the external device.

[0020] The client PC 111 includes a control unit 320. The control unit 320 controls the overall operation of the client PC 111. The control unit 320 includes a CPU 321, a ROM 322, a RAM 323, a HDD 324, and a network I / F 325. The CPU 321 controls the overall operation of the client PC 111 by reading and executing a control program stored in the ROM 322. The RAM 323 is used as a temporary storage area such as a main memory and a work area of ​​the CPU 321. The HDD 324 is a large-capacity storage device that stores image data and various programs. The network I / F 325 is an interface that connects the control unit 320 (client PC 111) to a computer network such as a LAN. The client PC 111 transmits a processing request to the MFP cooperation service 120, etc., and transmits and receives various information to and from an external device via the network I / F 325.

[0021] [Image processing system functional configuration] 4 is a block diagram showing an example of the functional configuration of the image processing system 100 according to the first embodiment. The MFP 110 has, as its functional configuration, a native function unit 410 and an additional function unit 420. The native function unit 410 has a scan execution unit 411 and an image data storage unit 412, and the additional function unit 420 has a display control unit 421, a scan instruction unit 422, and a cooperation service request unit 423. Each unit that the MFP 110 has as its functional configuration is realized by execution of a program by the CPU 211.

[0022] Each unit included in the native function unit 410 is a functional configuration that is provided as standard in the MFP 110. In contrast, each unit included in the additional function unit 420 is a functional configuration that functions by executing an application program that is additionally installed in the MFP 110. The function of the additional function unit 420 is realized by installing an application program based on Java (registered trademark), for example. Therefore, the function can be easily added to the MFP 110 by this installation. Note that other additional application programs for realizing functional configurations not shown in FIG. 4 may be installed in the MFP 110 additional function unit 420.

[0023] The display control unit 421 displays a UI (User Interface) screen for receiving operations by the user on the operation unit 220. Specifically, the display control unit 421 displays UI screens such as an input screen for authentication information for accessing the MFP linkage service 120, an operation screen for setting scans and starting scans, and a preview screen for viewing images obtained by scanning. The scan instruction unit 422 instructs the scan execution unit 411 to execute scan processing based on scan settings according to operations or instructions from the user input via the UI screen.

[0024] The scan execution unit 411 receives an instruction to execute a scan process from the scan instruction unit 422, and controls the operation of the scanner 222 via the scanner I / F 217 according to the scan setting included in the instruction. The scanner 222 generates image data corresponding to the document (hereinafter referred to as "scanned image") by reading a document placed on a document table or the like according to the control of the scan execution unit 411. The generated scanned image data is sent to the image data storage unit 412 via the scan execution unit 411. The scan execution unit 411 also generates a scanned image identifier expressed by a number or one or more symbols or alphabets, etc., for uniquely identifying the generated scanned image data. The generated scanned image identifier is sent to the image data storage unit 412 and the scan instruction unit 422. The image data storage unit 412 associates the scanned image data received from the scan execution unit 411 with the scanned image identifier and stores them in the HDD 214.

[0025] The scan instruction unit 422 acquires scan image data corresponding to the scan image identifier received from the scan execution unit 411 from the image data storage unit 412. The scan instruction unit 422 instructs the cooperation service request unit 423 to make a request to the MFP cooperation service 120 to perform processing on the acquired scan image data.

[0026] Upon receiving an instruction from the scan instruction unit 422, the collaboration service request unit 423 makes a request to the MFP collaboration service 120 to perform various processes on the scanned image data. For example, the collaboration service request unit 423 makes a login request to the MFP collaboration service 120, a request for image analysis of the scanned image data, and a request for transmission of the scanned image data after various processes. The communication between the collaboration service request unit 423 and the MFP collaboration service 120 is performed using a protocol such as REST (Representational state transfer) or SOAP (Simple object access protocol). Note that the communication is not limited to REST or SOAP and may be performed using other communication methods.

[0027] The MFP interaction service 120 has, as its functional components, a request control unit 431, an image processing unit 432, a cloud storage access unit 433, a data management unit 434, and a display control unit 435. Each unit that the MFP interaction service 120 has as its functional components is realized by the execution of a program by the CPU 311. The request control unit 431 waits in a state in which it can receive a processing request from an external device. When the request control unit 431 receives a processing request, it instructs the image processing unit 432, the cloud storage access unit 433, or the data management unit 434 to execute processing corresponding to the request as appropriate in response to the received request. The image processing unit 432 performs recognition and processing on an image, such as character area analysis processing, character recognition processing, concealment processing, rotation processing, and tilt correction processing.

[0028] The cloud storage access unit 433 issues processing requests to the cloud storages 130 and 140. Cloud storage services generally use protocols such as REST or SOAP to publish various interfaces for saving files in cloud storage and retrieving files stored in cloud storage. The cloud storage access unit 433 operates cloud storage using the published interface of the cloud storage service. The data management unit 434 holds user information and various setting data managed by the MFP cooperation service 120.

[0029] The display control unit 435 receives a processing request from a web browser operating on a terminal such as a personal computer or a mobile terminal (not shown in FIG. 1) connected via the Internet, and returns screen configuration information required for screen display. Here, the screen configuration information is composed of, for example, a markup language such as HTML (HyperText Markup Language) or a style sheet such as CSS (Cascading Style Sheets). The web browser generates a screen corresponding to the screen configuration information by rendering the screen configuration information received from the display control unit 435, and displays the screen on a display device. A user can check or change user information and scan settings registered in the MFP linkage service 120 by viewing and operating the screen displayed by the web browser.

[0030] In the above description, an additional program is installed in MFP 110 to add a functional configuration to additional function unit 420, but the present invention is not limited to this configuration. For example, client PC 111 may have a functional configuration equivalent to additional function unit 420, and an additional program may be installed in client PC 111 to add the functional configuration to client PC 111.

[0031] [Image processing system processing flow] Fig. 5 is a sequence diagram showing an example of the flow of processing in the image processing system 100 according to the first embodiment. Specifically, Fig. 5 is a sequence diagram showing an example of the flow of processing in the image processing system 100 when image data obtained by scanning in the MFP 110 is converted into a file and transmitted to the cloud storages 130 and 140. Here, the description will focus on the exchange between each device. Note that Fig. 5 shows a form in which the MFP 110 exchanges with the MFP cooperation service 120, but the process of obtaining the analysis result or displaying the screen, which will be described later, may be executed by the client PC 111 instead of the MFP 110.

[0032] In a normal state, the MFP 110 displays on the operation unit 220 a main screen on which buttons for implementing each function provided by the MFP 110 are arranged. An additional application (hereinafter referred to as a "scan application") for transmitting scan image data to cloud storages 130 and 140 is installed in the MFP 110. As a result, buttons for implementing functions realized by executing the scan application are displayed on the main screen. When this button is pressed by the user, a screen required for operations for transmitting scan image data to cloud storages 130 and 140 is displayed on the operation unit 220, and the process shown in the sequence diagram of FIG. 5 is executed. In the following description, the symbol "S" means a step.

[0033] First, in S501, display control unit 421 of MFP 110 displays on operation unit 220 a login screen that accepts input of authentication information for accessing MFP linkage service 120. A user inputs authentication information into the login screen via operation unit 220, and then performs an operation for a login request to MFP linkage service 120. Next, in S502, MFP 110 makes a login request to MFP linkage service 120 using authentication information such as a user name and password accepted via the login screen.

[0034] Next, in S503, the request control unit 431 of the MFP collaboration service 120 executes login processing. Specifically, as the login processing, the request control unit 431 verifies whether authentication information such as a user name and password included in the login request is correct (also called "user authentication"), and if correct, returns an access token to the MFP 110. Subsequent various processing requests made from the MFP 110 (scan application) to the MFP collaboration service 120 are made using this access token, and the user to be processed can be identified by this access token. The user authentication is performed using a known method such as authentication by Basic authentication, Digest authentication, or the like, or authorization using OAuth.

[0035] When receiving information indicating completion of the login process from the MFP cooperation service 120, in S504, the display control unit 421 of the MFP 110 displays a scan setting screen on the operation unit 220. The scan setting screen is a UI screen for receiving input of settings related to the scan mode of the paper form, the file format when the scanned image data obtained by scanning is saved as an image file, and the destination of the image file. Next, when the "Start Scan" button is pressed by the user's operation, in S505, the MFP 110 executes a scan process on the paper form to be scanned placed on the platen or ADF, etc., according to various scan settings set on the scan setting screen. This scan process digitizes the paper form, and scan image data is generated. Next, in S506, the MFP 110 transmits an image analysis request of the image indicated by the scan image data (scan image) to the MFP cooperation service 120 together with the information on the scan setting set in S504 and the scan image data generated in S505. The scan setting information, the scanned image data, and the image analysis request for the scanned image sent from the MFP 110 are received by a request control unit 431 of the MFP collaboration service 120.

[0036] When the request control unit 431 of the MFP collaboration service 120 receives the image analysis request for the scanned image, in S507 it instructs the image processing unit 432 of the MFP collaboration service 120 to execute image analysis processing on the received scanned image. In response to the instruction, the image processing unit 432 starts image analysis processing on the scanned image. Note that the request control unit 431, which has received the scan setting information, holds the received scan setting information so that it can be used in subsequent processing. When the request control unit 431 receives the image analysis request for the scanned image, it returns a processId, which is an identifier that uniquely indicates the requested image analysis processing, to the MFP 110 as a response to the request, without waiting for the image analysis processing of the scanned image by the image processing unit 432 to end. The processId sent from the MFP collaboration service 120 is received by the scan application of the MFP 110.

[0037] In S508, the image processing unit 432 performs image analysis processing such as character recognition on the scanned image, and generates a file name to be used when saving the image file based on the result of the image analysis processing. Details of the file name generation processing will be described later. Next, in S509, the image processing unit 432 performs an anonymization processing on the scanned image, and generates an image in which confidential information has been anonymized (hereinafter referred to as an "anonymized image"). Here, confidential information to be anonymized by the anonymization processing is not limited to confidential information such as transaction contents or contract contents included in the form. Confidential information includes all information that the user does not want to disclose to the developer, including, for example, personal information included in the form. Details of the anonymization processing will be described later.

[0038] Next, in S510, the image processing unit 432 generates screen configuration information that is the basis of a file name setting screen displayed on the MFP 110, using result information indicating the processing results of S508 and S509. Here, the result information is the scanned image after the image analysis processing (called the "analyzed scanned image"), information indicating the recognition result of the character recognition, information indicating the generation result of the file name, and data of the concealed image. The analyzed scanned image is an image after image processing such as tilt correction is performed on the scanned image transmitted in S506 in S508. After completing the processing of S510, the image processing unit 432 transmits a completion notification indicating the completion of the image analysis processing corresponding to processId to the request control unit 431.

[0039] In S511, the MFP 110 uses the processId received as a response to the image analysis request in S506 to periodically check the processing status of the image analysis process corresponding to the processId with the MFP cooperation service 120, such as every several hundred milliseconds to every few milliseconds. Note that "periodic" here is not necessarily limited to a timing strictly at a certain interval, but includes a timing at an approximately certain interval. The MFP 110 may also irregularly check the processing status of the image analysis process corresponding to the processId. The processing of S511 is continued until a response indicating the completion of the image analysis process of the MFP cooperation service 120 corresponding to the processId is obtained.

[0040] When the MFP cooperation service 120 receives the request to confirm the processing status of S511, the MFP cooperation service 120 confirms the processing status of the image analysis processing corresponding to the processId, and returns information on the processing status as a response. A character string indicating the current processing status is stored in "status" of the processing status information returned as a response. For example, when the character string "processing" is stored in "status", the response indicates that the image analysis processing is being performed by the MFP cooperation service 120. Also, for example, when the character string "completed" is stored in "status", the response indicates that the image analysis processing is completed. Note that after the image analysis processing is completed, that is, when "completed" is stored in "status", the response stores location information for acquiring screen configuration information corresponding to the processId together with the processing status information. Also, in addition to the processing status information and location information, the response may store information such as the analysis result of the scanned image and the scan settings. Also, when the image analysis processing fails, a character string other than "processing" or "completed", such as "failed", may be stored in "status".

[0041] When the MFP 110 detects the completion of the image analysis process from the response to the confirmation of the processing status of the image analysis process in S512, in S513, the MFP 110 acquires screen configuration information corresponding to the processId using the location information included in the response. For example, if the location information includes URL (Uniform Resource Locator) information indicating the location of the screen configuration information, the MFP 110 acquires the screen configuration information by accessing the URL. The screen configuration information acquired in S513 is information that is the basis of a file name setting screen that displays result information indicating the processing results of S508 and S509.

[0042] Next, in S514, the MFP 110 generates and displays a file name setting screen based on the screen configuration information acquired in S513, for example, by rendering the screen configuration information. Here, the file name setting screen is a UI screen for setting a file name. Details of the file name setting screen will be described later with reference to FIG. 10. Next, when a file name is set by a user operation on the file name setting screen and then an operation to confirm the file name is performed, the MFP 110 transmits information on the set file name, etc. to the MFP cooperation service 120 in S515. The file name information, etc. transmitted from the MFP 110 is received by the request control unit 431 of the MFP cooperation service 120. Upon receiving the file name information, the request control unit 431 starts a file generation process and returns a response indicating that the information has been normally received to the MFP 110. Upon receiving the response, the MFP 110 ends the series of processes from S504 onwards, returns to the process of S504, and displays the scan setting screen on the operation unit 220.

[0043] In S516, request control unit 431 generates a file corresponding to the scanned image data (hereinafter referred to as a "scanned image file"). Specifically, first, request control unit 431 acquires information on the file format of the file to be transmitted to cloud storage 130 from the information on the scan settings held in S507. Next, request control unit 431 generates a scanned image file corresponding to the scanned image data based on the information. Next, in S517, request control unit 431 assigns the file name received in S515 to the file name of the scanned image file generated in S516, and transmits the scanned image file after the assignment of the file name to cloud storage 130. When cloud storage 130 receives the scanned image file, it transmits a completion notification to MFP cooperation service 120 as a response indicating completion of reception.

[0044] Next, in S518, the request control unit 431 generates a file corresponding to the data of the anonymized image (hereinafter referred to as a "anonymized image file"). Specifically, first, the request control unit 431 acquires information on the file format of the file to be transmitted to the cloud storage 140 from information such as the storage format of the image data and the method of assigning a file name, which are registered in advance in the MFP cooperation service 120 by a developer or the like. Next, the request control unit 431 generates a anonymized image file corresponding to the anonymized image data based on the information. Note that the anonymized image file is assigned a file name determined based on the information on the method of assigning a file name. Note that the request control unit 431 may use the file name received in S515 as the file name to be assigned to the anonymized image file. Also, for example, the request control unit 431 may use a file name determined based on information indicating the date and time when the scan process was performed, information indicating the logged-in user, or the like.

[0045] Next, in S519, request control unit 431 transmits the anonymized image file generated in S518 after the file name is assigned to cloud storage 140. When cloud storage 140 receives the anonymized image file, it transmits a completion notification to MFP collaboration service 120 as a response indicating completion of reception. When MFP collaboration service 120 receives a response corresponding to the processing of S519, it ends the series of processing from S507 onwards, and waits in a state in which it is able to accept the next login request or image analysis request.

[0046] In addition, in the process of S519, the request control unit 431 may transmit information that can be obtained through a series of processes to the cloud storage 140 together with the concealed image file. The information that can be obtained through a series of processes may be, for example, a processing log for a character string area of ​​a character string including a character string used as a file name that can be obtained in the file name generation process in S508. In addition, for example, the information that can be obtained through a series of processes may be information on a file name set by a user in S514 and an operation log on a file name setting screen. In addition, for example, the information that can be obtained through a series of processes may be information indicating the presence or absence of confidential information, or a processing log of the position or attributes of confidential information in a scanned image that can be obtained in the concealment process in S509.

[0047] [File name generation process] FIG. 6 is a flowchart showing an example of the flow of file name generation processing in the image processing unit 432 according to the first embodiment, and is a flowchart showing an example of a detailed flow of processing in S508. First, in S601, the image processing unit 432 acquires information on a file name setting rule from the data management unit 434. The file name setting rule defines in advance the item attributes of a character string that can be used as a file name, and the order of arrangement of a plurality of character strings corresponding to the plurality of item attributes. The image processing unit 432 generates a file name according to the file name setting rule. The file name setting rule may be input via a dedicated UI for rule creation, or may be specified by importing a file indicating data of the file name setting rule in advance.

[0048] FIG. 7 is a diagram showing an example of a data structure of a file name setting rule according to the first embodiment. In the "key", "filename" indicating that it is used as a file name is set. In the "value", information on a plurality of item attributes used as a file name and information indicating a delimiter to be inserted between a plurality of character strings corresponding to the plurality of item attributes are set. In the example of the data structure shown in FIG. 7, "title", "sep", "date", "sep", and "id" are set in the "value". For example, in this case, a character string is generated in which character strings corresponding to the item attributes "title", "date", and "id" are arranged in order as a file name, and a character or character string corresponding to "sep" indicating a delimiter is inserted between the character strings.

[0049] After S601, in S602, the image processing unit 432 performs character region analysis processing on the scanned image. The character region analysis processing is processing for separating an image into a foreground region and a background region, classifying the foreground region into a character string region and other regions, and acquiring the character string region in the image. Specifically, for example, in S602, the image processing unit 432 first performs contour tracing on the scanned image binarized into black and white, and extracts a cluster of pixels surrounded by a contour composed of black pixels, that is, a cluster of black pixels. Subsequently in S602, for a cluster of black pixels having an area larger than a predetermined size, the image processing unit 432 performs contour tracing on the white pixels inside the cluster in the same manner as in the case of black pixels, thereby extracting a cluster of white pixels. At this time, for a cluster of white pixels having an area equal to or larger than a predetermined size, a cluster of black pixels is recursively extracted from the inside of the cluster. The image processing unit 432 separates the cluster of black pixels thus obtained from the background region as a foreground region. Next, in S602, the image processing unit 432 classifies each of the separated foreground regions into one of the following attributes based on the size and shape of the region:

[0050] For example, the image processing unit 432 determines that a foreground region whose aspect ratio is close to 1 and whose size is within a certain range corresponds to a character. Furthermore, the image processing unit 432 determines that a plurality of regions determined to correspond to characters are close to each other and can be grouped in an aligned state as a region corresponding to a character string (hereinafter referred to as a "TEXT attribute"). Furthermore, the image processing unit 432 determines that a foreground region whose shape is flat corresponds to a line (hereinafter referred to as a "LINE attribute"). Furthermore, the image processing unit 432 determines that a foreground region whose size is equal to or larger than a predetermined size and which contains a plurality of aligned rectangular clusters of white pixels corresponds to a table (hereinafter referred to as a "TABLE attribute"). Furthermore, the image processing unit 432 determines that a region in which irregular foreground regions are scattered corresponds to a photograph (hereinafter referred to as a "PHOTO attribute"). Furthermore, the image processing unit 432 determines that a foreground region that does not belong to any of the TEXT attribute, LINE attribute, TABLE attribute, and PHOTO attribute is a region corresponding to a picture (hereinafter referred to as a "PICTURE attribute").

[0051] Subsequently in S602, the image processing unit 432 acquires, as a character string region, the foreground region determined to have the TEXT attribute from among the foreground regions classified by attribute in this manner. Note that, in order to increase the accuracy of acquiring the character string region, the image processing unit 432 may perform correction processing such as tilt correction processing or rotation correction processing on the scanned image in advance, and perform the above-mentioned separation and classification processing on the scanned image after the processing.

[0052] In the skew correction process, first, the angle of the skew of the scanned image is detected, and the skew of the scanned image is corrected by rotating the scanned image in the opposite direction by an angle corresponding to the detected skew angle. The skew that is the subject of the skew correction occurs, for example, when a paper document is scanned in a skewed state due to wear of the rollers in the document feeder of the scanner 222 when the paper document is scanned. Or, when the document is printed, the paper is printed in a skewed state, and the character string that is the subject of the skew correction is also printed in a skewed state with respect to the paper. As a method for detecting the skew angle, first, a plurality of character objects included in the scanned image are detected, and adjacent character objects within a predetermined distance in the horizontal or vertical direction are connected. That is, when the document is written horizontally, adjacent character objects in the horizontal direction are connected, and when the document is written vertically, adjacent character objects in the vertical direction are connected, and the character objects that constitute one character string are identified. Next, the angle of a straight line (e.g., a regression line) calculated based on the central positions of each of the multiple objects included in the connected object group is detected by calculating the angle between the horizontal or vertical direction.

[0053] The method of detecting the angle of inclination is not limited to the above-mentioned method. For example, the central coordinates of each object included in the scanned image are obtained as a central coordinate group, and the central coordinate group is gradually rotated in minute units such as 0.1 degrees with the central position of the scanned image as the center of rotation. At this time, the angle of rotation when the proportion of the central coordinate group aligned in the horizontal or vertical direction is the highest may be detected as the angle of inclination of the scanned image.

[0054] The rotation correction process is a process of rotating a scanned image in 90 degree increments so that the image of characters included in the scanned image is oriented upright. For example, a method of the rotation correction process is to first use the scanned image after tilt correction as a reference image, and prepare four images: the reference image, an image obtained by rotating the reference image by 90 degrees, an image obtained by rotating the reference image by 180 degrees, and an image obtained by rotating the reference image by 270 degrees. Then, a simple character recognition process that allows high-speed processing is performed on each image, and the image with the largest number of characters recognized with a certainty level equal to or higher than a predetermined value is used as the image after the rotation correction. However, the method of the rotation correction process is not limited to this method.

[0055] Fig. 8 is a diagram showing an example of extraction of a foreground region in the image processing unit 432 according to the first embodiment. Fig. 8(a) shows an example of a scanned image 800 after correction processes such as tilt correction and rotation correction have been applied. Fig. 8(b) shows the result of extraction of the foreground region for the scanned image 800 shown in Fig. 8(a), and the rectangle shown by the dashed line in Fig. 8(b) represents the circumscribing rectangle of the foreground region. Note that, although attributes are determined for all the foreground regions in Fig. 8(b), only the attributes for some of the foreground regions are shown by attribute names excluding the word "attribute", such as "TEXT".

[0056] After S602, in S603, the image processing unit 432 performs character recognition processing on each of the multiple character string areas (foreground areas that belong to the TEXT attribute) acquired in S602, and acquires character strings included in each character string area. By executing the processes of S602 and S603, character strings included in the scanned image are extracted. Next, in S604, the image processing unit 432 extracts character strings that match the item attributes from the multiple character strings acquired in S603, according to the file name setting rule acquired in S601. When the file name setting rule of the data structure shown as an example in FIG. 7 is used, character strings that match each of the item attributes of "title", "date", and "id" except for "sep" indicating a delimiter are extracted.

[0057] A method for extracting a string that matches an item attribute may use rule-based item extraction. Rule-based item extraction is a method for extracting items based on the position, font size, and format of a string in a scanned image, as well as the relative position to a key string corresponding to a specific keyword, according to a predefined rule. Specifically, in rule-based item extraction, for example, a string that exists at the top of a scanned image and has a font size larger than a predetermined size is extracted as an item corresponding to a form title indicated by the item attribute "title". In addition, as an item corresponding to a form issue date indicated by the item attribute "date", a string such as "2019-04-01" that matches the format "yyyy-mm-dd" is extracted using "date" or "issue date" as a key string. In addition, as an item corresponding to a form number indicated by the item attribute "id", a string such as "1234" is extracted using "issue number", "No.", or "#" as a key string.

[0058] The method of extracting a character string that matches an item attribute is not limited to rule-based item extraction. For example, a character string at a corresponding position may be extracted as a character string corresponding to an item attribute by using information on the position of each item attribute that is prepared in advance for each form. Also, a character string corresponding to an item attribute may be extracted by using a trained model obtained in advance as a result of training such as machine learning.

[0059] After S604, in S605, image processing unit 432 uses the character string extracted in S604 to generate a file name in accordance with the file name setting rule acquired in S601. When using the file name setting rule with the data structure shown as an example in Fig. 7, image processing unit 432 generates a file name as follows. Specifically, image processing unit 432 replaces "sep" indicating a delimiter with a delimiter such as an underscore (_), and generates a file name by combining the character strings corresponding to each item attribute extracted in S604 with the delimiter between them. After S605, image processing unit 432 ends the process of the flowchart shown in Fig. 6, i.e., the file name generation process.

[0060] [Scanned document list screen] FIG. 9 is a diagram showing an example of a scanned form list screen (hereinafter referred to as a "list screen") 900 displayed by the MFP 110 or the client PC 111 according to the first embodiment. Hereinafter, the list screen 900 will be described as being displayed on the operation unit 220 of the MFP 11. By referring to the list screen 900, the user can browse a list of scanned image data corresponding to scanned image files before transmission to the cloud storage 130, for which the scanning process of the paper form and the image analysis process of the scanned image have been completed. The list screen 900 includes a display area 901, a send button 902, an edit button 903, and a delete button 904. The display area 901 is a screen area for displaying a list of scanned image data for which the scanning process of the paper form and the image analysis process of the scanned image, which are the processes from S505 to S509, have been completed. The display area 901 includes a display area 905, a display area 906, and a display area 907. The display area 905 displays the date and time when the scanning process of S505 was performed. The file name generated in S508 is displayed in a display area 906. The destination of the scanned image file generated based on the scanned image data is displayed in a display area 907.

[0061] Send button 902 is a button for instructing MFP collaboration service 120 to send the scanned image file to a destination displayed in display area 907. When the user presses send button 902 with any scanned image data selected in display area 901, MFP 110 sends information about the set file name to MFP collaboration service 120. When MFP collaboration service 120 receives the file name information, it creates a scanned image file with the received file name and sends the created scanned image file to the specified destination. Note that when the transmission of the scanned image file is successfully completed, the display of the scanned image data corresponding to that scanned image file is deleted from display area 901.

[0062] The edit button 903 is a button for transitioning to a file name setting screen, which will be described later. When the edit button 903 is pressed with any scanned image data selected in the display area 901 by a user operation, the MFP 110 displays a file name setting screen (not shown in Fig. 9) on the operation unit 220. The user can edit and change the file name corresponding to the scanned image data selected on the list screen 900 on the file name setting screen. Details of the file name setting screen will be described later with reference to Fig. 10.

[0063] The delete button 904 is a button for deleting scanned image data. When the user presses the delete button 904 with any scanned image data selected in the display area 901, the MFP 110 transmits a deletion request for the selected scanned image data to the MFP collaboration service 120. Upon receiving the deletion request, the MFP collaboration service 120 deletes the scanned image data held by the MFP collaboration service 120. In addition to the scanned image data, the MFP collaboration service 120 may also delete the data, information, and processing logs generated in S508 to S510, such as the confidential image data and file name information corresponding to the scanned image data.

[0064] [File name setting screen] Fig. 10 is a diagram showing an example of a file name setting screen displayed by the MFP 110 or the client PC 111 according to the first embodiment. In the following, the file name setting screen will be described as being displayed on the operation unit 220 of the MFP 11. Fig. 10 shows, as an example, a file name setting screen based on the processing result of the image analysis processing on the scanned image 800 shown in Fig. 8(a). The file name setting screen shown as an example in Fig. 10 includes a preview area 1000, buttons 1001 to 1003, a text field 1004, one or more item areas 1005, a check box 1009, a transmission image confirmation button 1010, and a decision button 1011.

[0065] Scanned image 800 obtained by scanning a paper document is displayed as a preview image in preview area 1000. Also, in preview area 1000, a rectangle indicating the position of a character string area including the character string used to generate a file name in the file name generation process of S508 is superimposed and highlighted on the preview image of scanned image 800. Furthermore, MFP 110 also superimposes and highlights a rectangle indicating the position of a character string area set by a user instruction on the preview image of scanned image 800 displayed in preview area 1000.

[0066] Buttons 1001 and 1002 are buttons for changing the display magnification of the preview image, and when button 1001 is pressed by a user, the preview image is enlarged and displayed, and when button 1002 is pressed, the preview image is reduced and displayed. Button 1003 is a button for enlarging or reducing the preview image so that the preview image fits the width or height of preview area 1000.

[0067] The text field 1004 displays the file name generated in the file name generation process of S508. Each item area 1005 includes an attribute name area 1006, a partial preview area 1007, and a text field 1008. The attribute name area 1006 displays the name of the item attribute used in the file name in the file name generation process of S508. The partial preview area 1007 displays an image that is partially cut out from the scanned image 800 of a character string area that includes the character string used in the file name in the file name generation process of S508. The text field 1008 is an area where the character recognition result of the character string area is displayed, and where the user can correct the character recognition result if there is an error in the character recognition result, for example.

[0068] If the file name generation process of S508 fails to extract a character string that matches an item attribute indicated in the file name setting rule, a message such as "Please select from the preview image" may be displayed in partial preview area 1007. In this case, text field 1008 may be displayed in an empty state. For example, in this case, MFP 101 may accept from the user a designation of an arbitrary character string area in preview area 1000 for each item attribute for which a character string could not be extracted. When MFP 101 accepts the designation of a character string area from the user, it updates the display of partial preview area 1007 and text field 1008 to match the designated character string area. Note that the correction procedure is similar when there is an error in determining a character string area that includes a character string used in a file name in the file name generation process of S508.

[0069] The check box 1009 is for setting whether or not data of the anonymized image generated in the anonymization process in S509 corresponding to the scanned image 800 may be transmitted to the developer as data for improving the service. When the user approves the transmission of the anonymized image data to the developer, the user can give an instruction indicating approval, such as checking the check box 1009. The transmitted image confirmation button 1010 is a button for confirming the anonymized image (hereinafter referred to as a "transmitted image for improvement") transmitted to the developer for improving the service. When the transmitted image confirmation button 1010 is pressed by a user's operation, the MFP 101 displays a screen for confirming the transmitted image for improvement (hereinafter referred to as a "confirmation screen for the image for improvement"), not shown in FIG. 10. Details of the confirmation screen for the image for improvement will be described later with reference to FIG. 11.

[0070] When the user has confirmed or corrected the file name and pressed decision button 1011, MFP 101 transmits information about the file name set on the file name setting screen to MFP cooperation service 120. After receiving the file name information transmitted from MFP 101, MFP cooperation service 120 executes the processes from S516 to S519 shown in FIG. 5. First, MFP cooperation service 120 executes processes related to the generation and transmission of a scanned image file (S516, S517). Next, if check box 1009 is checked, MFP cooperation service 120 executes processes related to the generation and transmission of a concealed image file (S518, S519). After transmitting the file name information, MFP 101 displays, for example, a list screen 900 shown in FIG. 9.

[0071] [Image confirmation screen for improvement] 11 is a diagram showing an example of an improvement image confirmation screen displayed by the MFP 110 or the client PC 111 according to the first embodiment. In the following, the improvement image confirmation screen will be described as being displayed on the operation unit 220 of the MFP 11. A user can confirm the transmission image for improvement, that is, the anonymized image, by referring to the improvement image confirmation screen. The improvement image confirmation screen includes a preview area 1100, buttons 1101 to 1103, and a back button 1104. In the preview area 1100, the anonymized image, which is the transmission image for improvement, is displayed as a preview image.

[0072] Buttons 1101 and 1102 are buttons for changing the display magnification of the preview image, and when the button 1101 is pressed by a user operation, the preview image is enlarged and displayed, and when the button 1102 is pressed, the preview image is reduced and displayed. Button 1103 is a button for enlarging or reducing the preview image so that the preview image fits the width or height of the preview area 1100. When the user has finished checking the concealed image and pressed the back button 1104 by the user operation, the MFP 101 returns the display to the file name setting screen shown as an example in FIG.

[0073] [Details of the masking process] FIG. 12 is a flowchart showing an example of the flow of the concealment process in the image processing unit 432 according to the first embodiment, and is a flowchart showing an example of a detailed flow of the process in S509 in FIG. 5. The process of the flowchart is executed after S508. First, in S1201, the image processing unit 432 acquires information on a character string included in a scanned image (hereinafter referred to as "character string information"). In this embodiment, the character string information acquired in S603 in FIG. 6 is acquired. Note that the character string information acquired in S1201 is not limited to the character string information acquired in S603. For example, in S1201, the image processing unit 432 may separately perform a process of detecting character string information on the scanned image, and acquire information on the character string detected by the detection process as the character string information.

[0074] Next, in S1202, the image processing unit 432 performs an anonymization process on the character string information acquired in S1201, and acquires character string information after the anonymization process (hereinafter referred to as "confidential character information"). Details of the anonymization process on the character string information will be described later. Next, in S1203, the image processing unit 432 generates a concealed image using the character string information acquired in S1201 and the concealed character information acquired in S1202. Specifically, the image processing unit 432 generates the concealed image by rendering, as a font, information on the character code of the character string after the anonymization process, which is included in the concealed character information, and reconstructing an image.

[0075] FIG. 13 is a flowchart showing an example of the flow of the anonymization process in the image processing unit 432 according to the first embodiment, and is a flowchart showing an example of a detailed flow of the process in S1202 in FIG. 12. The process of the flowchart is executed after S1201. First, in S1301, the image processing unit 432 detects character string information corresponding to information not corresponding to confidential information (hereinafter referred to as "non-confidential information") from among the multiple character string information acquired in S1201. For example, a non-confidential character string list in which information of character strings corresponding to non-confidential information is listed is prepared in advance. As a method for creating the non-confidential character string list, for example, information of character strings that frequently appear in a form may be listed in advance as non-confidential information. Specifically, for example, the non-confidential character string list is a list of information of character strings that generally frequently appear in a form, such as the title of a form, key names of items, and column names of a table structure, as non-confidential information. The non-confidential character string list is not limited to the above-mentioned list, and may be, for example, a list of information of character strings that a user has determined to be allowed to disclose to a developer. The image processing unit 432 uses the non-confidential character string list to detect, from among the multiple pieces of character string information acquired in S1201, character string information included in the non-confidential character string list as character string information corresponding to non-confidential information.

[0076] FIG. 14 is a diagram showing an example of a process for detecting character string information corresponding to non-confidential information using the non-confidential character string list according to the first embodiment. FIG. 14(a) shows an example of a scan image 1400 before the anonymization process, and the scan image 1400 is the same as the scan image 800 shown in FIG. 8(a). FIG. 14(b) shows an example of a plurality of pieces of character string information detected from the scan image 1400 shown in FIG. 14(a) and a position of a character string area including each character string in the scan image 1400, showing a state after the process of S1201. FIG. 14(c) shows an example of a non-confidential character string list. FIG. 14(d) shows an area of ​​each character string corresponding to non-confidential information included in the non-confidential character string list shown in FIG. 14(c) among the plurality of character strings shown in FIG. 14(b), showing a state after the process of S1301. Specifically, in FIG. 14(d), an area surrounded by a dashed rectangle is an area of ​​a character string corresponding to non-confidential information among the character strings shown in FIG. 14(b).

[0077] After S1301, in S1302, the image processing unit 432 detects string information corresponding to confidential information from the string information acquired in S1201, excluding the string information corresponding to non-confidential information detected in S1301. Specifically, for example, the image processing unit 432 detects string information corresponding to confidential information using a personal information detection model, which is a trained model obtained as a result of learning such as machine learning. Here, the personal information detection model is a model trained to detect information that can identify a specific company or individual, such as an organization name, a name, an address, a telephone number, an email address, and a date, using string information as an input. The personal information detection model can detect string information corresponding to confidential information such as personal information by linking it to attributes of a string that may include confidential information, such as an organization name, a name, an address, a telephone number, an email address, and a date.

[0078] In addition, the personal information detection model may be configured to detect string information corresponding to confidential information, for example, by using information on strings present around the region of the string to be processed. In the case of such a model, if the string information obtained in S1201 after removing the string information corresponding to non-confidential information detected in S1301 is input to the personal information detection model, the detection accuracy may be reduced. Therefore, in the case of such a model, it is preferable to input the string information obtained in S1201 as it is to the personal information detection model. In this case, the string information detected as confidential information output by the personal information detection model, excluding the string information corresponding to non-confidential information detected in S1301, may be treated as the string information finally detected as confidential information.

[0079] The method of detecting character string information corresponding to confidential information is not limited to the method of using a personal information detection model, which is a trained model. For example, the method of detecting character string information corresponding to confidential information may be any method capable of detecting character string information corresponding to confidential information from among the character string information acquired in S1201 by linking it with an attribute of a character string that may include confidential information. For example, the image processing unit 432 may detect character string information corresponding to confidential information using a confidential character string list prepared in advance. The image processing unit 432 may detect character string information corresponding to confidential information using a regular expression, a confidential character string list, a list of key character strings, and a group of rules that specify the correspondence between them. Here, as an example of the rule, a rule that uses a regular expression of a date to detect character string information of a date attribute is considered. As another example of the rule, a rule that uses a specific key character string such as "TEL" to detect a character string including a number that exists to the right or below the key character string in the scanned image is considered as a character string of a phone number attribute is considered.

[0080] After S1302, in S1303, the image processing unit 432 performs an anonymization process on the character string information corresponding to the confidential information detected in S1302 among the character string information acquired in S1201, as a first anonymization process. Specifically, for example, the image processing unit 432 anonymizes the character string information corresponding to the confidential information detected in S1302 by replacing the character string information corresponding to the confidential information detected in S1302 with information of another character string not including the confidential information. Note that, in the replacement with another character string information, it is preferable that the character string information is anonymized while retaining information indicating what the original character string information was as much as possible. Therefore, the image processing unit 432 switches the method of anonymization process for each attribute of the character string, for example, as follows.

[0081] For example, if the string information corresponding to the confidential information is a string of attributes such as an organization name or a name, a dummy string list that lists dummy string information corresponding to each attribute is prepared in advance. The image processing unit 432 replaces the string information corresponding to the confidential information with dummy string information randomly selected from the dummy string list. For example, as the dummy string list corresponding to the attribute of an organization name, information on strings representing organization names such as "Sample Co., Ltd." or "Sample Information Association" is listed in advance. Also, as the dummy string list corresponding to the attribute of a name, information on strings representing names such as "Shimomaruko Taro" or "Kawasaki Hanako" is listed in advance.

[0082] The dummy string list corresponding to the name attribute may be prepared by separately listing dummy string information corresponding to the surname and the first name. In this case, the image processor 432 may randomly select dummy string information corresponding to the surname and dummy string information corresponding to the first name, and combine them to create dummy string information for replacing the string information indicating the name. When selecting the dummy string information, the image processor 432 may select one of one or more pieces of dummy string information having the same number of characters as the original string. Through such selection, the string information corresponding to confidential information is replaced with dummy string information having the same number of characters and the same string attribute.

[0083] Also, for example, if the character string information corresponding to the confidential information is information on a character string with an attribute such as a telephone number or a date, the image processing unit 432 generates dummy character string information by replacing the numbers while maintaining the format of the character string corresponding to the confidential information. Furthermore, the image processing unit 432 replaces the character string information corresponding to the confidential information with the generated dummy character string information. For example, if the character string information corresponding to the confidential information is information on a character string with an attribute of a telephone number such as "987-654-3210", the numbers included in the character string information are replaced with other numbers while maintaining the format of three digits, -, three digits, -, and four digits. Here, the numbers used to replace the numbers included in the character string information are selected at random, for example. If the character string information corresponding to the confidential information is information on a character string with an attribute of a date, the image processing unit 432 may randomly select and replace the numbers included in the character string information with numbers that are valid as a date, rather than simply replacing the numbers included in the character string information with numbers selected at random. By making such a selection, it becomes easier to recognize that the character string information corresponding to the confidential information before the replacement was character string information with a date attribute, even in the character string information after the replacement.

[0084] Also, for example, if the character string information corresponding to the confidential information is information on a character string of an address attribute, the image processing unit 432 first randomly selects character string information from among a plurality of character string information indicating place names such as cities, wards, towns, and villages that exist. Next, the image processing unit 432 randomly generates character string information consisting of numbers indicating information after the address by randomly selecting numbers. Furthermore, the image processing unit 432 generates dummy character string information by combining the randomly selected character string information indicating the place name and the randomly generated character string information indicating information after the address. Finally, the image processing unit 432 replaces the character string information corresponding to the confidential information with the generated dummy character string information. At this time, the image processing unit 432 may generate dummy character string information having a character string length similar to that of the character string information corresponding to the confidential information. Furthermore, the image processing unit 432 may identify the country of the address indicated by the character string information corresponding to the confidential information, and then generate dummy character string information using character string information indicating a place name that exists in the identified country.

[0085] Also, for example, if the string information corresponding to the confidential information is information on a string of an attribute of an email address, the image processing unit 432 first randomly selects string information and generates dummy string information indicating the local part before @. Next, the image processing unit 432 selects one piece of dummy string information from information that lists dummy string information indicating a domain, which is prepared in advance, as dummy string information indicating the domain after @. Furthermore, the image processing unit 432 generates dummy string information corresponding to an email address by combining the generated dummy string information indicating the local part and the selected dummy string information indicating the domain. Finally, the image processing unit 432 replaces the string information corresponding to the confidential information with the generated dummy string information.

[0086] In this way, by switching the method of anonymization for each attribute of the character string and replacing character string information corresponding to confidential information with another dummy character string, it is possible to anonymize the character string while maintaining information indicating the attributes of the original character string corresponding to confidential information. Note that it does not matter if the dummy character string after replacement matches information such as an existing person, organization, address, or telephone number. This is because the confidential character information is rewritten from the original character string information corresponding to confidential information, and therefore can be treated as having no relationship to the existing character string information.

[0087] FIG. 15 is a diagram showing an example of a process of detecting character string information corresponding to confidential information and a first concealment process in the image processing unit 432 according to the first embodiment. FIG. 15(a) shows an example of information of a plurality of character strings detected from a scanned image and a character string area including each character string, which is similar to that shown in FIG. 14(b). FIG. 15(b) shows an example of character string information determined to be confidential information, and shows a state after the process of S1302. In FIG. 15(b), an area surrounded by a dashed rectangle shows an area of ​​character strings detected as character string information corresponding to confidential information. Furthermore, FIG. 15(b) shows that these character string information are linked to information indicating attributes such as a date (DATE), a telephone number (PHONE NUMBER), or an organization name (ORGANIZATION). FIG. 15(c) shows a state after the first concealment process is performed on the character strings detected as character string information corresponding to confidential information shown in FIG. 15(b), that is, a state after the process of S1303. As shown in the area surrounded by a dashed rectangle in FIG. 15(c), the character string in the area corresponding to that area in FIG. 15(b) has been replaced with a different character string in FIG. 15(c).

[0088] After S1303, in S1304, the image processing unit 432 performs a second anonymization process. Specifically, in the process of S1304, the image processing unit 432 first identifies remaining character string information from the character string information acquired in S1201, excluding the non-confidential information detected in S1301 and the character string information corresponding to the confidential information detected in S1302. In the process of S1304, the image processing unit 432 then performs a second anonymization process on the identified remaining character string information.

[0089] The remaining string information, excluding the string information corresponding to non-confidential information or confidential information among the string information acquired in S1201, may contain confidential information that the user does not want to disclose to the developer. The confidential information that the user does not want to disclose to the developer is, for example, transaction details such as product names or amounts in accounting documents such as invoices or delivery notes, or contract details in contract documents such as contracts. The confidential information detection process in S1302 detects string information of attributes that are generally considered confidential, and therefore cannot detect string information whose attributes cannot be defined in advance. Therefore, such string information remains without being concealed in the first concealment process in S1303. In addition, string information of attributes that should be detected in the confidential information detection process in S1302 may not be detected. In this case, the string information remains without being concealed in the first concealment process in S1303. This is because it is difficult for the confidential information detection process in S1302 using the trained model or the rule group to detect string information of attributes that should be detected with a recall rate of 100% without causing detection failure.

[0090] That is, the aim of the second concealment is to realize complete concealment of all character string information that may contain confidential information among the character string information acquired in S1201 and that could not be detected in S1302. In the second concealment process, a concealment process is performed to abstract the character string information that is the target of the concealment process. Specifically, for example, for each character that constitutes a character string, a process is performed in which symbol characters are not changed, and characters of other character types are replaced with a predetermined character according to the character type of the target character on a character-by-character basis (hereinafter referred to as "character masking process"). For example, if the character type of the target character is an alphabetic character, the target character is replaced with the character "X", and if it is a number, the target character is replaced with the character "0".

[0091] FIG. 16 is a diagram showing an example of the second anonymization process by the image processing unit 432 according to the first embodiment. FIG. 16(a) shows an example of a region of a character string to be subjected to the second anonymization process. Specifically, in FIG. 16(a), the region of a character string to be subjected to the second anonymization process is shown by a region surrounded by a dashed rectangle. The character string information to be subjected to the second anonymization process is character string information that is not detected as character string information corresponding to non-classified information or confidential information in S1301 or S1302 among the character string information acquired in S1201. FIG. 16(b) shows a result of performing the second anonymization process on the character string to be subjected to the second anonymization process shown in FIG. 16(a). As shown in the region surrounded by a dashed rectangle in FIG. 16(b), in the character string to be subjected to the second anonymization process shown in FIG. 16(a), characters other than symbols are subjected to obscuring process according to the character type. By such a character masking process, it is possible to completely conceal the character string that is the target of the second concealment process while maintaining the number and type of characters of the character string.

[0092] The second concealment process is not limited to the above-mentioned method. For example, in the concealment process, the image processing unit 432 may replace characters in the target character string whose character type is a number with another number selected at random. The image processing unit 432 may abstract the target character string composed of characters other than numbers by replacing the character string with a word corresponding to a higher concept on a word-by-word basis using a thesaurus dictionary or the like. The image processing unit 432 may replace the character string according to a regular expression that matches the target character string. For example, when the target character string matches a regular expression of a date, the image processing unit 432 may replace the character string with another character string that is valid as a date. When checking the match with the regular expression, the image processing unit 432 may also include, in the target character string, character string information corresponding to non-confidential information detected in S1301 among the character string information acquired in S1201. This is because, in the process of detecting character string information corresponding to non-confidential information in S1301, if the non-confidential character string list contains symbols such as "-" or " / " that indicate date separators, the character string information may not match the regular expression. For example, in this case, the character string information indicating the date will not match the regular expression for dates.

[0093] After S1304, the image processing unit 432 ends the process of the flowchart shown in Fig. 13, i.e., the process of S1202 shown in Fig. 12. After that, in S1203, the image processing unit 432 generates a concealed image based on the result of the process of the flowchart shown in Fig. 13. Note that an image obtained by deleting the rectangle drawn by the dashed lines from the image shown in Fig. 16(b) is an example of a concealed image corresponding to the scanned image 1400 shown in Fig. 14(a).

[0094] According to the MFP interaction service 120 configured as above, in a scanned image including confidential information, it is possible to completely conceal the portion other than the non-confidential information while maintaining the non-confidential information. Therefore, according to the MFP interaction service 120, it is possible to generate an anonymized image that does not include information that should be concealed. This makes it easier for a developer to obtain consent from a user for the collection of data used in improving a service. Furthermore, according to the MFP interaction service 120, in a scanned image including confidential information, it is possible to conceal the confidential information while maintaining information indicating the attributes of the original character string. Furthermore, according to the MFP interaction service 120, in a scanned image including confidential information, it is possible to conceal the non-confidential information and character string information other than the confidential information while maintaining as much information as possible by abstracting information of the original character string. This enables a developer to collect data of an anonymized image in which as much information contained in the scanned image as possible is left, and it becomes possible to improve a service with higher accuracy.

[0095] <Second embodiment> In the first embodiment, a concealed image is generated by replacing a character string in a scanned image with concealed character information obtained by performing concealment processing on character string information extracted from a scanned image. However, when a character string in a scanned image is simply replaced, it may be difficult for a user to recognize whether the character string is the same as the character string included in the scanned image or the character string replaced by the concealment processing when checking a transmission image for improvement. In addition, it may be difficult to recognize what attribute of confidential information the replaced character string corresponds to and has been detected as character string information.

[0096] In this embodiment, the visibility of the result of the anonymization process in the anonymized image is improved by using the detection result of the character string information corresponding to non-classified information and the detection result of the character string information corresponding to classified information obtained in the process of the anonymization process for the character string. Specifically, the visibility of the result of the anonymization process is improved by making the expression of the character string area including the character string as it is in the scanned image and the expression of the character string area including the character string replaced by the anonymization process different from each other in the anonymized image. Note that only the points that are different from the first embodiment will be described below.

[0097] FIG. 17 is a diagram showing an example of the result of the anonymization process by the image processing unit 432. FIG. 17(a) shows an example of the anonymized character information obtained by the anonymization process in S1202 shown in FIG. 12, and the position of the character string area after each anonymization process. FIG. 17(b) shows an example of a data structure showing the information and area of ​​each character string after the anonymization process shown in FIG. 17(a). Note that the data structure shown in FIG. 17(b) shows a part of the data showing the information and area of ​​the character string after the anonymization process shown in FIG. 17(a). In "regions" showing a character string area group, information showing the position of the character string area and information showing the result of the anonymization process for the character string information are stored for each character string area extracted in S1201 shown in FIG. 12. In "text" of each character string area, the character string information acquired in S1201 is stored. The "anonymized_text" stores character string information after replacement by the anonymization process in S1303 or S1304, and "anonymized_text" exists only for character string regions that have been subjected to the anonymization process.

[0098] In "type", information indicating the attribute of the string information corresponding to each string area is stored. In the "type" of a string area detected in S1301 as string information corresponding to non-confidential information, "NON-CONFIDENTIAL" indicating that the string information is non-confidential information is stored. In addition, in the "type" of a string area detected in S1302 as string information corresponding to confidential information, information according to the attribute of the string information is stored. For example, if the attribute of the string information detected in S1302 is date, "DATE" is stored in "type", and if it is a telephone number, "PHONE_NUMBER" is stored. In addition, "OTHER" is stored in the "type" of a string area not detected in S1301 or S1302 as string information corresponding to non-confidential information or confidential information.

[0099] "Rect" stores information indicating the position and size of the character string area. Specifically, for example, "x" and "y" store the X and Y coordinates indicating the top left position of the character string area in the scanned image, and "width" and "height" store the number of pixels in the X and Y directions of the character string area in the scanned image. Also, "style" stores information indicating the style of the characters that make up the character string included in the character string area. Specifically, for example, "font" stores information indicating the font type of the characters that make up the character string, and "size" stores information indicating the font size of the characters that make up the character string.

[0100] FIG. 18 is a flowchart showing an example of the flow of the anonymization process in the image processing unit 432 according to the second embodiment, and is a flowchart showing an example of a detailed flow of the process in S509 shown in FIG. 5. The processes of S1201 and S1202 shown in FIG. 18 are similar to the processes of S1201 and S1202 shown in FIG. 12, and therefore will not be described. After S1202, in S1801, the image processing unit 432 generates an anonymized image. After S1801, the image processing unit 432 ends the process of the flowchart shown in FIG. 18. Here, the generation process of the anonymized image in S1801 is different from that of S1203 shown in FIG. 12. Specifically, in S1801, the image processing unit 432 generates an anonymized image in which the expression of the character string area is switched according to the value stored in "type" included in the data shown as an example in FIG. 17(b).

[0101] FIG. 19 is a diagram showing an example of anonymized images 1900 and 1910 generated by the image processing unit 432 according to the second embodiment. FIG. 19(a) shows an anonymized image 1900 in which the background color of a character string region in which "NON-CONFIDENTIAL" is stored in "type" is unchanged, and the background color of a character string region in which a value other than "NON-CONFIDENTIAL" is stored is changed. That is, in the anonymized image 1900 shown in FIG. 19(a), a character string region in which a value other than "NON-CONFIDENTIAL" is stored in "type" is highlighted in the display of a preview image of the confirmation screen for the improvement image by changing the background color. In this way, the anonymization process makes the representation of the region of the character string that is not replaced and the representation of the region of the character string that is replaced different from each other, thereby improving the visibility for users and developers.

[0102] The method of switching the expression in the anonymized image is not limited to changing the background color. For example, the image processing unit 432 may generate an anonymized image in which the color, thickness, or font of the characters constituting the character string included in the character string area is switched. Also, for example, the image processing unit 432 may generate an anonymized image in which an annotation indicating that the character string after the anonymization process, that is, the character string replaced by the anonymization process, is added to the vicinity of the region of the character string. FIG. 19(b) shows an anonymized image 1910 in which the above-mentioned annotation is added to the vicinity of the corresponding character string area for the anonymized image 1900 shown in FIG. 19(a). In this way, by adding an annotation corresponding to the attribute of the character string information, the user and developer can easily visually recognize on the anonymized image what attribute the character string information corresponding to the confidential information has been detected as. Note that the switching of the expression according to the attribute of the character string information is not limited to adding an annotation. For example, the image processing unit 432 may generate a concealed image in which the color, thickness, font, or background color of characters is changed according to the attribute of the character string.

[0103] In the present embodiment, as an example, a form in which the region of the character string replaced by the anonymization process or the expression of the characters constituting the character string is changed when generating a concealed image has been described, but the change of the expression is not limited to the change of the expression in the concealed image. For example, the expression of the region of the character string replaced by the anonymization process may be changed as follows. For example, in the generation process of the concealed image in S1801, the image processing unit 432 performs the same process as the generation process of the concealed image in S1203, as in the image processing unit 432 according to the first embodiment. Meanwhile, the MFP 110 or the client PC 111 changes the expression of the region in the preview image corresponding to the region of the character string replaced by the anonymization process in the display of the preview region 1100 in the confirmation screen of the improvement image shown as an example in FIG. 11. Specifically, the MFP 110 or the client PC 111 may change the expression of the region in the preview image by overlaying the region in the preview image in the confirmation screen of the improvement image, or by attaching an image indicating an annotation.

[0104] In the second embodiment, the image processing system 100 is configured to switch between a representation of a character string region where the character string is not replaced and a representation of a character string region where the character string is replaced, using the detection results of non-confidential information and confidential information obtained in the process of concealing the character string. This allows users and developers to easily visually identify character string regions where the character string has been replaced. As a result, users and developers can easily identify character string regions that may contain confidential information.

[0105] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0106] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0107] <Configuration of the present disclosure> The present disclosure includes the following configurations, methods, and programs.

[0108] [Configuration 1] An acquisition means for acquiring information on a first character string which is a character string included in a document image; a first detection means for detecting a second character string corresponding to the non-confidential character string from among information on the first character strings, using information on a plurality of non-confidential character strings corresponding to the non-confidential information; a second detection means for detecting a third character string corresponding to confidential information from among the first character strings; a concealment means for performing a first concealment process on information of the third character string, and performing a second concealment process different from the first concealment process on information of a fourth character string obtained by excluding the second character string and the third character string from the first character string; a generating means for generating a concealed image corresponding to the document image based on information of the second character string which is a detection result of the first detecting means, a processing result of the first concealment processing, and a processing result of the second concealment processing; 13. An image processing device comprising:

[0109] [Configuration 2] the first detection means detects the second character string using a non-confidential character string list that lists information on a plurality of non-confidential character strings corresponding to the non-confidential information; 2. The image processing device according to claim 1,

[0110] [Configuration 3] the second detection means detects the third character string from a fifth character string remaining after excluding the second character string from the first character string; 3. The image processing device according to configuration 1 or 2,

[0111] [Configuration 4] the second detection means detects the third character string by using information on an attribute of a character string that may include the confidential information; 4. The image processing device according to any one of configurations 1 to 3,

[0112] [Configuration 5] the second detection means detects the third character string in association with an attribute of a character string that may include the confidential information; the anonymization means switches a method of the first anonymization process for the third character string in accordance with an attribute of the third character string; 5. The image processing device according to configuration 4,

[0113] [Configuration 6] the first anonymization process is a process of replacing the third character string with a dummy character string corresponding to an attribute of the third character string; 6. The image processing device according to configuration 5,

[0114] [Configuration 7] the second concealment process is a process of replacing the fourth character string with an abstract character string obtained by abstracting the fourth character string; 7. The image processing device according to any one of configurations 1 to 6,

[0115] [Configuration 8] The character string obtained by abstracting the fourth character string has the same number of characters as the fourth character string; 8. The image processing device according to configuration 7,

[0116] [Configuration 9] The character string obtained by abstracting the fourth character string is a character string in which one or more characters constituting the fourth character string are replaced with an arbitrarily selected character for each character; 9. The image processing device according to configuration 7 or 8,

[0117] [Configuration 10] The character string obtained by abstracting the fourth character string is a character string in which one or more characters constituting the fourth character string are replaced with a predetermined character; 9. The image processing device according to configuration 7 or 8,

[0118] [Configuration 11] the generating means generates the anonymized image expressing at least one of a character string resulting from the first anonymization process and a character string resulting from the second anonymization process in a manner different from a manner in which the second character string is expressed; 11. The image processing device according to any one of configurations 1 to 10,

[0119] [Configuration 12] the second detection means detects the third character string in association with an attribute of a character string that may include the confidential information; the generating means generates the anonymized image by switching an expression of the third character string according to an attribute of the third character string; 12. The image processing device according to any one of configurations 1 to 11,

[0120] [method] An acquisition step of acquiring information on a first character string which is a character string included in a document image; a first detection step of detecting a second character string corresponding to the non-confidential character string from among information on the first character strings, using information on a plurality of non-confidential character strings corresponding to the non-confidential information; a second detection step of detecting a third character string corresponding to confidential information from among the first character strings; a concealment process of performing a first concealment process on information of the third character string, and performing a second concealment process different from the first concealment process on information of a fourth character string obtained by excluding the second character string and the third character string from the first character string; a generating step of generating a concealed image corresponding to the document image based on information of the second character string which is a detection result of the first detection step, a processing result of the first concealment processing, and a processing result of the second concealment processing; 13. An image processing method comprising:

[0121] [program] A program for causing a computer to function as the image processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0122] 120 MFP Collaboration Service 300 Image Processing Server 432 Image Processing Unit

Claims

1. an acquiring means for acquiring information on a first character string which is a character string included in a document image; a first detection means for detecting a second character string corresponding to the non-confidential character string from among information on the first character strings, using information on a plurality of non-confidential character strings corresponding to the non-confidential information; a second detection means for detecting a third character string corresponding to confidential information from among the first character strings; a concealment means for performing a first concealment process on information of the third character string, and performing a second concealment process different from the first concealment process on information of a fourth character string obtained by excluding the second character string and the third character string from the first character string; a generating means for generating a concealed image corresponding to the document image based on information of the second character string which is a detection result of the first detecting means, a processing result of the first concealment processing, and a processing result of the second concealment processing; 13. An image processing device comprising:

2. the first detection means detects the second character string using a non-confidential character string list that lists information on a plurality of non-confidential character strings corresponding to the non-confidential information; 2. The image processing device according to claim 1,

3. the second detection means detects the third character string from a fifth character string remaining after excluding the second character string from the first character string; 2. The image processing device according to claim 1,

4. the second detection means detects the third character string by using information on an attribute of a character string that may include the confidential information; 2. The image processing device according to claim 1,

5. the second detection means detects the third character string in association with an attribute of a character string that may include the confidential information; the anonymization means switches a method of the first anonymization process for the third character string in accordance with an attribute of the third character string; 5. The image processing device according to claim 4,

6. the first anonymization process is a process of replacing the third character string with a dummy character string corresponding to an attribute of the third character string; 6. The image processing device according to claim 5,

7. the second concealment process is a process of replacing the fourth character string with an abstract character string obtained by abstracting the fourth character string; 2. The image processing device according to claim 1,

8. a character string obtained by abstracting the fourth character string has the same number of characters as the fourth character string; 8. The image processing device according to claim 7,

9. The character string obtained by abstracting the fourth character string is a character string in which one or more characters constituting the fourth character string are replaced with an arbitrarily selected character for each character; 8. The image processing device according to claim 7,

10. the character string obtained by abstracting the fourth character string is a character string in which one or more characters constituting the fourth character string are replaced with a predetermined character; 8. The image processing device according to claim 7,

11. the generating means generates the anonymized image expressing at least one of a character string resulting from the first anonymization process and a character string resulting from the second anonymization process in a manner different from an expression of the second character string; 2. The image processing device according to claim 1,

12. the second detection means detects the third character string in association with an attribute of a character string that may include the confidential information; the generating means generates the concealed image by switching an expression of the third character string according to an attribute of the third character string; 2. The image processing device according to claim 1,

13. An acquisition step of acquiring information on a first character string which is a character string included in a document image; a first detection step of detecting a second character string corresponding to the non-confidential character string from among information on the first character strings, using information on a plurality of non-confidential character strings corresponding to the non-confidential information; a second detection step of detecting a third character string corresponding to confidential information from among the first character strings; a concealment step of performing a first concealment process on information of the third character string, and performing a second concealment process different from the first concealment process on information of a fourth character string obtained by excluding the second character string and the third character string from the first character string; a generating step of generating a concealed image corresponding to the document image based on information of the second character string which is a detection result of the first detection step, a processing result of the first concealment processing, and a processing result of the second concealment processing; 13. An image processing method comprising:

14. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12.

Images