Image processing device, image processing method, and program

Abstract

To provide an image processing device, method, and program that conceal objects containing confidential information while retaining information necessary for development in a form image.SOLUTION: An image processing device is configured to perform the following steps acquiring one or more character strings contained in a document image, non-character string objects that are objects other than character strings belonging to a first attribute, and non-character string objects belonging to a second attribute that is different from the first attribute, performing a first concealing process on at least some of the one or more character strings, generating character strings after the concealing process, performing a second concealing process on the non-character string object belonging to the first attribute, generating a non-character string object after the concealing process, and generating a concealing image corresponding to the document image using the string after the concealing process, the non-character string object after the concealing process, and the non-character string object belonging to the second attribute.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating an image in which confidential information included in a form is concealed. [Background technology]

[0002] There is a system that manages forms by generating form images by scanning and digitizing paper forms and automatically setting file names based on the contents of the forms to files containing the data of the generated form images. In order for developers to improve such systems to better suit the actual usage of users, it is important to collect and utilize form image data corresponding to forms actually used in the user's business (hereinafter referred to as "real forms"). In this case, in order to collect form image data corresponding to real forms, it is necessary to obtain consent from users to collect form image data, and to ensure that confidential information is not leaked by performing masking processing on confidential information contained in the form images of the real forms. On the other hand, in order for developers to use form image data to improve the system, it is necessary to collect form image data in a state where information contained in the real forms is left as much as possible.

[0003] In order to achieve both concealment of confidential information and retention of information necessary for system improvement, there is a method of detecting character strings and objects other than character strings that correspond to confidential information from a form image and replacing them with other objects that do not contain confidential information. Patent Document 1 discloses a technology that uses an execution module such as a neural network that is the result of learning such as machine learning to detect text and objects other than character strings that contain confidential information in a form image and conceal them. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2022-506866 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it is difficult to accurately detect objects containing confidential information from a form image. Therefore, the technology disclosed in Patent Document 1 has a problem that confidential information may be leaked if an object to be concealed cannot be detected. The present disclosure is intended to solve this problem, and aims to provide a technology that can conceal character strings that may contain confidential information and information of objects other than character strings in a form image while leaving information necessary for improving the system. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes an acquisition means for acquiring one or more character strings and non-character string objects, which are objects other than character strings, included in a document image, the non-character string objects belonging to a first attribute and the non-character string objects belonging to a second attribute different from the first attribute; a first processing means for performing a first anonymization process on at least a portion of the one or more character strings to generate a character string after the anonymization process; a second processing means for performing a second anonymization process on the non-character string objects belonging to the first attribute to generate a non-character string object after the anonymization process; and a generation means for generating a concealed image corresponding to the document image using the character strings after the anonymization process, the non-character string objects after the anonymization process, and the non-character string objects belonging to the second attribute. Effect of the Invention

[0007] According to the present disclosure, in a form image, it is possible to conceal character strings that may include confidential information and information on objects other than character strings, while leaving information necessary for system improvement. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 is a block diagram showing an example of a configuration of an image processing system. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of an MFP. [Diagram 3] 2 is a block diagram showing an example of the hardware configuration of a client PC and an image processing server that provides an MFP cooperation service. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing system. [Diagram 5] FIG. 2 is a sequence diagram showing an example of a processing flow of the image processing system. [Figure 6] 10 is a flowchart showing an example of the flow of a file name generation process in an image processing unit. [Figure 7] FIG. 11 is a diagram illustrating an example of a data structure of a file name setting rule. [Figure 8] 11A and 11B are diagrams illustrating an example of a result of extraction of a foreground region in an image processing unit. [Figure 9] FIG. 13 illustrates an example of a scanned form list screen. [Figure 10] FIG. 13 illustrates an example of a file name setting screen. [Figure 11] FIG. 13 is a diagram showing an example of a confirmation screen for an improvement image. [Figure 12] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit. [Figure 13] 11A and 11B are diagrams illustrating an example of a concealment process for character string information in an image processing unit. [Figure 14] 11 is a diagram illustrating an example of a concealment process for a confidential non-character string object in an image processing unit. FIG. [Figure 15] FIG. 11 is a diagram showing an example of a process for generating a concealed image in an image processing unit. [Figure 16] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit according to the second embodiment. [Figure 17] 13 is a flowchart showing an example of the flow of a concealment process in an image processing unit according to the third embodiment. [Figure 18]13A to 13C are diagrams illustrating an example of a background image generation process in an image processing unit according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the disclosed solution.

[0010] First Embodiment [Overall configuration] 1 is a block diagram showing an example of the configuration of an image processing system 100 according to the first embodiment. The image processing system 100 includes an MFP (Multifunction Peripheral) 110, a client PC 111, an MFP cooperation service 120, and cloud storages 130 and 140. The MFP 110 and the client PC 111 are communicably connected to a server that provides various services on the Internet via a LAN (Local Area Network).

[0011] The MFP 110 is a multifunction machine having multiple functions such as a scan function and a print function, and is an example of an image reading device. The client PC 111 is configured with a personal computer or the like, and receives a service requested from the MFP cooperation service 120. The MFP cooperation service 120 is a service provided by one device configured with an image processing server or the like, or by the cooperation of multiple devices. Specifically, the MFP cooperation service 120 has a function of saving an image file obtained by scanning a paper document or the like in the MFP 110 in a storage device within the device, or transferring the image file to a device that provides a service capable of saving files, such as a storage service. The cloud storages 130 and 140 are services provided by one device configured with a storage server or the like, or by the cooperation of multiple devices. Specifically, the cloud storages 130 and 140 are services that save a file received via the Internet in a storage device within the device, or read a file specified using a web browser or the like from the storage device and transmit it to an external device.

[0012] In this embodiment, cloud storage 130 is used to store image files obtained by scanning paper forms and the like by users (hereinafter simply referred to as "users") who use MFP linkage service 120, and to reuse the image files. Cloud storage 140 is also used to accumulate data used by providers or developers (hereinafter simply referred to as "developers") of MFP linkage service 120 when improving the service.

[0013] Hereinafter, image processing system 100 will be described as including MFP 110, client PC 111, MFP collaboration service 120, and cloud storages 130 and 140, but the configuration of image processing system 100 is not limited to this. For example, MFP 110 may serve as at least one of client PC 111 and MFP collaboration service 120. Furthermore, MFP collaboration service 120 may be provided by a device located on a LAN rather than on the Internet. Furthermore, cloud storage 130 may be replaced with a service provided by a mail server or the like, and an image file obtained by scanning may be attached to an email and transmitted to an external device.

[0014] [MFP hardware configuration] 2 is a block diagram showing an example of the hardware configuration of the MFP 110 according to the first embodiment. The MFP 110 has, as its hardware configuration, a control unit 210, an operation unit 220, a printer 221, a scanner 222, and a modem 223. The control unit 210 controls the overall operation of the MFP 110. The control unit 210 has, as its hardware configuration, a CPU 211, a ROM 212, a RAM 213, and a HDD 214. In addition to the above-mentioned hardware configuration, the control unit 210 also has an operation unit I / F 215, a printer I / F 216, a scanner I / F 217, a modem I / F 218, and a network I / F 219.

[0015] The CPU 211 reads and executes a control program stored in the ROM 212, thereby executing and controlling various functions of the MFP 110, such as reading and printing images, and transmitting and receiving data. The RAM 213 is used as a temporary storage area such as the main memory and work area of ​​the CPU 211. Note that in this embodiment, one CPU 211 executes each process shown in the flowcharts described later using one memory (RAM 213 or HDD 214), but is not limited to this. For example, a configuration may be adopted in which multiple CPUs, or multiple RAMs or HDDs work together to execute each process shown in the flowcharts described later. The HDD 214 is a large-capacity storage device that stores image data, various programs, and the like.

[0016] The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The operation unit 220 is provided with a touch panel, a keyboard, or the like, and accepts operations, inputs, or instructions from a user. In the following, the operation unit 220 will be described as being configured with a liquid crystal display device having a touch panel function. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data for printing is transferred from the control unit 210 to the printer 221 via the printer I / F 216, and an image corresponding to the image data is printed on a recording medium by the printer 221.

[0017] The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads an original placed on a platen or an ADF (Auto Document Feeder) not shown in FIG. 2 to generate image data, and outputs the generated image data to the control unit 210 via the scanner I / F 217. The MFP 110 has a copy function of printing an image corresponding to the image data generated by the scanner 222 with the printer 221, as well as a function of transmitting the image data to the outside or attaching it to an e-mail and transmitting it to the outside. The modem I / F 218 is an interface that connects the modem 223 and the control unit 210. The modem 223 communicates image data by facsimile with a facsimile device connected via a PSTN (Public Switched Telephone Network). The network I / F 219 is an interface that connects the control unit 210 (MFP 110) to a computer network such as a LAN. The MFP 110 transmits image data and various information to various services on the Internet via a network I / F 219, and receives various information from various services.

[0018] [Hardware configuration of client PC and device providing MFP linkage service 120] Fig. 3 is a block diagram showing an example of the hardware configuration of a client PC 111 according to the first embodiment, and an image processing server 300 that provides an MFP linkage service 120. Specifically, Fig. 3(a) shows an example of the hardware configuration of the image processing server 300, and Fig. 3(b) shows an example of the hardware configuration of the client PC 111.

[0019] The image processing server 300 includes a control unit 310. The control unit 310 controls the overall operation of the image processing server 300 to provide the MFP cooperation service 120. The control unit 310 includes a CPU 311, a ROM 312, a RAM 313, a HDD 314, and a network I / F 315. The CPU 311 controls the overall operation of the image processing server 300 by reading and executing a control program stored in the ROM 312. The RAM 313 is used as a temporary storage area such as a main memory and a work area of ​​the CPU 311. The HDD 314 is a large-capacity storage device that stores image data and various programs. The network I / F 315 is an interface that connects the control unit 310 (image processing server 300) to a computer network such as the Internet. The image processing server 300 receives a processing request from an external device such as the MFP 110 via the network I / F 315, and transmits and receives various information to and from the external device.

[0020] The client PC 111 includes a control unit 320. The control unit 320 controls the overall operation of the client PC 111. The control unit 320 includes a CPU 321, a ROM 322, a RAM 323, a HDD 324, and a network I / F 325. The CPU 321 controls the overall operation of the client PC 111 by reading and executing a control program stored in the ROM 322. The RAM 323 is used as a temporary storage area such as a main memory and a work area of ​​the CPU 321. The HDD 324 is a large-capacity storage device that stores image data and various programs. The network I / F 325 is an interface that connects the control unit 320 (client PC 111) to a computer network such as a LAN. The client PC 111 transmits a processing request to the MFP cooperation service 120, etc., and transmits and receives various information to and from an external device via the network I / F 325.

[0021] [Image processing system functional configuration] 4 is a block diagram showing an example of the functional configuration of the image processing system 100 according to the first embodiment. The MFP 110 has, as its functional configuration, a native function unit 410 and an additional function unit 420. The native function unit 410 has a scan execution unit 411 and an image data storage unit 412, and the additional function unit 420 has a display control unit 421, a scan instruction unit 422, and a cooperation service request unit 423. Each unit that the MFP 110 has as its functional configuration is realized by execution of a program by the CPU 211.

[0022] Each unit included in the native function unit 410 is a functional configuration that is provided as standard in the MFP 110. In contrast, each unit included in the additional function unit 420 is a functional configuration that functions by executing an application program that is additionally installed in the MFP 110. The function of the additional function unit 420 is realized by installing an application program based on Java (registered trademark), for example. Therefore, the function can be easily added to the MFP 110 by this installation. Note that other additional application programs for realizing functional configurations not shown in FIG. 4 may be installed in the MFP 110 additional function unit 420.

[0023] The display control unit 421 displays a UI (User Interface) screen for receiving operations by the user on the operation unit 220. Specifically, the display control unit 421 displays UI screens such as an input screen for authentication information for accessing the MFP linkage service 120, an operation screen for setting scans and starting scans, and a preview screen for viewing images obtained by scanning. The scan instruction unit 422 instructs the scan execution unit 411 to execute scan processing based on scan settings according to operations or instructions from the user input via the UI screen.

[0024] The scan execution unit 411 receives an instruction to execute a scan process from the scan instruction unit 422, and controls the operation of the scanner 222 via the scanner I / F 217 according to the scan setting included in the instruction. The scanner 222 generates image data corresponding to the document (hereinafter referred to as "scanned image") by reading a document placed on a document table or the like according to the control of the scan execution unit 411. The generated scanned image data is sent to the image data storage unit 412 via the scan execution unit 411. The scan execution unit 411 also generates a scanned image identifier expressed by a number or one or more symbols or alphabets, etc., for uniquely identifying the generated scanned image data. The generated scanned image identifier is sent to the image data storage unit 412 and the scan instruction unit 422. The image data storage unit 412 associates the scanned image data received from the scan execution unit 411 with the scanned image identifier and stores them in the HDD 214.

[0025] The scan instruction unit 422 acquires scan image data corresponding to the scan image identifier received from the scan execution unit 411 from the image data storage unit 412. The scan instruction unit 422 instructs the cooperation service request unit 423 to make a request to the MFP cooperation service 120 to perform processing on the acquired scan image data.

[0026] Upon receiving an instruction from the scan instruction unit 422, the collaboration service request unit 423 makes a request to the MFP collaboration service 120 to perform various processes on the scanned image data. For example, the collaboration service request unit 423 makes a login request to the MFP collaboration service 120, a request for image analysis of the scanned image data, and a request for transmission of the scanned image data after various processes. The communication between the collaboration service request unit 423 and the MFP collaboration service 120 is performed using a protocol such as REST (Representational state transfer) or SOAP (Simple object access protocol). Note that the communication is not limited to REST or SOAP and may be performed using other communication methods.

[0027] The MFP interaction service 120 has, as its functional components, a request control unit 431, an image processing unit 432, a cloud storage access unit 433, a data management unit 434, and a display control unit 435. Each unit that the MFP interaction service 120 has as its functional components is realized by the execution of a program by the CPU 311. The request control unit 431 waits in a state in which it can receive a processing request from an external device. When the request control unit 431 receives a processing request, it instructs the image processing unit 432, the cloud storage access unit 433, or the data management unit 434 to execute processing corresponding to the request as appropriate in response to the received request. The image processing unit 432 performs recognition and processing on an image, such as character area analysis processing, character recognition processing, concealment processing, rotation processing, and tilt correction processing.

[0028] The cloud storage access unit 433 issues processing requests to the cloud storages 130 and 140. Cloud storage services generally use protocols such as REST or SOAP to publish various interfaces for saving files in cloud storage and retrieving files stored in cloud storage. The cloud storage access unit 433 operates cloud storage using the published interface of the cloud storage service. The data management unit 434 holds user information and various setting data managed by the MFP cooperation service 120.

[0029] The display control unit 435 receives a processing request from a web browser operating on a terminal such as a personal computer or a mobile terminal (not shown in FIG. 1) connected via the Internet, and returns screen configuration information required for screen display. Here, the screen configuration information is composed of, for example, a markup language such as HTML (HyperText Markup Language) or a style sheet such as CSS (Cascading Style Sheets). The web browser generates a screen corresponding to the screen configuration information by rendering the screen configuration information received from the display control unit 435, and displays the screen on a display device. A user can check or change user information and scan settings registered in the MFP linkage service 120 by viewing and operating the screen displayed by the web browser.

[0030] In the above description, an additional program is installed in MFP 110 to add a functional configuration to additional function unit 420, but the present invention is not limited to this configuration. For example, client PC 111 may have a functional configuration equivalent to additional function unit 420, and an additional program may be installed in client PC 111 to add the functional configuration to client PC 111.

[0031] [Image processing system processing flow] Fig. 5 is a sequence diagram showing an example of the flow of processing in the image processing system 100 according to the first embodiment. Specifically, Fig. 5 is a sequence diagram showing an example of the flow of processing in the image processing system 100 when image data obtained by scanning in the MFP 110 is converted into a file and transmitted to the cloud storages 130 and 140. Here, the description will focus on the exchange between each device. Note that Fig. 5 shows a form in which the MFP 110 exchanges with the MFP cooperation service 120, but the process of obtaining the analysis result or displaying the screen, which will be described later, may be executed by the client PC 111 instead of the MFP 110.

[0032] In a normal state, the MFP 110 displays on the operation unit 220 a main screen on which buttons for implementing each function provided by the MFP 110 are arranged. An additional application (hereinafter referred to as a "scan application") for transmitting scan image data to cloud storages 130 and 140 is installed in the MFP 110. As a result, buttons for implementing functions realized by executing the scan application are displayed on the main screen. When this button is pressed by the user, a screen required for operations for transmitting scan image data to cloud storages 130 and 140 is displayed on the operation unit 220, and the process shown in the sequence diagram of FIG. 5 is executed. In the following description, the symbol "S" means a step.

[0033] First, in S501, display control unit 421 of MFP 110 displays on operation unit 220 a login screen that accepts input of authentication information for accessing MFP linkage service 120. A user inputs authentication information into the login screen via operation unit 220, and then performs an operation for a login request to MFP linkage service 120. Next, in S502, MFP 110 makes a login request to MFP linkage service 120 using authentication information such as a user name and password accepted via the login screen.

[0034] Next, in S503, the request control unit 431 of the MFP collaboration service 120 executes login processing. Specifically, as the login processing, the request control unit 431 verifies whether authentication information such as a user name and password included in the login request is correct (also called "user authentication"), and if correct, returns an access token to the MFP 110. Subsequent various processing requests made from the MFP 110 (scan application) to the MFP collaboration service 120 are made using this access token, and the user to be processed can be identified by this access token. The user authentication is performed using a known method such as authentication by Basic authentication, Digest authentication, or the like, or authorization using OAuth.

[0035] When receiving information indicating completion of the login process from the MFP cooperation service 120, in S504, the display control unit 421 of the MFP 110 displays a scan setting screen on the operation unit 220. The scan setting screen is a UI screen for receiving input of settings related to the scan mode of the paper form, the file format when the scanned image data obtained by scanning is saved as an image file, and the destination of the image file. Next, when the "Start Scan" button is pressed by the user's operation, in S505, the MFP 110 executes a scan process on the paper form to be scanned placed on the platen or ADF, etc., according to various scan settings set on the scan setting screen. This scan process digitizes the paper form, and scan image data is generated. Next, in S506, the MFP 110 transmits an image analysis request of the image indicated by the scan image data (scan image) to the MFP cooperation service 120 together with the information on the scan setting set in S504 and the scan image data generated in S505. The scan setting information, the scanned image data, and the image analysis request for the scanned image sent from the MFP 110 are received by a request control unit 431 of the MFP collaboration service 120.

[0036] When the request control unit 431 of the MFP collaboration service 120 receives the image analysis request for the scanned image, in S507 it instructs the image processing unit 432 of the MFP collaboration service 120 to execute image analysis processing on the received scanned image. In response to the instruction, the image processing unit 432 starts image analysis processing on the scanned image. Note that the request control unit 431, which has received the scan setting information, holds the received scan setting information so that it can be used in subsequent processing. When the request control unit 431 receives the image analysis request for the scanned image, it returns a processId, which is an identifier that uniquely indicates the requested image analysis processing, to the MFP 110 as a response to the request, without waiting for the image analysis processing of the scanned image by the image processing unit 432 to end. The processId sent from the MFP collaboration service 120 is received by the scan application of the MFP 110.

[0037] In S508, the image processing unit 432 performs image analysis processing such as character recognition on the scanned image, and generates a file name to be used when saving the image file based on the result of the image analysis processing. Details of the file name generation processing will be described later. Next, in S509, the image processing unit 432 performs an anonymization processing on the scanned image, and generates an image in which confidential information has been anonymized (hereinafter referred to as an "anonymized image"). Here, confidential information to be anonymized by the anonymization processing is not limited to confidential information such as transaction contents or contract contents included in the form. Confidential information includes all information that the user does not want to disclose to the developer, including, for example, personal information included in the form. Details of the anonymization processing will be described later.

[0038] Next, in S510, the image processing unit 432 generates screen configuration information that is the basis of a file name setting screen displayed on the MFP 110, using result information indicating the processing results of S508 and S509. Here, the result information is the scanned image after the image analysis processing (called the "analyzed scanned image"), information indicating the recognition result of the character recognition, information indicating the generation result of the file name, and data of the concealed image. The analyzed scanned image is an image after image processing such as tilt correction is performed on the scanned image transmitted in S506 in S508. After completing the processing of S510, the image processing unit 432 transmits a completion notification indicating the completion of the image analysis processing corresponding to processId to the request control unit 431.

[0039] In S511, the MFP 110 uses the processId received as a response to the image analysis request in S506 to periodically check the processing status of the image analysis process corresponding to the processId with the MFP cooperation service 120, such as every several hundred milliseconds to every few milliseconds. Note that "periodic" here is not necessarily limited to a timing strictly at a certain interval, but includes a timing at an approximately certain interval. The MFP 110 may also irregularly check the processing status of the image analysis process corresponding to the processId. The processing of S511 is continued until a response indicating the completion of the image analysis process of the MFP cooperation service 120 corresponding to the processId is obtained.

[0040] When the MFP cooperation service 120 receives the request to confirm the processing status of S511, the MFP cooperation service 120 confirms the processing status of the image analysis processing corresponding to the processId, and returns information on the processing status as a response. A character string indicating the current processing status is stored in "status" of the processing status information returned as a response. For example, when the character string "processing" is stored in "status", the response indicates that the image analysis processing is being performed by the MFP cooperation service 120. Also, for example, when the character string "completed" is stored in "status", the response indicates that the image analysis processing is completed. Note that after the image analysis processing is completed, that is, when "completed" is stored in "status", the response stores location information for acquiring screen configuration information corresponding to the processId together with the processing status information. Also, in addition to the processing status information and location information, the response may store information such as the analysis result of the scanned image and the scan settings. Also, when the image analysis processing fails, a character string other than "processing" or "completed", such as "failed", may be stored in "status".

[0041] When the MFP 110 detects the completion of the image analysis process from the response to the confirmation of the processing status of the image analysis process in S512, in S513, the MFP 110 acquires screen configuration information corresponding to the processId using the location information included in the response. For example, if the location information includes URL (Uniform Resource Locator) information indicating the location of the screen configuration information, the MFP 110 acquires the screen configuration information by accessing the URL. The screen configuration information acquired in S513 is information that is the basis of a file name setting screen that displays result information indicating the processing results of S508 and S509.

[0042] Next, in S514, the MFP 110 generates and displays a file name setting screen based on the screen configuration information acquired in S513, for example, by rendering the screen configuration information. Here, the file name setting screen is a UI screen for setting a file name. Details of the file name setting screen will be described later with reference to FIG. 10. Next, when a file name is set by a user operation on the file name setting screen and then an operation to confirm the file name is performed, the MFP 110 transmits information on the set file name, etc. to the MFP cooperation service 120 in S515. The file name information, etc. transmitted from the MFP 110 is received by the request control unit 431 of the MFP cooperation service 120. Upon receiving the file name information, the request control unit 431 starts a file generation process and returns a response indicating that the information has been normally received to the MFP 110. Upon receiving the response, the MFP 110 ends the series of processes from S504 onwards, returns to the process of S504, and displays the scan setting screen on the operation unit 220.

[0043] In S516, request control unit 431 generates a file corresponding to the scanned image data (hereinafter referred to as a "scanned image file"). Specifically, first, request control unit 431 acquires information on the file format of the file to be transmitted to cloud storage 130 from the information on the scan settings held in S507. Next, request control unit 431 generates a scanned image file corresponding to the scanned image data based on the information. Next, in S517, request control unit 431 assigns the file name received in S515 to the file name of the scanned image file generated in S516, and transmits the scanned image file after the assignment of the file name to cloud storage 130. When cloud storage 130 receives the scanned image file, it transmits a completion notification to MFP cooperation service 120 as a response indicating completion of reception.

[0044] Next, in S518, the request control unit 431 generates a file corresponding to the data of the anonymized image (hereinafter referred to as a "anonymized image file"). Specifically, first, the request control unit 431 acquires information on the file format of the file to be transmitted to the cloud storage 140 from information such as the storage format of the image data and the method of assigning a file name, which are registered in advance in the MFP cooperation service 120 by a developer or the like. Next, the request control unit 431 generates a anonymized image file corresponding to the anonymized image data based on the information. Note that the anonymized image file is assigned a file name determined based on the information on the method of assigning a file name. Note that the request control unit 431 may use, for example, the file name received in S515 as the file name to be assigned to the anonymized image file. Also, for example, the request control unit 431 may use a file name determined based on information indicating the date and time when the scan process was performed, information indicating the logged-in user, or the like.

[0045] Next, in S519, request control unit 431 transmits the anonymized image file generated in S518 after the file name is assigned to cloud storage 140. When cloud storage 140 receives the anonymized image file, it transmits a completion notification to MFP collaboration service 120 as a response indicating completion of reception. When MFP collaboration service 120 receives a response corresponding to the processing of S519, it ends the series of processing from S507 onwards, and waits in a state in which it is able to accept the next login request or image analysis request.

[0046] In addition, in the process of S519, the request control unit 431 may transmit information that can be obtained through a series of processes to the cloud storage 140 together with the concealed image file. The information that can be obtained through a series of processes may be, for example, a processing log for a character string area of ​​a character string including a character string used as a file name that can be obtained in the file name generation process in S508. In addition, for example, the information that can be obtained through a series of processes may be information on a file name set by a user in S514 and an operation log on a file name setting screen. In addition, for example, the information that can be obtained through a series of processes may be information indicating the presence or absence of confidential information, or a processing log of the position or attributes of confidential information in a scanned image that can be obtained in the concealment process in S509.

[0047] [File name generation process] FIG. 6 is a flowchart showing an example of the flow of file name generation processing in the image processing unit 432 according to the first embodiment, and is a flowchart showing an example of a detailed flow of processing in S508. First, in S601, the image processing unit 432 acquires information on a file name setting rule from the data management unit 434. The file name setting rule defines in advance the item attributes of a character string that can be used as a file name, and the order of arrangement of a plurality of character strings corresponding to the plurality of item attributes. The image processing unit 432 generates a file name according to the file name setting rule. The file name setting rule may be input via a dedicated UI for rule creation, or may be specified by importing a file indicating data of the file name setting rule in advance.

[0048] FIG. 7 is a diagram showing an example of a data structure of a file name setting rule according to the first embodiment. In the "key", "filename" indicating that it is used as a file name is set. In the "value", information on a plurality of item attributes used as a file name and information indicating a delimiter to be inserted between a plurality of character strings corresponding to the plurality of item attributes are set. In the example of the data structure shown in FIG. 7, "title", "sep", "date", "sep", and "id" are set in the "value". For example, in this case, a character string is generated in which character strings corresponding to the item attributes "title", "date", and "id" are arranged in order as a file name, and a character or character string corresponding to "sep" indicating a delimiter is inserted between the character strings.

[0049] After S601, in S602, the image processing unit 432 performs character region analysis processing on the scanned image. The character region analysis processing is processing for separating an image into a foreground region and a background region, classifying the foreground region into a character string region and other regions, and acquiring the character string region in the image. Specifically, for example, in S602, the image processing unit 432 first performs contour tracing on the scanned image binarized into black and white, and extracts a cluster of pixels surrounded by a contour composed of black pixels, that is, a cluster of black pixels. Subsequently in S602, for a cluster of black pixels having an area larger than a predetermined size, the image processing unit 432 performs contour tracing on the white pixels inside the cluster in the same manner as in the case of black pixels, thereby extracting a cluster of white pixels. At this time, for a cluster of white pixels having an area equal to or larger than a predetermined size, a cluster of black pixels is recursively extracted from the inside of the cluster. The image processing unit 432 separates the cluster of black pixels thus obtained from the background region as a foreground region. Next, in S602, the image processing unit 432 classifies each of the separated foreground regions into one of the following attributes based on the size and shape of the region:

[0050] For example, the image processing unit 432 determines that a foreground region whose aspect ratio is close to 1 and whose size is within a certain range corresponds to a character. Furthermore, the image processing unit 432 determines that a plurality of regions determined to correspond to characters are close to each other and can be grouped in an aligned state as a region corresponding to a character string (hereinafter referred to as a "TEXT attribute"). Furthermore, the image processing unit 432 determines that a foreground region whose shape is flat corresponds to a line (hereinafter referred to as a "LINE attribute"). Furthermore, the image processing unit 432 determines that a foreground region whose size is equal to or larger than a predetermined size and which contains a plurality of aligned rectangular clusters of white pixels corresponds to a table (hereinafter referred to as a "TABLE attribute"). Furthermore, the image processing unit 432 determines that a region in which irregular foreground regions are scattered corresponds to a photograph (hereinafter referred to as a "PHOTO attribute"). Furthermore, the image processing unit 432 determines that a foreground region that does not belong to any of the TEXT attribute, LINE attribute, TABLE attribute, and PHOTO attribute is a region corresponding to a picture (hereinafter referred to as a "PICTURE attribute").

[0051] Subsequently in S602, the image processing unit 432 acquires, as a character string region, the foreground region determined to have the TEXT attribute from among the foreground regions classified by attribute in this manner. Note that, in order to increase the accuracy of acquiring the character string region, the image processing unit 432 may perform correction processing such as tilt correction processing or rotation correction processing on the scanned image in advance, and perform the above-mentioned separation and classification processing on the scanned image after the processing.

[0052] In the skew correction process, first, the angle of the skew of the scanned image is detected, and the skew of the scanned image is corrected by rotating the scanned image in the opposite direction by an angle corresponding to the detected skew angle. The skew that is the subject of the skew correction occurs, for example, when a paper document is scanned in a skewed state due to wear of the rollers in the document feeder of the scanner 222 when the paper document is scanned. Or, when the document is printed, the paper is printed in a skewed state, and the character string that is the subject of the skew correction is also printed in a skewed state with respect to the paper. As a method for detecting the skew angle, first, a plurality of character objects included in the scanned image are detected, and adjacent character objects within a predetermined distance in the horizontal or vertical direction are connected. That is, when the document is written horizontally, adjacent character objects in the horizontal direction are connected, and when the document is written vertically, adjacent character objects in the vertical direction are connected, and the character objects that constitute one character string are identified. Next, the angle of a straight line (e.g., a regression line) calculated based on the central positions of each of the multiple objects included in the connected object group is detected by calculating the angle between the horizontal or vertical direction.

[0053] The method of detecting the angle of inclination is not limited to the above-mentioned method. For example, the central coordinates of each object included in the scanned image are obtained as a central coordinate group, and the central coordinate group is gradually rotated in minute units such as 0.1 degrees with the central position of the scanned image as the center of rotation. At this time, the angle of rotation when the proportion of the central coordinate group aligned in the horizontal or vertical direction is the highest may be detected as the angle of inclination of the scanned image.

[0054] The rotation correction process is a process of rotating a scanned image in 90 degree increments so that the image of characters included in the scanned image is oriented upright. For example, a method of the rotation correction process is to first use the scanned image after tilt correction as a reference image, and prepare four images: the reference image, an image obtained by rotating the reference image by 90 degrees, an image obtained by rotating the reference image by 180 degrees, and an image obtained by rotating the reference image by 270 degrees. Then, a simple character recognition process that allows high-speed processing is performed on each image, and the image with the largest number of characters recognized with a certainty level equal to or higher than a predetermined value is used as the image after the rotation correction. However, the method of the rotation correction process is not limited to this method.

[0055] Fig. 8 is a diagram showing an example of extraction of a foreground region in the image processing unit 432 according to the first embodiment. Fig. 8(a) shows an example of a scanned image 800 after correction processes such as tilt correction and rotation correction have been applied. Fig. 8(b) shows the result of extraction of the foreground region for the scanned image 800 shown in Fig. 8(a), and the rectangle shown by the dashed line in Fig. 8(b) represents the circumscribing rectangle of the foreground region. Note that, although attributes are determined for all the foreground regions in Fig. 8(b), only the attributes for some of the foreground regions are shown by attribute names excluding the word "attribute", such as "TEXT".

[0056] After S602, in S603, the image processing unit 432 performs character recognition processing on each of the multiple character string areas (foreground areas that belong to the TEXT attribute) acquired in S602, and acquires character strings included in each character string area. By executing the processes of S602 and S603, character strings included in the scanned image are extracted. Next, in S604, the image processing unit 432 extracts character strings that match the item attributes from the multiple character strings acquired in S603, according to the file name setting rule acquired in S601. When the file name setting rule of the data structure shown as an example in FIG. 7 is used, character strings that match each of the item attributes of "title", "date", and "id" except for "sep" indicating a delimiter are extracted.

[0057] A method for extracting a string that matches an item attribute may use rule-based item extraction. Rule-based item extraction is a method for extracting items based on the position, font size, and format of a string in a scanned image, as well as the relative position to a key string corresponding to a specific keyword, according to a predefined rule. Specifically, in rule-based item extraction, for example, a string that exists at the top of a scanned image and has a font size larger than a predetermined size is extracted as an item corresponding to a form title indicated by the item attribute "title". In addition, as an item corresponding to a form issue date indicated by the item attribute "date", a string such as "2019-04-01" that matches the format "yyyy-mm-dd" is extracted using "date" or "issue date" as a key string. In addition, as an item corresponding to a form number indicated by the item attribute "id", a string such as "1234" is extracted using "issue number", "No.", or "#" as a key string.

[0058] The method of extracting a character string that matches an item attribute is not limited to rule-based item extraction. For example, a character string at a corresponding position may be extracted as a character string corresponding to an item attribute by using information on the position of each item attribute that is prepared in advance for each form. Also, a character string corresponding to an item attribute may be extracted by using a trained model obtained in advance as a result of training such as machine learning.

[0059] After S604, in S605, image processing unit 432 uses the character string extracted in S604 to generate a file name in accordance with the file name setting rule acquired in S601. When using the file name setting rule with the data structure shown as an example in Fig. 7, image processing unit 432 generates a file name as follows. Specifically, image processing unit 432 replaces "sep" indicating a delimiter with a delimiter such as an underscore (_), and generates a file name by combining the character strings corresponding to each item attribute extracted in S604 with the delimiter between them. After S605, image processing unit 432 ends the process of the flowchart shown in Fig. 6, i.e., the file name generation process.

[0060] [Scanned document list screen] FIG. 9 is a diagram showing an example of a scanned form list screen (hereinafter referred to as a "list screen") 900 displayed by the MFP 110 or the client PC 111 according to the first embodiment. Hereinafter, the list screen 900 will be described as being displayed on the operation unit 220 of the MFP 11. By referring to the list screen 900, the user can browse a list of scanned image data corresponding to scanned image files before transmission to the cloud storage 130, for which the scanning process of the paper form and the image analysis process of the scanned image have been completed. The list screen 900 includes a display area 901, a send button 902, an edit button 903, and a delete button 904. The display area 901 is a screen area for displaying a list of scanned image data for which the scanning process of the paper form and the image analysis process of the scanned image, which are the processes from S505 to S509, have been completed. The display area 901 includes a display area 905, a display area 906, and a display area 907. The display area 905 displays the date and time when the scanning process of S505 was performed. The file name generated in S508 is displayed in a display area 906. The destination of the scanned image file generated based on the scanned image data is displayed in a display area 907.

[0061] Send button 902 is a button for instructing MFP collaboration service 120 to send the scanned image file to a destination displayed in display area 907. When the user presses send button 902 with any scanned image data selected in display area 901, MFP 110 sends information about the set file name to MFP collaboration service 120. When MFP collaboration service 120 receives the file name information, it creates a scanned image file with the received file name and sends the created scanned image file to the specified destination. Note that when the transmission of the scanned image file is successfully completed, the display of the scanned image data corresponding to that scanned image file is deleted from display area 901.

[0062] The edit button 903 is a button for transitioning to a file name setting screen, which will be described later. When the edit button 903 is pressed with any scanned image data selected in the display area 901 by a user operation, the MFP 110 displays a file name setting screen (not shown in Fig. 9) on the operation unit 220. The user can edit and change the file name corresponding to the scanned image data selected on the list screen 900 on the file name setting screen. Details of the file name setting screen will be described later with reference to Fig. 10.

[0063] The delete button 904 is a button for deleting scanned image data. When the user presses the delete button 904 with any scanned image data selected in the display area 901, the MFP 110 transmits a deletion request for the selected scanned image data to the MFP collaboration service 120. Upon receiving the deletion request, the MFP collaboration service 120 deletes the scanned image data held by the MFP collaboration service 120. In addition to the scanned image data, the MFP collaboration service 120 may also delete the data, information, and processing logs generated in S508 to S510, such as the confidential image data and file name information corresponding to the scanned image data.

[0064] [File name setting screen] Fig. 10 is a diagram showing an example of a file name setting screen displayed by the MFP 110 or the client PC 111 according to the first embodiment. In the following, the file name setting screen will be described as being displayed on the operation unit 220 of the MFP 11. Fig. 10 shows, as an example, a file name setting screen based on the processing result of the image analysis processing on the scanned image 800 shown in Fig. 8(a). The file name setting screen shown as an example in Fig. 10 includes a preview area 1000, buttons 1001 to 1003, a text field 1004, one or more item areas 1005, a check box 1009, a transmission image confirmation button 1010, and a decision button 1011.

[0065] Scanned image 800 obtained by scanning a paper document is displayed as a preview image in preview area 1000. Also, in preview area 1000, a rectangle indicating the position of a character string area including the character string used to generate a file name in the file name generation process of S508 is superimposed and highlighted on the preview image of scanned image 800. Furthermore, MFP 110 also superimposes and highlights a rectangle indicating the position of a character string area set by a user instruction on the preview image of scanned image 800 displayed in preview area 1000.

[0066] Buttons 1001 and 1002 are buttons for changing the display magnification of the preview image, and when button 1001 is pressed by a user, the preview image is enlarged and displayed, and when button 1002 is pressed, the preview image is reduced and displayed. Button 1003 is a button for enlarging or reducing the preview image so that the preview image fits the width or height of preview area 1000.

[0067] The text field 1004 displays the file name generated in the file name generation process of S508. Each item area 1005 includes an attribute name area 1006, a partial preview area 1007, and a text field 1008. The attribute name area 1006 displays the name of the item attribute used in the file name in the file name generation process of S508. The partial preview area 1007 displays an image that is partially cut out from the scanned image 800 of a character string area that includes the character string used in the file name in the file name generation process of S508. The text field 1008 is an area where the character recognition result of the character string area is displayed, and where the user can correct the character recognition result if there is an error in the character recognition result, for example.

[0068] If the file name generation process of S508 fails to extract a character string that matches an item attribute indicated in the file name setting rule, a message such as "Please select from the preview image" may be displayed in partial preview area 1007. In this case, text field 1008 may be displayed in an empty state. For example, in this case, MFP 101 may accept from the user a designation of an arbitrary character string area in preview area 1000 for each item attribute for which a character string could not be extracted. When MFP 101 accepts the designation of a character string area from the user, it updates the display of partial preview area 1007 and text field 1008 to match the designated character string area. Note that the correction procedure is similar when there is an error in determining a character string area that includes a character string used in a file name in the file name generation process of S508.

[0069] The check box 1009 is for setting whether or not data of the anonymized image generated in the anonymization process in S509 corresponding to the scanned image 800 may be transmitted to the developer as data for improving the service. When the user approves the transmission of the anonymized image data to the developer, the user can give an instruction indicating approval, such as checking the check box 1009. The transmitted image confirmation button 1010 is a button for confirming the anonymized image (hereinafter referred to as a "transmitted image for improvement") transmitted to the developer for improving the service. When the transmitted image confirmation button 1010 is pressed by a user's operation, the MFP 101 displays a screen for confirming the transmitted image for improvement (hereinafter referred to as a "confirmation screen for the image for improvement"), not shown in FIG. 10. Details of the confirmation screen for the image for improvement will be described later with reference to FIG. 11.

[0070] When the user has confirmed or corrected the file name and pressed decision button 1011, MFP 101 transmits information about the file name set on the file name setting screen to MFP cooperation service 120. After receiving the file name information transmitted from MFP 101, MFP cooperation service 120 executes the processes from S516 to S519 shown in FIG. 5. First, MFP cooperation service 120 executes processes related to the generation and transmission of a scanned image file (S516, S517). Next, if check box 1009 is checked, MFP cooperation service 120 executes processes related to the generation and transmission of a concealed image file (S518, S519). After transmitting the file name information, MFP 101 displays, for example, a list screen 900 shown in FIG. 9.

[0071] [Image confirmation screen for improvement] 11 is a diagram showing an example of an improvement image confirmation screen displayed by the MFP 110 or the client PC 111 according to the first embodiment. In the following, the improvement image confirmation screen will be described as being displayed on the operation unit 220 of the MFP 11. A user can confirm the transmission image for improvement, that is, the anonymized image, by referring to the improvement image confirmation screen. The improvement image confirmation screen includes a preview area 1100, buttons 1101 to 1103, and a back button 1104. In the preview area 1100, the anonymized image, which is the transmission image for improvement, is displayed as a preview image.

[0072] Buttons 1101 and 1102 are buttons for changing the display magnification of the preview image, and when the button 1101 is pressed by a user operation, the preview image is enlarged and displayed, and when the button 1102 is pressed, the preview image is reduced and displayed. Button 1103 is a button for enlarging or reducing the preview image so that the preview image fits the width or height of the preview area 1100. When the user has finished checking the concealed image and pressed the back button 1104 by the user operation, the MFP 101 returns the display to the file name setting screen shown as an example in FIG.

[0073] [Details of the masking process] FIG. 12 is a flowchart showing an example of the flow of the concealment process in the image processing unit 432 according to the first embodiment, and is a flowchart showing an example of a detailed flow of the process in S509 in FIG. 5. The process of the flowchart is executed after S508. First, in S1201, the image processing unit 432 acquires information on a character string included in a scanned image (hereinafter referred to as "character string information"). In this embodiment, the information on the character string acquired in S603 in FIG. 6 is acquired. Note that the character string information acquired in S1201 is not limited to the character string information acquired in S603. For example, in S1201, the image processing unit 432 may separately perform a character string detection process on the scanned image and acquire information on the character string detected by the detection process as the character string information.

[0074] Next, in S1202, the image processing unit 432 performs an anonymization process on the character string information acquired in S1201, and acquires character string information after the anonymization process (hereinafter referred to as "confidential character information"). For example, a non-confidential character string list that lists character string information that does not correspond to confidential information is prepared in advance. A method for creating the non-confidential character string list, for example, may be to list character string information that frequently appears in forms as character string information that does not correspond to confidential information in advance. The image processing unit 432 acquires confidential character information by using the non-confidential character string list to replace character string information not included in the non-confidential character string list among the character string information acquired in S1201 with information of another character string.

[0075] FIG. 13 is a diagram showing an example of the concealment process for character string information in the image processing unit 432 according to the first embodiment. FIG. 13(a) shows an example of a scanned image 1300 before the concealment process is performed, and the scanned image 1300 is the same as the scanned image 800 shown in FIG. 8(a). FIG. 13(b) shows a plurality of character strings detected in the scanned image 1300 shown in FIG. 13(a) and the positions of character string areas including each character string in the scanned image 1300. Note that objects other than character strings, such as a logo 1301 and ruled lines 1302 (hereinafter referred to as "non-character string objects"), are not detected as character strings. Furthermore, a character string 1303 is a character string for which the process of S1201 failed to obtain character string information. Therefore, the non-character string objects, such as the logo 1301 and ruled lines 1302, and the character string 1303 are not included in FIG. 13(b).

[0076] FIG. 13(c) shows an example of the non-confidential character string list. The image processing unit 432 performs a process of replacing a character string not included in the non-confidential character string list among the detected multiple character strings shown as an example in FIG. 13(b) with another character string, thereby acquiring confidential character information, which is information on the character string after the anonymization process. FIG. 13(d) shows an example of a character string after the anonymization process. The replacement of the character string may be performed, for example, by replacing all characters constituting the character string not included in the non-confidential character string list with a predetermined character such as "x". Note that the replacement of the character string is not limited to the above-mentioned method, and for example, each character constituting all the detected character strings may be replaced with a different character according to the character type. Furthermore, the anonymization process of the character string is not limited to the replacement of the character string, and a predetermined image not including confidential information may be pasted in the area of ​​the character string not included in the non-confidential character string list.

[0077] In addition, the concealment process of the character string is not limited to the method using the non-confidential character string list. For example, the concealment process of the character string may detect a character string corresponding to confidential information by a rule-based detection method, or may detect a character string corresponding to confidential information by using a trained model obtained as a result of learning such as machine learning. However, in these methods of directly detecting a character string corresponding to confidential information, there is a possibility that information of the character string to be concealed will not be concealed if the detection of the corresponding character string fails. Therefore, a character string corresponding to confidential information may be detected by combining a rule-based detection method or a detection method using a trained model with a method using a non-confidential character string list. By combining such detection methods, it is possible to detect a character string that is not detected by a method of directly detecting a character string corresponding to confidential information, and it is possible to conceal a character string to be concealed with a higher accuracy.

[0078] After S1202, in S1203, the image processing unit 432 acquires information on non-character string objects included in the image (hereinafter referred to as "non-character string information"). In this embodiment, information on the foreground region with an attribute other than the TEXT attribute acquired in S602 of FIG. 6 is acquired as the non-character string information. Note that the non-character string information acquired in S1203 is not limited to information on the foreground region with an attribute other than the TEXT attribute acquired in S602. For example, in S1203, the image processing unit 432 may separately perform a non-character string object detection process on the scanned image, and acquire information on the non-character string objects detected by the detection process as the non-character string information.

[0079] Non-character string information expected to be acquired in S1203 is, for example, foreground area information of a PICTURE attribute such as a stamp or logo, foreground area information of a LINE attribute such as a ruled line or underline, and foreground area information of a TABLE attribute such as a frame line. These non-character string objects are classified into objects of a first attribute and objects of a second attribute. In this embodiment, a non-character string object that may contain confidential information is classified into objects of a first attribute, and a non-character string object that may not contain confidential information is classified into objects of a second attribute, but the classification method is not limited to this. For example, an object of a PICTURE attribute may contain confidential information. In contrast, an object of a LINE attribute or a TABLE attribute is not classified into confidential information.

[0080] Non-character string objects that may contain confidential information (hereinafter referred to as "confidential non-character string objects") may be classified in more detail for subsequent concealment processing. For example, confidential non-character string objects made up of pixels of multiple colors are classified as objects with a logo attribute. In contrast, confidential non-character string objects made up of pixels only of a single color are first compared with logo images registered as graphic trademarks. Then, as a result of this comparison, confidential non-character string objects for which a similar logo image exists are classified as objects with a logo attribute, while confidential non-character string objects for which no similar logo image exists are classified as objects with a stamp attribute. Note that the method of classifying confidential non-character string objects is not limited to the method described above.

[0081] After S1203, in S1204, the image processing unit 432 performs an anonymization process on the confidential non-character string object classified as an object of the first attribute in S1203, and acquires non-character string information after the anonymization process (hereinafter referred to as "confidential non-character information"). Specifically, for example, the image processing unit 432 acquires confidential non-character information by performing a predetermined image processing on the confidential non-character string object.

[0082] FIG. 14 is a diagram showing an example of an anonymization process for confidential non-character string objects in the image processing unit 432 according to the first embodiment. FIG. 14(a) shows an example of a scan image 1400 before the anonymization process is performed, and the scan image 1400 is the same as the scan image 1300 shown in FIG. 13(a). FIG. 14(b) shows a plurality of non-character string objects detected in the scan image 1400 shown in FIG. 14(a) and the positions of each non-character string object in the scan image 1400. The image processing unit 432 performs an anonymization process for confidential non-character string objects, regarding non-character string objects classified as PICTURE attributes in the process of S602 among these non-character string objects as confidential non-character string objects. Specifically, for example, the image processing unit 432 switches the contents of the anonymization process, i.e., the contents of the image processing, according to the result of the detailed classification of the confidential non-character string objects performed in S1203.

[0083] 14(c) and (d) show an example of image processing for a confidential non-character string object. Specifically, FIG. 14(c) shows an example of image processing for a confidential non-character string object classified as a logo attribute. FIG. 14(c) shows three image processing methods as an example of image processing for a confidential non-character string object classified as a logo attribute. The first method shown in FIG. 14(c) is a method of filling the entire area corresponding to the confidential non-character string object with a specific color based on a statistical value such as an average pixel value of pixels constituting the confidential non-character string object. The second method shown in FIG. 14(c) is a method of replacing the confidential non-character string object with a dummy logo, which is an image including the character string "Logo". The third method shown in FIG. 14(c) is a method of searching for an image similar to the confidential non-character string object from a group of images not including confidential information based on the feature amount of the image, and replacing the confidential non-character string object with the similar image detected by the search. Note that the image processing method for a confidential non-character string object classified as a logo attribute shown in FIG. 14(c) is merely an example, and is not limited thereto.

[0084] FIG. 14(d) shows an example of image processing for a confidential non-character string object classified as a stamp attribute. FIG. 14(d) shows two image processing methods as an example of image processing for a confidential non-character string object classified as a stamp attribute. The first method shown in FIG. 14(d) is a method of filling the entire confidential non-character string object with a specific color based on a statistical value such as an average pixel value of pixels constituting the confidential non-character string object. The second method shown in FIG. 14(d) is a method of replacing the confidential non-character string object with a dummy logo, which is an image including the character string "Stamp". Note that the image processing method for a confidential non-character string object classified as a stamp attribute shown in FIG. 14(d) is merely an example and is not limited thereto.

[0085] This type of processing can also conceal confidential non-character string objects contained in the scanned image. In addition, the attributes of the confidential non-character string objects are further classified into more detailed categories, and different concealment processes are performed for each detailed attribute of the confidential non-character string object. This type of concealment processing can conceal confidential non-character string objects while retaining information about the location and type of confidential non-character string object present in the scanned image.

[0086] After S1204, in S1205, the image processing unit 432 generates a concealed image using the character string information acquired in S1201, the concealed character information acquired in S1202, the non-character string information acquired in S1203, and the concealed non-character information acquired in S1204. Note that the generation of a concealed image here does not mean adding a modification process to the original scanned image 1400, but means generating a new concealed image using the acquired character string information, concealed character information, non-character string information, and concealed non-character information. Specifically, the image processing unit 432 generates a concealed image by performing, for example, the following process in S1205.

[0087] First, the image processing unit 432 generates a background image that is the background of the concealed image, and has the same size as the scanned image 1400 before the concealment process, and in which all pixels are white pixels. Next, the image processing unit 432 arranges the string information, concealed character information, non-string information, and concealed non-string information acquired in any of S1201 to S1204 on the generated background image. Specifically, the image processing unit 432 arranges these strings and non-string objects on the background image so that they have the same coordinates and size as the corresponding objects in the scanned image 1400 before the concealment process. Note that in the above-mentioned arrangement process, for the strings and non-string objects concealed in S1202 or S1204, the strings and non-string objects after the concealment process are arranged on the background image, not the strings and non-string objects before the concealment process.

[0088] FIG. 15 is a diagram showing an example of a process of generating a concealed image in the image processing unit 432 according to the first embodiment. FIG. 15(a) shows an example of a scanned image 1500 before the concealment process, and the scanned image 1500 is the same as the scanned image 1300 or 1400 shown in FIG. 13(a) or 14(a). FIG. 15(b) shows the scanned image 1500 after the concealment process, that is, a concealed image 1550 corresponding to the scanned image 1500. A character string such as a character string 1501 that is acquired as character string information in S1201 and is not the subject of the concealment process in S1202 is arranged as it is in the concealed image 1550, such as a character string 1506. For a non-character string object such as a logo 1502 that is acquired as non-character string information in S1203 and is the subject of the concealment process in S1204, a non-character string object after the concealment process by image processing is arranged in the concealed image, such as a logo 1507.

[0089] Furthermore, for a character string such as character string 1503 that is acquired as character string information in S1201 and is the subject of anonymization processing in S1202, the character string after anonymization processing such as replacement with another character string is arranged in the anonymized image as character string 1508. A non-character string object such as ruled line 1504 that is acquired as non-character string information in S1203 and is not the subject of anonymization processing in S1204 is arranged as it is in the anonymized image as ruled line 1509. Here, a character string that is not acquired as character string information in S1201 such as character string 1505 is not arranged in the anonymized image as region 1510. Therefore, according to the anonymized image 1550 generated in this manner, it is possible to suppress leakage of confidential information from a region not detected as a character string in the scanned image 1500.

[0090] As described above, the MFP collaboration service 120 is configured to generate a new anonymized image from the string information and non-string information detected from the scanned image, and the concealed character information and concealed non-string information obtained by anonymizing the string information and non-string information. The MFP collaboration service 120 configured in this manner can generate an anonymized image in which information that should be concealed is concealed while retaining information such as strings that frequently appear in forms and types and positional relationships of non-string objects. Therefore, such an anonymized image can suppress leakage of confidential information from areas not detected as strings in the original scanned image, while retaining information necessary for analysis by a developer to improve the service.

[0091] As described above, MFP interaction service 120 is configured to detect character strings corresponding to confidential information from a scanned image using the non-confidential character string list, and to perform a concealment process on the detected character strings corresponding to confidential information. MFP interaction service 120 configured in this manner can prevent leakage of confidential information that may occur when only a method of directly detecting character strings corresponding to confidential information is used.

[0092] <Second embodiment> In the first embodiment, a concealment process is performed on a character string detected from a scanned image, and the concealed character string is arranged in a background image so as to have the same position and size as the character string in the scanned image, to generate a concealed image. However, if the concealed character string is simply arranged in the same position and size as the character string before the concealment process, information about the character style, such as the font, thickness, or color, of the character string before the concealment process is lost in the concealed image. Therefore, in the second embodiment, a concealed image is generated by estimating the style of characters constituting the character string detected from the scanned image, applying the estimated character style to the character string after the concealment process, and arranging the character string after the application in a background image. Note that the configuration of the image processing system according to the second embodiment and the hardware configuration and functional configuration of each device or service included in the image processing system are the same as those of the image processing system 100 according to the first embodiment. Only differences from the first embodiment will be described below.

[0093] Fig. 16 is a flowchart showing an example of the flow of the anonymization process in the image processing unit 432 according to the second embodiment, and shows a flowchart showing an example of a detailed flow of the process in S509 in Fig. 5. The difference between the flowchart shown in Fig. 16 and the flowchart shown in Fig. 12 is that S1601 is added between S1201 and S1202, and the process of S1602 is different from the generation process of the anonymized image in S1205.

[0094] After S1201, in S1601, the image processing unit 432 estimates the style of characters constituting the character string for the character string information acquired in S1201, and acquires character style information (hereinafter referred to as "character style"). The character style information is information on the appearance of the character, such as the type of font, whether it is bold, whether it is italic, whether it is underlined, whether it is struck through, and the color of the character. The image processing unit 432 may estimate the character style based on the shape of a cluster of black pixels in an image of a character string region binarized into black and white, and the positional relationship between the cluster and other clusters of black pixels, or may estimate the character style using a learned model obtained as a result of learning such as machine learning. After S1601, the process of S1202 is executed.

[0095] After S1204, in S1602, the image processing unit 432 generates a concealed image using the character string information, confidential character information, non-character string information, and confidential non-character information acquired in any of S1201 to S1204, and the character style information acquired in S1601. Specifically, the image processing unit 432 first generates an image in which all pixels are white pixels and has the same size as the scanned image before the concealment process, as a background image. Next, the image processing unit 432 arranges the character string information and confidential character information acquired in S1201 or S1202 on the generated background image. Specifically, the image processing unit 432 arranges these character strings on the background image using the character style information acquired in S1601 so that the coordinates, size, and character style are the same as those of the corresponding character strings in the scanned image before the concealment process.

[0096] Furthermore, the image processing unit 432 places the non-character string information and concealed non-character information acquired in S1203 or S1204 on the generated background image. Specifically, the image processing unit 432 places these non-character string objects on the background image so that they have the same coordinates and size as the corresponding non-character string objects in the scanned image before the concealment process. Note that in the above-mentioned placement process, for the character strings and objects concealed in S1202 or S1204, the character strings and non-character string objects after the concealment process are placed on the background image, not the character strings and non-character string objects before the concealment process.

[0097] As described above, in the second embodiment, the MFP collaboration service 120 is configured to estimate the character style of each character string detected from a scanned image, and generate a concealed image so that the character style is also the same as that of the character string included in the scanned image. With the MFP collaboration service 120 configured in this way, it is possible to leave information about the appearance of the characters on a form for anonymized character strings. Therefore, with the MFP collaboration service 120 according to the second embodiment, it is possible to leave information in the concealed image that is more useful for a developer to analyze and improve the service.

[0098] <Third embodiment> In the first embodiment, a concealed image is generated by arranging a character string and a non-character string object on a background image in which all pixels are white pixels. However, in such a background image, information about the background area of ​​the form may be lost. In this regard, in the third embodiment, a background image is generated using information about the background area of ​​a scanned image. Note that the configuration of the image processing system according to the third embodiment, and the hardware configuration and functional configuration of each device or service of the image processing system are the same as those of the image processing system 100 according to the first embodiment. Only the differences from the first embodiment will be described below.

[0099] Fig. 17 is a flowchart showing an example of the flow of the anonymization process in the image processing unit 432 according to the third embodiment, and is a flowchart showing an example of a detailed flow of the process in S509 in Fig. 5. The difference between the flowchart shown in Fig. 17 and the flowchart shown in Fig. 12 is that the process in S1602 is different from the process of generating an anonymized image in S1205. Specifically, the difference is that the process in S1205 generates a background image in which all pixels are white pixels, whereas the process in S1701 generates a background image using information on the background region in the scanned image.

[0100] Fig. 18 is a diagram showing an example of background image generation processing in the image processing unit 432 according to the third embodiment. Fig. 18(a) shows an example of a scanned image 1800 before concealment processing. Fig. 18(b) shows an example of a background image 1850 generated by the image processing unit 432 according to the third embodiment. Areas 1801, 1802, and 1803 in the scanned image 1800 shown in Fig. 18(a) correspond to characteristic background areas of the form corresponding to the scanned image 1800.

[0101] The image processing unit 432 extracts a background region from the scan image 1800 to generate a background image 1850. Specifically, for example, the image processing unit 432 may extract the background region by deleting the region separated as the foreground region in S602 from the scan image 1800. Furthermore, the image processing unit 432 sets the pixel value of the extracted background region in the scan image 1800 to the pixel value of the region of the background image 1850 corresponding to the background region. At this time, the pixel value of the region of the background image 1850 corresponding to the portion in the scan image 1800 where the foreground region exists is matched to the pixel value of the region surrounding the region, thereby complementing the portion hidden by the foreground region to generate the background image 1850. In this embodiment, the image processing unit 432 generates a concealed image by arranging character strings and non-character string objects in the background image 1850 shown as an example in FIG. 18(b).

[0102] As described above, in the third embodiment, MFP collaboration service 120 is configured to use background image 1850 generated by extracting the background region of scanned image 1800 to generate a concealed image. MFP collaboration service 120 configured in this way can leave information of the background of the form in the concealed image. Therefore, MFP collaboration service 120 according to the second embodiment can leave information in the concealed image that is more useful for a developer to analyze and improve the service.

[0103] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0104] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0105] <Configuration of the present disclosure> The present disclosure includes the following configurations, methods, and programs.

[0106] [Configuration 1] an acquisition means for acquiring one or more character strings and non-character string objects, which are objects other than character strings, included in a document image, the non-character string objects belonging to a first attribute and the non-character string objects belonging to a second attribute different from the first attribute; a first processing means for performing a first anonymization process on at least a part of the one or more character strings to generate a character string after the anonymization process; a second processing means for performing a second anonymization process on the non-character string object belonging to the first attribute to generate a non-character string object after the anonymization process; a generating means for generating an anonymized image corresponding to the document image by using the character string after the anonymization processing, the non-character string object after the anonymization processing, and the non-character string object belonging to the second attribute; 13. An image processing device comprising:

[0107] [Configuration 2] the non-character string objects belonging to the first attribute are non-character string objects that may include confidential information, and the non-character string objects belonging to the second attribute are non-character string objects that may not include confidential information; 2. The image processing device according to claim 1,

[0108] [Configuration 3] the second concealment process is an image process for the non-character string object belonging to the first attribute; 3. The image processing device according to configuration 1 or 2,

[0109] [Configuration 4] the second processing means generates a post-concealment non-character string object corresponding to the non-character string object by filling the entire non-character string object with a color based on a statistical value of values ​​of a plurality of pixels constituting the non-character string object belonging to the second attribute; 4. The image processing device according to any one of configurations 1 to 3,

[0110] [Configuration 5] the second processing means generates an image including information indicating a type of the non-character string object belonging to the second attribute as a post-anonymization non-character string object corresponding to the non-character string object; 5. The image processing device according to any one of configurations 1 to 4,

[0111] [Configuration 6] an estimation means for estimating a style of characters constituting a character string to be subjected to the first anonymization processing; Further comprising: the generating means generates the anonymized image by matching the character string after the anonymization processing to a style of the characters constituting the character string to be subjected to the first anonymization processing, based on information on a style of the characters; 6. The image processing device according to any one of configurations 1 to 5,

[0112] [Configuration 7] the information on the style of the character includes at least any one of information indicating a color of the character, information indicating a font type of the character, information indicating whether the character is bold, information indicating whether the character is italic, information indicating whether the character is underlined, and information indicating whether the character is strikethrough; 7. The image processing device according to configuration 6,

[0113] [Configuration 8] the generating means acquires a background image that is a base of the concealed image, and generates the concealed image by arranging the character string after the concealment processing, the non-character string object after the concealment processing, and the non-character string object belonging to the second attribute on the acquired background image; 8. The image processing device according to any one of configurations 1 to 7,

[0114] [Configuration 9] the generating means obtains the background image by generating the background image based on a background region of the document image; 9. The image processing device according to configuration 8,

[0115] [Configuration 10] the first processing means performs the first anonymization process on a character string that may include confidential information among the one or more character strings; 10. The image processing device according to any one of configurations 1 to 9,

[0116] [method] an acquiring step of acquiring one or more character strings and non-character string objects, which are objects other than character strings, included in a document image, the non-character string objects belonging to a first attribute and the non-character string objects belonging to a second attribute different from the first attribute; a first processing step of performing a first anonymization process on at least a part of the one or more character strings to generate a character string after the anonymization process; a second processing step of performing a second anonymization process on the non-character string object belonging to the first attribute to generate a non-character string object after the anonymization process; a generating step of generating an anonymized image corresponding to the document image by using the character string after the anonymization process, the non-character string object after the anonymization process, and the non-character string object belonging to the second attribute; 13. An image processing method comprising:

[0117] [program] A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0118] 120 MFP Collaboration Service 300 Image Processing Server 432 Image Processing Unit

Claims

1. an acquisition means for acquiring one or more character strings and non-character string objects, which are objects other than character strings, included in a document image, the non-character string objects belonging to a first attribute and the non-character string objects belonging to a second attribute different from the first attribute; a first processing means for performing a first anonymization process on at least a part of the one or more character strings to generate a character string after the anonymization process; a second processing means for performing a second anonymization process on the non-character string object belonging to the first attribute to generate a non-character string object after the anonymization process; a generating means for generating an anonymized image corresponding to the document image by using the character string after the anonymization process, the non-character string object after the anonymization process, and the non-character string object belonging to the second attribute; 13. An image processing device comprising:

2. the non-character string object belonging to the first attribute is the non-character string object that may include confidential information, and the non-character string object belonging to the second attribute is the non-character string object that may not include confidential information; 2. The image processing device according to claim 1,

3. the second concealment process is an image process for the non-character string object belonging to the first attribute; 2. The image processing device according to claim 1,

4. the second processing means generates a post-concealment non-character string object corresponding to the non-character string object by filling the entire non-character string object with a color based on a statistical value of values ​​of a plurality of pixels constituting the non-character string object belonging to the second attribute; 2. The image processing device according to claim 1,

5. the second processing means generates an image including information indicating a type of the non-character string object belonging to the second attribute as a post-anonymization non-character string object corresponding to the non-character string object; 2. The image processing device according to claim 1,

6. an estimation means for estimating a style of characters constituting a character string to be subjected to the first anonymization processing; Further comprising: the generating means generates the anonymized image by matching the character string after the anonymization processing to a style of the characters constituting the character string to be subjected to the first anonymization processing, based on information on a style of the characters; 2. The image processing device according to claim 1,

7. the information on the style of the character includes at least any one of information indicating a color of the character, information indicating a font type of the character, information indicating whether the character is bold, information indicating whether the character is italic, information indicating whether the character is underlined, and information indicating whether the character is strikethrough; 7. The image processing device according to claim 6,

8. the generating means acquires a background image that is a base of the concealed image, and generates the concealed image by arranging, on the acquired background image, the character string after the concealment processing, the non-character string object after the concealment processing, and the non-character string object that belongs to the second attribute; 2. The image processing device according to claim 1,

9. the generating means obtains the background image by generating the background image based on a background region of the document image; The image processing device according to claim 8 .

10. the first processing means performs the first anonymization process on a character string that may include confidential information among the one or more character strings; 2. The image processing device according to claim 1,

11. an acquiring step of acquiring one or more character strings and non-character string objects, which are objects other than character strings, included in a document image, the non-character string objects belonging to a first attribute and the non-character string objects belonging to a second attribute different from the first attribute; a first processing step of performing a first anonymization process on at least a part of the one or more character strings to generate a character string after the anonymization process; a second processing step of performing a second anonymization process on the non-character string object belonging to the first attribute to generate a non-character string object after the anonymization process; a generating step of generating a concealed image corresponding to the document image by using the character string after the anonymization process, the non-character string object after the anonymization process, and the non-character string object belonging to the second attribute; 13. An image processing method comprising:

12. A program for causing a computer to function as the image processing device according to any one of claims 1 to 10.

Images