Information processing device, information processing method, and program

Abstract

This reduces the burden on users when marking areas and simplifies the process of linking marked areas to other settings. [Solution] The information processing device includes a detection means for detecting a marked area from a document image obtained by reading a marked document, a display control means for displaying a first setting screen for setting a process to be associated with the detected area based on user specifications, and a processing means for executing the process set via the first setting screen on the detected area.

Description

Technical Field

[0006] , ,

[0005] , , ,

[0007] , ,

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is a technique of extracting a character string from a specific area marked by a user, such as filling it with a marker pen, from the entire document image obtained by scanning a document or the like, and using it for the file name of the document image or the like.

[0003] Patent Document 1 discloses a technique of associating a specific process with each color that can be used for marking, and executing a specific process on a specific area marked in a document image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique of Patent Document 1, it is necessary to store information indicating a process associated with each color that can be used for marking, which is defined in advance, and when marking, it is necessary to carefully select the color of the marker pen according to the stored information. In particular, when the number of colors that can be used for marking increases, the number of combinations of colors and processes increases, which may increase the burden on the user when memorizing and differentiating them.

[0006] The present disclosure provides a technique that reduces the burden on the user when marking and enables easy setting of associations with marked areas.

Means for Solving the Problems

[0007] An information processing device according to one aspect of the technology of this disclosure is characterized by comprising: detection means for detecting a marked area from a document image obtained by reading a marked document; display control means for displaying a first setting screen for setting a process to be associated with the detected area based on user specifications; and processing means for executing the process set via the first setting screen on the detected area. [Effects of the Invention]

[0008] The technology disclosed herein reduces the burden on the user when marking and makes it easy to set up associations with marked areas. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows an example of the system configuration. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This diagram shows an example of the software configuration for each device that makes up the system. [Figure 4] This is a diagram illustrating a pre-printed form. [Figure 5] This is a diagram for setting the markings and the processes to be executed. [Figure 6] This is a flowchart showing the document processing flow for the comparative example. [Figure 7] This is a diagram showing an example of the display screen. [Figure 8] This figure shows an example of a settings screen for document processing. [Figure 9] This figure shows an example of a report settings screen that includes processes that cannot be executed. [Figure 10] This is a flowchart showing the document processing flow. [Figure 11] This is a flowchart showing the flow of report processing using learning information. [Figure 12] This figure shows an example of a settings screen for report processing using learning information. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the technology of this disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the technology of this disclosure as defined in the claims. Not all combinations of features described in the embodiments are essential as solutions of the technology of this disclosure, and multiple features may be combined arbitrarily. The same components will be denoted by the same reference numerals. Furthermore, each step in the flowchart will be indicated by the letter "S" at the beginning.

[0011] <<Embodiment 1>> (System Configuration) Figure 1 shows an example of the configuration of an image processing system according to this embodiment. The image processing system of this embodiment includes an MFP (Multi Function Peripheral) terminal 111, a scanned document processing server 121, an external server 131, and a client terminal 141. These devices are interconnected via a network 101 such as the Internet or an intranet.

[0012] The MFP terminal 111 is a multifunction device that has multiple functions, such as a scanner and a printer, and performs image formation and various processing tasks.

[0013] The scan document processing server 121 receives the scanned image data of the document obtained by the MFP terminal 111 together with a processing request, and performs the requested image processing on the scanned image data of the document. Then, the scan document processing server 121 has a function of cooperating with the MFP terminal 111 and the external server 131, such as transmitting the scanned image data of the document after the image processing to a file storage destination including the external server 131, etc. The scan document processing server 121 is a server device that provides services having various functions, such as a function of managing setting information according to, for example, an external server, an MFP terminal, or user information to be cooperated, and a function of performing image processing on the scanned image data of the document. Note that these functions may be separated by function into a plurality of different servers and configured to cooperate with the plurality of servers.

[0014] The external server 131 is a server device that provides, for example, a cloud storage service for storing the scanned image data of the document. Note that the external server 131 is not limited to this, and may also be a server device that provides other external services that execute a business workflow such as registration, update, and execution of business data based on information extracted from the scanned image data of the document. Further, although the external server 131 is described assuming that it is provided as a web application, it may be in an on-premises form that is executed in a server environment within the company intranet.

[0015] The client terminal 141 is a device such as a computer, a tablet, or a smartphone that can display the web UI provided by the scan document processing server 121 and the external server 131 in an application such as a browser installed in the client terminal.

[0016] (Hardware Configuration of Information Processing Apparatus) FIG. 2 is a block diagram showing the hardware configurations of the MFP terminal 111, the scan document processing server 121, the external server 131, and the client terminal 141 as information processing apparatuses.

[0017] The information processing device includes a secondary storage device 201, RAM 202, ROM 203, CPU 204, a network interface (hereinafter referred to as network I / F) 205, a user interface (hereinafter referred to as user I / F) 206, and an operating device 207. The MFP terminal 111 of the information processing device further includes a scanner interface (hereinafter referred to as scanner I / F) 208, a scanner 209, a printer interface (hereinafter referred to as printer I / F) 210, and a printer 211. The information processing device has a system bus 212 to which each device connects, and is configured to exchange data with each other via the system bus 212.

[0018] Secondary storage device 201 is a secondary storage device such as an HDD or flash memory. RAM 202 has a temporary memory area. ROM 203 stores embedded programs and data. CPU 204 loads programs read from secondary storage device 201, RAM 202, ROM 203, etc., into RAM 202 and executes them.

[0019] The Network I / F205 connects to a network such as a LAN to communicate with other computers, MFPs, printers, network devices, and external services. The communication method can be either wired or wireless.

[0020] The user interface 206 controls input and output from operating devices 207, such as displays, keyboards, mice, buttons, and touch panels. Computers that lack this hardware can be operated by connecting from another computer via remote desktop or remote shell.

[0021] Scanner I / F 208 controls input and output to and from scanner 209. Printer I / F 210 controls input and output to and from printer 211.

[0022] (Software configuration) Figure 3 shows an example of the software configuration of each device constituting the image processing system according to this embodiment. The CPU 208 executes the software installed on the MFP terminal 111, the scanned document processing server 121, the external server 131, and the client terminal 141, thereby realizing predetermined functions. As indicated by the network connection arrows, the devices are configured to communicate with each other.

[0023] (Software configuration of MFP terminal 111) The MFP terminal 111 can be broadly divided into two parts: the basic function unit 311 and the MFP application 321. The basic function unit 311 consists of functions that are standardly provided in the MFP terminal 111. The MFP application 321 is an application that is additionally installed on the MFP terminal 111. The MFP application 321 is a Java®-based application, which makes it easy to add functions to the MFP terminal 111. In this embodiment, for the sake of simplicity, it is assumed that the login application 322 and the marker application 323 are installed as the MFP application 321, but it is not limited to this. For example, these functions may be bundled into a single application. Alternatively, these functions may be installed separately and coordinated, resulting in a configuration consisting of multiple applications.

[0024] The basic function unit 311 includes an external system communication unit 312, a display control unit 313, an image data processing unit 314, and a scan execution unit 315.

[0025] The external system communication unit 312 performs various processing requests to the scanned document processing server 121 and analyzes the responses of those processing results. For example, it makes processing requests to the relaying scanned document processing server 121, such as login requests, processing requests for scanned images, and requests to retrieve information held by the external server 131 and to execute processing. Communication with the scanned document processing server 121 uses protocols such as REST and SOAP, but other communication methods may also be used.

[0026] The display control unit 313 controls the display and input / output of UI screens for accepting user operations via the user interface 206 of the MFP terminal 111 to the operating device 207. For example, it displays UI screens such as input for authentication information to log in to the device, scan settings and execution operations, and a preview screen, and accepts input for these.

[0027] The image data processing unit 314 performs image processing on the scanned image data received from the scan execution unit 315, and saves the processed scanned image data to the secondary storage device 201. The image data processing unit 314 also retrieves the saved processed scanned image data from the secondary storage device 201 upon request.

[0028] When the scan execution unit 315 receives a scan request containing scan setting information, it reads the document placed on the document glass or feeder using the scanner 209 via the scanner I / F 208 according to the request and generates scanned image data of the document. The generated scanned image data of the document is sent to the image data processing unit 314. The scanned image data of the document is assumed to be an image file such as JPEG or a document file such as PDF. The scan execution unit 315 returns a scanned image identifier that uniquely identifies the saved scanned image data of the document to the scan requester. The scanned image identifier is a number, symbol, alphabet, etc. (not shown) used to uniquely identify the document image obtained by scanning the document at the MFP terminal 111.

[0029] The login application 322 links authentication information from both the MFP terminal 111 and the scanned document processing server 121, and allows login using any method such as ID, username, password, PIN code, or card touch. Upon successful login, it becomes possible to operate various functions of the MFP terminal 111 and retrieve necessary information such as user information, application settings, and scan settings managed by the scanned document processing server 121.

[0030] Marker app 323 is an application that, via the display control unit 313, presents the user with a scan screen or a screen for setting scan parameters, and executes a scan using the scan execution unit 315 of the basic function unit 311 based on the configured settings. Marker app 323 obtains scan image data corresponding to the scan image identifier received from the scan execution unit 315 from the secondary storage device 201 using the image data processing unit 314. Marker app 323 presents the user with a screen via the display control unit 313 for setting the processing content to be performed on the scan image data. Marker app 323 requests the scan document processing server 121 via the external system communication unit 312 to obtain the acquired scan image data, the processing content to be performed on the scan image data, and information held by the external server 131 necessary for each display. The user information used in this request is based on the user information logged in with login app 322. Marker app 323 analyzes the response and displays the scan screen, settings screen, and processing result screen.

[0031] (Software configuration of scanned document processing server 121) The scanned document processing application 331 is a web application provided by the scanned document processing server 121, and performs image processing on scanned image data of documents obtained by the MFP terminal 111 and transmits the scanned image data of documents to an external server 131. In this embodiment, it is described as operating as a web application, but other application implementation forms are also possible. The scanned document processing application 331 has an API (Application Programming Interface) 332 and a WebUI 333.

[0032] API332 is an application programming interface provided by the scanned document processing application 331. WebUI333 is a user interface provided by the scanned document processing application 331 for configuring the application and server from a web browser.

[0033] The data store 341 is a storage area for saving and storing data used by the scanned document processing application 331 or the backend application 351 described later. The data store 341 stores various data from the scanned document storage unit 342, the scanned document job queue 343, the configuration information management unit 344, and the job information management unit 345.

[0034] The scanned document storage unit 342 stores the scanned image data received from the MFP terminal 111 via the external system communication unit 354 (described later) on the scanned document processing server 121.

[0035] The scan document job queue 343 holds a queue for managing jobs awaiting processing. A job is an execution unit for processing a scanned document requested from the MFP terminal 111. For simplicity, a single queue is used in this explanation, but multiple queues may be provided depending on the job scheduling priority and processing content.

[0036] The configuration information management unit 344 manages the configuration information used during scanning. It also manages user information necessary for the cooperation between the MFP terminal 111 and the scanned document processing server 121, and between the scanned document processing server 121 and the external server 131, linking the MFP terminal, external server, and user to each other. Processing information corresponding to the marking described later is also managed in conjunction with user information and document type. In addition, it manages the settings for the entire scanned document processing server. For the sake of simplicity, the management of various settings is shown as a single unit in the configuration information management unit 344, but it is also possible to configure it to separate the management of different types of information.

[0037] The Job Information Management Unit 345 manages detailed job information for each job, such as processing status, registration date, and configuration information when the job was instructed to run.

[0038] The backend application 351 is responsible for background processing and consists of a report processing unit 352, a job control unit (status management unit) 353, and an external system communication unit 354.

[0039] The document processing unit 352 performs document processing on scanned or captured image data. In this embodiment, it extracts the marked area of ​​a document marked with a marker pen of a certain color, such as a highlighter, for confirmation or instruction, and performs processing on the extracted area. Examples of processing on the extracted area include character recognition processing by the OCR unit 355, masking processing by the masking unit 356, signature processing by the signature unit 357, and stamping processing by the stamp unit 358. In this embodiment, the document processing will be described as being performed by each processing component of the document processing unit 352. Alternatively, the basic functions of the image data processing unit 314 of the MFP terminal 111 or functions added to the MFP application 321 may be used. Alternatively, an image processing system or server that works in conjunction with the scanned document processing server 121 may be prepared separately and used in conjunction with it.

[0040] Other general document processing functions include the document processing unit 352 acquiring input images from the scanned document storage unit 342 and performing OCR, compression, splitting, and conversion on the entire document. In addition, it performs document discrimination processing such as pattern recognition and machine learning, page extraction, rotation, deletion, and insertion of specific pages.

[0041] The job control unit 353 checks the status of jobs or the processing status of the scanned document processing server 121 and the external server (business server) 131 in the backend application 351, and executes processing according to each status.

[0042] The external system communication unit 354 performs processing related to the transmission and reception of scanned image data and various processing requests and response results sent from the MFP terminal 111, as well as the transmission and reception of various processing requests and response results to the external server 131. While protocols such as REST and SOAP are used for communication with the MFP application 321 and the external server 131, other communication methods may also be used. Examples of various processing requests made to the external server 131 include authentication and processing requests for scanned image data executed via the scanned document processing server 121 from the MFP terminal 111, and the acquisition of various information managed by the external server 131.

[0043] On the external server 131, the external service application 361, which is a web application for an external service, is executed. The external service application 361 includes an API 362, a Web UI 363, a data storage unit 364, a data processing unit 365, and an external system communication unit 366.

[0044] API 362 is provided by the external service application 361. WebUI 363 is provided by the external service application 361. Data storage unit 364 stores data used by the external service application 361.

[0045] The data processing unit 365 executes the processing requested by the MFP terminal 111 and the scanned document processing server 121. The external system communication unit 366 analyzes the requests received from the MFP terminal 111 and the scanned document processing server 121 and performs processing related to sending and receiving, such as sending the processing results as a response.

[0046] For the sake of simplicity, the form and configuration of the external server 131 have been consolidated into a single application, but this is not the only way to do so. Similar to the scanned document processing server 121, it may be provided in a form and configuration in which the system and applications are separated according to their respective roles, such as backend applications and data stores.

[0047] One or more external servers 131 that cooperate with the scan document processing server are assumed to transmit, process, store, or execute the processing results of the MFP terminal 111 and the scan document processing server 121.

[0048] On client terminal 141, client application 371 is executed. Client application 371 is a browser or similar application that runs on client terminal 141. It displays the WebUI provided by the scanned document processing application 331 and the external service application 361, allowing users to view and edit their respective settings. These applications may also be provided as desktop applications, such as executable files, provided by each service.

[0049] The problems arising in the comparative example will be explained using Figures 4 to 6.

[0050] Figure 4 shows an example of a form that has been marked using a marker pen such as a highlighter. A form 400 that has been marked by the user has areas 401, 402, and 403 marked with a marker pen. Area 401 is marked in red 411. Area 402 is marked in blue 412. Area 403 is marked in green 413. Examples of marking methods include filling in areas 421, bordering areas 422, and underlining areas 423. Filling in areas 421 indicates an area where the entire area is filled in with a marker. Bordering areas 422 indicates a border around a designated area drawn with a marker. Underlining areas 423 indicates an area where the bottom of a designated area is filled in with a marker. Note that the marking methods are not limited to these, and any analysis means may be combined with marking, for example, if the text areas within the form are determined by form analysis, the system may specify that the nearest text area be read using check marks or black dots.

[0051] (Document processing settings screen) Figure 5 shows an example of a UI screen for setting the processing associated with each color and the method of specifying the marking in form processing. In Figure 5, "Purchase Order" is entered in input area 501, OCR (used for file name) is selected in the red processing setting field 502, and fill is selected in the specification method field 504.

[0052] The document processing settings screen 500 is a UI screen for configuring the processing associated with marker colors. When applied to the configuration in Figure 3, the settings screen 500 is displayed on the client application 371 by the WebUI 333 provided by the scanned document processing application 331. The settings screen 500 allows editing of the marking processing content associated with users and documents stored in the data store 341. The settings screen 500 has an input area 501 for the setting name, a processing setting field 502, and a specification method field 504. The settings screen 500 also has a save button 506 and a back button 507.

[0053] The input field 501 for setting names is an area that accepts input for setting names to manage setting information. It is desirable to manage this by using a form type name or similar so that the form type, setting, and processing details are clear.

[0054] The processing settings field 502 is used to configure which processing is associated with the marked color. In the processing settings field 502, each processing can be selected from a pull-down menu 503. For example, one of the following options can be selected: OCR (used for file names), OCR (used for folder names), OCR (used for metadata), masking, signing, or stamping.

[0055] The Specification Method field 504 is used to specify the method of marking with the marker. Similar to the Processing Settings field 502, each specification method can be selected from a pull-down menu 505. For example, one of the following options can be selected: fill, frame, or underline.

[0056] When the save button 506 is pressed by the user, it saves the settings specified on the settings screen 500. In the example in Figure 5, the settings for OCR for red (used in the file name) and the setting for fill as the specification method will be saved as processing settings. When the back button 507 is pressed by the user, the settings on the settings screen 500 are canceled and the display switches to the screen that was displayed before the settings screen 500.

[0057] (Document processing related to comparative examples) Figure 6 is a flowchart showing the document processing flow performed by the system in the comparative example. In Figure 6, the processing flow is shown separately for the user, MFP terminal, scanned document processing server, and external server actors. To explain the issues in the comparative example's processing, processes that are not necessary for explaining the issues, such as login, are not shown, and details will be explained later.

[0058] In S601, the user first marks a task with a color, as shown in Figure 5, and then configures which process will be executed when that task is used, by linking each color to the corresponding process on a settings screen displayed on the client application 371.

[0059] In S602, the scanned document processing server 121 stores the user-defined colors and corresponding processing details in the configuration information management unit 344. The process from S601 to S602 completes the registration of the pre-configured information to the scanned document processing server.

[0060] In S603, the user marks the forms that are to be scanned and processed using the specified method with a marker pen of the corresponding color, based on the settings made in S601 and S602.

[0061] In S604, the user places the completed form into the MFP terminal 111 and instructs it to perform a scan.

[0062] In S605, the MFP terminal 111 performs a scan on the set document, which is a form, and obtains scanned image data of the form (document). The obtained scanned image data of the form is uploaded to the scanned document processing server 121 via the external system communication units 312 and 354.

[0063] In S606, the scanned document processing server 121 stores the scanned image data of the uploaded form in the scanned document storage unit 342, stacks it in the scanned document job queue 343, and detects the marked area by the form processing unit 352.

[0064] The detection of these marked areas is generally performed by defining thresholds for each parameter of a color space such as RGB, HSV, or HSL, and identifying areas containing color information that exist within that range. It is assumed that the colors to be detected are pre-defined, such as blue, red, green, and yellow. Furthermore, the thresholds for each color are also pre-defined.

[0065] In S607, the form processing unit 352 obtains the corresponding processing from the configuration information management unit 344 and executes the processing associated with each color.

[0066] For example, if OCR (used for filenames) is specified, the OCR functional component receives information about the original image and the relevant area, extracts and stores the string from the relevant area, and uses it as a parameter in the request when saving to the external server 131. If OCR (used for folder names) or OCR (used for metadata) is specified, it is the same as OCR (used for filenames), and a detailed explanation is omitted. If masking is specified, the masking functional component receives information about the original image and the relevant area, and the relevant area is masked. If signature is specified, the signature functional component receives information about the name to be assigned to the original image and the relevant area, and assigns a name to the relevant area, which is used as a parameter in the request when saving to the external server 131. If stamp is specified, the stamp functional component receives information about the string to be assigned to the original image and the relevant area, and assigns a string to the relevant area, which is used as a parameter in the request when saving to the external server 131.

[0067] In S608, the scanned document processing server 121 makes a processing request to the external server 131 once the document processing specified by the document processing unit 352 is complete. For example, if the external server 131 is cloud storage, a request is made to save the scanned documents.

[0068] In S609, the external server 131 executes the requested processing via the external system communication units 354 and 366.

[0069] The above describes a series of processes that are performed on forms marked by a user with a marker pen, according to the content of the markings.

[0070] However, the method described using Figures 4 to 6 requires storing predefined information indicating the associated process for each color that can be used for marking, and then, when marking, the user has to manually select the appropriate marker pen color according to the stored information. In particular, as the number of colors that can be used for marking increases, the number of color-process combinations also increases, potentially increasing the burden on the user in terms of remembering and selecting the correct combinations.

[0071] Therefore, this embodiment describes a preview display method that allows users to arbitrarily set the processing to be performed on the marked area of ​​a marked document after reading it, without having to be aware of the colors or processing used for marking on the device. In addition, a method for simplifying settings in the future by learning the set processing is also described. This embodiment will be explained using Figures 7 to 12.

[0072] (display screen) Figure 7 illustrates a series of screen examples from (a) to (d) up to scanning a printed form with the MFP terminal 111. For simplicity, it is assumed that user input and the output of the screens presented to the user by the MFP terminal 111 are operations performed by an operating device 207, such as a touch panel, which is controlled by the display control unit 313 via the user interface.

[0073] (Login screen) Figure 7(a) shows the login screen presented to the user when the login application 322 is launched. The login screen 700 is displayed on the MFP terminal 111 by the display control unit 313 of the MFP terminal 111. The login screen 700 has a username input field 701, a password input field 702, and a Login button 703.

[0074] The username input field 701 is an input field or control presented to the user necessary for login. The password input field 702 is the input field for the password required for login. The Login button 703 is a button that accepts user input when logging in using the username entered in the username input field 701 and the password entered in the password input field 702. The login application 322 executes the login based on the information entered by the user and the operation of the controls.

[0075] The login screen 700 illustrates a typical login form using a username and password, but it is not limited to this. The fields displayed on the login screen 700 can be changed or switched to an ID, PIN code, or a control prompting card touch, and login can be performed by accepting the corresponding operation, or by any other means.

[0076] (Menu screen) Figure 7(b) shows an example of a UI screen after login, where the logged-in user's information is displayed. UI screen 710 has a display area 711 for the logged-in user. UI screen 710 is a menu screen and has an icon 712 for the marker app, an icon 713 for copying, an icon 714 for scanning, an icon 715 for the inbox, an icon 716 for the address book, and an icon 717 for settings.

[0077] The login user display area 711 also serves as the logout control. When logged out, the login session ends, and the display returns to that shown in Figure 7(a). Upon successful login, a list of applications available to the logged-in user is displayed. In Figure 7(b), icons for the marker application, copy, scan, inbox, address book, and settings are displayed. The marker application icon 712 displays a control for launching the marker application, which configures the processing of the corresponding area of ​​the marked document according to this embodiment. The marker application is launched when the user presses the marker application icon 712.

[0078] (UI screen of the marker app) Figure 7(c) shows an example of the UI screen of the marker app. UI screen 720 is the screen that appears when the marker app icon 712 is pressed and the marker app is launched. UI screen 720 displays six shortcut buttons 721 to 726 on a single screen.

[0079] Shortcut button 721 is a shortcut button that has been pre-configured with scan settings and destinations registered on the MFP terminal 111 or the scanned document processing server 121. As will be described later, the color and document processing settings are also linked to each button setting. The settings are stored in the setting information management unit 344 of the scanned document processing server 121. The user can select the desired shortcut button from among several shortcut buttons to display the scan screen corresponding to the selected shortcut button.

[0080] (Scan screen) Figure 7(d) shows the UI screen 730 for displaying the scan screen. The UI screen 730 has a scan preview area 731, a basic scan settings field (hereinafter also referred to as scan settings) 732, a scan button 733, and a send button 734.

[0081] The scan preview area 731 is an area for displaying a preview of a document scanned according to user instructions.

[0082] The basic scan settings field 732 is an area that displays editable information about the destination and scan settings associated with the shortcut button 721 selected by the user. This is done via a screen (not shown) for changing each setting, destination, or save format such as folders.

[0083] The scan button 733 is pressed by the user after the document has been placed on the platen or feeder to initiate a scan. The send button 734 is pressed by the user after the scan is complete and the scan preview area 731 and scan settings 732 have been confirmed, sending the scanned image of the document obtained from the scan to the external server 131 via the scanned document processing server 121.

[0084] (Settings for processing reports) Figure 8 shows an example of a UI screen for setting up document processing associated with each marked area extracted from a scanned document in this embodiment.

[0085] (Settings screen for marker selection method) Figure 8(a) shows a UI screen for specifying the marking method performed by the user. The UI screen 800 has a field 801 for setting the marker specification method.

[0086] Field 801 is a control for setting the marking method. Field 801 accepts user input to set the marking method actually applied to the form from a list of marking methods displayed in the control. Input is possible using pull-down menu 802. In pull-down menu 802, the user can select one of the following options: fill 803, border 804, underline 805, or other 806. Fill 803, border 804, and underline 805 each display the name of the specification method and a schematic diagram of the specification method. Note that the display of specification methods in the pull-down menu is not limited to these. For example, only the name of the specification method may be displayed, or only a schematic diagram of the specification method may be displayed. An example of selecting the desired specification method using a pull-down menu has been shown, but it is not limited to this. Other display methods such as a list display or radio buttons may also be used.

[0087] (Screen for setting the association for specific processes) Figure 8(b) shows an example of a screen for setting the association of specific processes. On the screen for setting the association of specific processes, settings are made to associate specific processes with each marked area. The UI screen 810 has a preview field 811, marker area color specification fields 812-814, report processing setting controls 815-817, an OK button 818, and a back button 819.

[0088] The preview field 811 is a field that displays a preview of the document image obtained by scanning the document. The user can specify the corresponding form processing for each color while viewing the document image displayed in the preview field 811. In Figure 8(b), the document image 820 obtained by scanning the printed document is displayed. The document image 820 is the same as the form image obtained by scanning the form 400 described above, and has colored areas 821, 822, and 823.

[0089] The color specification fields 812, 813, and 814 of the marker area are controls that indicate the actual detected marked color. As mentioned above, the detection of the marked area is performed by setting thresholds for each parameter in color spaces such as RGB, HSV, and HSL, and identifying the area containing color information that exists within that range.

[0090] The report processing setting controls 815-817 are controls used to configure report processing for each process.

[0091] (Pull-down menu screen) Figure 8(c) shows an example of the pull-down menu screen for document processing displayed in the document processing setting controls 815-817. The pull-down menu screen is displayed by obtaining information about the functional components of the document processing unit 352 of the scanned document processing server 121, respectively, via the document processing setting controls 815-817. In Figure 8(c), OCR (used for file names) is shown as specified by the user.

[0092] The pull-down menu screen 830 displays the option to select one of the following processes: OCR (for file names) 831, OCR (for folder names) 832, OCR (for metadata) 833, masking 834, signature 835, stamp 836, and other 837.

[0093] Note that the pull-down menu screen is not limited to the example shown in Figure 8(c). As mentioned above, the MFP terminal 111 may obtain an image processing list from its own image processing functions or from an external image processing system and display that list. For example, annotations may be specified. If annotations are specified, the original image and the annotation information to be applied to the relevant area are passed to the annotation's functional component to apply the annotation to the relevant area, and this information is used as a parameter in the request when saving to the external server 131.

[0094] When the OK button 818 is pressed by the user, it saves the settings specified on the UI screen 810. When the Back button 819 is pressed by the user, it cancels the settings on the UI screen 810 and switches to the display screen that was shown before the UI screen 810.

[0095] (Document processing settings control) Figure 9 shows another example of the form processing setting control 815.

[0096] The report processing settings control 815 is designed to hide report processing that cannot be executed, but if, for example, a report processing that is normally executed cannot be executed when using a different destination or device than usual, it is difficult for the user to identify the cause if the processing is hidden. In such cases, a pull-down menu screen 900 is displayed.

[0097] The pull-down menu screen 900 displays the following document processing options: OCR (used for file names) 831, OCR (used for folder names) 832, masking 834, stamping 836, and other 837, and is displayed in an activated state. This allows the user to specify one of these options: OCR (used for folder names) 832, masking 834, stamping 836, or other 837.

[0098] The pull-down menu screen 900 displays document processing that cannot be executed on the MFP terminal 111, the scanned document processing server 121, and the external server 131 in an inactive state, making it impossible for the user to select. In Figure 9, document processing that cannot be performed, such as OCR (used for metadata) 901 and signature 904, is displayed in an inactive state and is impossible for the user to select. The pull-down menu screen 900 displays help controls 902 and 905. When user operations such as hovering or clicking are performed on help controls 902 and 905, controls 903 and 906 respectively provide hints to the user as to why that processing is unavailable. Controls 903 and 906 are intended to be tooltips, but they may also be displayed as dialogs or banners. Examples of reasons why a function may not be available include those dependent on the destination, such as metadata, folder structure, prohibited characters, and connection information. Furthermore, reasons may include those dependent on device functions, or licenses / permissions associated with the device or user, or reasons why it cannot be executed depending on the marking method or color.

[0099] (UI screen display control processing) Figure 10 is a flowchart showing the flow of the display control process for the UI screen according to this embodiment. The screens described using Figures 7 to 9 are used in the process shown in Figure 10.

[0100] In S1001, the user marks the scanned forms (documents) with a highlighter or other marker pen. The user distinguishes the process associated with the marking by the color of the marker and performs the marking work with the marker pen.

[0101] In S1002, the user initiates device operation, and the login application 322 of the MFP terminal 111 displays a login screen to the user. For example, the login screen 700 shown in Figure 7(a) is displayed.

[0102] In S1003, the user enters login information into the login screen displayed on the operating device 207 of the MFP terminal 111. Specifically, the user enters their user ID and password into the designated input fields on the login screen and then presses the Login button.

[0103] In S1004, based on the login information entered by the user, the login application 322 of the MFP terminal 111 sends an authentication request to the external system communication unit 354 of the scanned document processing server 121 via the external system communication unit 312.

[0104] In S1005, the scanned document processing server 121 analyzes the request and verifies whether the login information sent in S1004 matches the information of the corresponding user managed by the configuration information management unit 344.

[0105] In S1006, if the verification is successful and there are no problems, a success response is returned to the MFP terminal 111, and the MFP terminal 111 presents the post-login screen to the user. For example, the UI screen 710 shown in Figure 7(b) is displayed as the post-login screen.

[0106] In S1007, the user selects the marker app icon 712 from the list of available functions on the login screen displayed on the MFP terminal 111 to launch the marker app.

[0107] In S1008, when the marker app icon 712 is selected in S1007, the MFP terminal 111 launches the marker app 323.

[0108] In S1009, when the marker application 323 is launched, the scanned document processing server 121 retrieves the scan settings information associated with the user stored in the settings information management unit 344 of the scanned document processing server 121, based on the information of the logged-in user.

[0109] In the S1010, the MFP terminal 111 displays a list of acquired configuration information as shortcut buttons on the marker application 323. For example, the UI screen 720 shown in Figure 7(c) is displayed as the UI screen of the marker application.

[0110] In S1011, the user selects a shortcut button depending on the form they are using and the content they are processing. For example, shortcut button 721 is selected on UI screen 720.

[0111] In S1012, the MFP terminal 111 displays a scan screen that reflects the destination and scan settings according to the selected shortcut button. For example, the UI screen 730 shown in Figure 7(d) is displayed as the scan screen.

[0112] In the S1013, the user places the printed form on the document glass or feeder, changes the scan settings as desired, and then instructs the system to perform the scan.

[0113] In S1014, the scan execution unit 315 of the MFP terminal 111 performs a scan on the set form based on the settings made on the scan screen.

[0114] In S1015, the user sets the method of marking the scanned documents. For example, the user sets the marking method via the UI screen 800 shown in Figure 8(a). Note that the marking method is set after the scanning process, but is not limited to this. The marking method may be set before the scanning process, as long as it is before the upload described later.

[0115] In S1016, the MFP terminal 111 uploads the scanned image data of the document obtained by performing the configured marking method and scanning process to the scanned document processing server 121.

[0116] In S1017, the scanned document processing server 121 stores the scanned image data of the uploaded document in the scanned document storage unit 342, stacks it in the scanned document job queue 343, and the document processing unit 352 detects the marker area.

[0117] In S1018, the MFP terminal 111 returns processing result information as a response to the marker application 323, including the color information, coordinates, string information within the detected area, and a list of document processing functions supported by the document processing unit 352. Based on the results, the marker application 323 displays a screen for setting the document processing for each detected color area, along with the area information and scanned image data detected by the device. For example, the UI screen 810 shown in Figure 8(b) is displayed as the screen for setting the processing.

[0118] In S1019, the user specifies one of several options for the report processing to be performed for each color area.

[0119] In S1020, the marker application 323 of the MFP terminal 111 receives user input and requests the scanned document processing server 121 to process the document for the specified color area.

[0120] In S1021, the scanned document processing server 121 executes processing using the functional components of the form processing unit 352 according to the relevant area of ​​the requested document and the specified processing.

[0121] In S1022, once the scanned document processing server 121 has completed the report processing for the scanned document, it generates request parameters based on the scanned image data after report processing and information such as the string obtained as a result of the report processing. The scanned document processing server 121 then requests processing from the external server 131, which is the final storage destination. The external server 131 is assumed to be cloud storage, so it executes the save request. The external server 131 may also work in conjunction with different business workflows and execute requests for processing such as workflows.

[0122] In S1023, the data processing unit 365 of the external server 131 executes processing based on the request content. Upon completion, it returns the processing result to the scanned document processing server 121.

[0123] In S1024, the scanned document processing server 121 notifies the MFP terminal 111 of the processing results received from the external server 131 via the marker application 323.

[0124] In S1025, the marker application 323 of the MFP terminal 111 displays a completion screen to indicate to the user that the process is complete. The completion screen (not shown) may be a dedicated screen, or it may be displayed by any means or method, such as a notification via a dialog or banner, notification via email, or transitioning to the home screen UI screen 720 shown in Figure 7(c). It may also transition to the UI screen 710 shown in Figure 7(b).

[0125] The above describes a preview display method that detects marked areas on a form and allows users to specify the processing to be performed on those areas from their device. This reduces the burden on users when marking and makes it easy to configure settings to link to marked areas.

[0126] (UI screen display control processing) Figure 11 is a flowchart illustrating a modified version of the UI screen display control process, which adds learning to the process shown in Figure 10. Figure 11 shows a method to simplify the preview operation performed by the user on the device by learning the user's marking method, the marked areas, and the report processing associated with each area. Specifically, as an effect of learning, if learning information exists, the settings performed in Figure 8, S1015, and S1018 will already be reflected or skipped, further simplifying the setting operation. The differences from Figure 10 will be explained in detail.

[0127] In S1101, when a user selects a form button, the marker application 323 on the MFP terminal 111 queries the scanned document processing server 121 to see if there is any learning information associated with that form (shortcut button).

[0128] In S1102, the scanned document processing server 121 analyzes the request and checks whether learning information associated with the user who made the request and the document exists from the learning information held by the configuration information management unit 344. The confirmation result is returned to the marker application 323.

[0129] In S1103, the marker application 323 of the MFP terminal 111 internally stores the received results in order to reflect them in the processing described later or to switch the flow.

[0130] Subsequently, the processing described in Figure 10 is performed from S1012 to S1017, and the marked scanned image data and processing request are uploaded to the scanned document processing server 121, where the marked area is extracted. As a supplement to the learning flow, information such as an ID that uniquely identifies the form button selected by the user is included in the request during this upload, and the form shortcut button selected on the device is linked to the learning information.

[0131] Furthermore, in S1015, the learning information is used to display the UI screen 1200, which is a setting screen for the marker specification method shown in Figure 12(a). As the marker specification method, information reflecting the learning information may be displayed with "Fill 1201" selected, as in the example in Figure 12(a).

[0132] In S1104, the document processing unit 352 of the scanned document processing server 121 compares the extracted marked area with the already learned extracted area and color to determine if they match. In S1105, the document processing unit 352 of the scanned document processing server 121 determines if there is learned information and if the extracted marked area matches the already learned extracted area and color.

[0133] If there is learned information and the comparison results in a match (YES in S1105), steps S1018 to S1020 are skipped, and the process moves to S1021. The learned process previously set by the user, that is, the expected process, is executed as is. On the other hand, if there is no learned information, or the comparison results in a match (NO in S1105), the process moves to S1018.

[0134] In S1018, the MFP terminal displays a screen for configuring the processing for the detected area. Then, in S1019, the user specifies the extracted color area and the report processing to be performed on that area.

[0135] However, even if the marked region extracted in S1104 does not match the already learned extracted region and color, if there is a partial match, the setting operation can be simplified by partially suggesting the learned processing settings for each color region. This will be explained using Figure 12.

[0136] In S1018, a settings screen reflecting the learned information may be displayed. That is, specific processes associated with colors, such as the UI screen 1210 shown in Figure 12(b), may be displayed with OCR (used for folder names) 1211, masking 1212, and OCR (used for metadata) 1213 pre-selected.

[0137] In S1106, if the marker application 323 of the scanned document processing server 121 does not match the learned information or there is no learned information, it learns or updates the learned information with the settings after the setting operation is completed. However, to account for cases of false detection, the user may be allowed to arbitrarily choose whether or not to update. The learned information may be, for example, a learned model that takes a document image obtained by scanning a document as input and outputs the region detected in the document image and the processing set for that detected region.

[0138] Then, the process proceeds to steps S1021 and beyond, and a request is made for processing such as saving the document, which has been processed according to the user's specified content, to the designated external server 131.

[0139] (Form processing settings screen (with learned information)) Figure 12 illustrates the settings screen for report processing when pre-learned information is available. (Settings screen for marker selection method) Figure 12(a) shows an example of the settings screen for the marker specification method. The explanation will focus on the differences between this screen and the settings screen for the marker specification method shown in Figure 8(a). The UI screen 1200 is a configuration screen for report processing when setting information linked to the user and report buttons exists as learned information. The field 801 for setting the marker specification method is displayed with the learned content, fill 1201, reflected.

[0140] (Screen for setting the association for specific processes) Figure 12(b) shows an example of a screen for setting up the linking of specific processes. The explanation will focus on the differences from the screen for setting up the linking of specific processes shown in Figure 8(b). The UI screen 1210 is a screen for setting up the association of a specific process when setting information linked to the user and report buttons exists as learned information, and it displays the learned content in each of the report processing setting controls 815 to 817. The report processing setting control 815 displays the OCR (used for folder names) 1211. The report processing setting control 816 displays the masking 1212. The report processing setting control 817 displays the OCR (used for metadata) 1213. In addition, if a different area from the previously learned content is detected and a different specification method is used than last time, an error message 1214 may be displayed below the corresponding control 816 to prompt the user to confirm.

[0141] As described above, this embodiment provides a preview UI for easily configuring the document processing to be performed on the marked document. This reduces the burden on the user when marking and makes it easy to configure settings to link to the marked area.

[0142] <Other Embodiments> This disclosure is not limited to the embodiments described above, and various modifications may be made, or parts of the embodiments may be combined as appropriate.

[0143] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the program may be recorded on a recording medium readable by a computer and provided.

[0144] This embodiment includes the following configuration examples. (Composition 1) A detection means for detecting the marked area from a document image obtained by reading a marked document, A display control means that displays a first settings screen for setting the process associated with the detected region based on user specifications, Processing means for executing the process set via the first settings screen on the detected area, An information processing device characterized by having the following: (Configuration 2) The display control means displays a second setting screen for setting the marking method of the area detected by the detection means based on user specifications. The information processing apparatus according to configuration 1, characterized in that the detection means detects an area marked by a marking method set via the second setting screen. (Composition 3) The marking method is one of the following: filling in the entire predetermined area, drawing a border around the predetermined area, or drawing an underline below the predetermined area. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The detection means detects the marked areas for each color, The display control means displays the first setting screen for setting the process of associating each color with the detected area based on the user's specifications. The processing means executes the processing set for each color via the first settings screen on the detected area for each color. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The display control means, as the process, A process to perform character recognition processing on the aforementioned region and set the obtained processing result as the file name of the document image. A process to perform character recognition processing on the aforementioned region and set the obtained processing result as the folder name for saving the document image. A process to mask the aforementioned region, A process that performs character recognition on the aforementioned region and uses the obtained processing result as a signature. The first settings screen is displayed, allowing the user to specify one of the following processes: performing character recognition processing on the aforementioned area, or using the obtained processing result as a stamp. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The display control means displays, among a plurality of candidate processes, any process that the information processing device cannot execute in an unselectable state. The information processing apparatus according to configuration 5, characterized by the features described herein. (Composition 7) The display control means displays information regarding the reason why the information processing device cannot perform a process if such a process exists. The information processing device according to configuration 6, characterized by the features described therein. (Composition 8) The system further includes a storage means for saving the document image and the processing set for the region detected in the document image in association with each other. An information processing device according to any one of configurations 1 to 7, characterized by the above. (Composition 9) The processing means, if there is a trained model that takes the document image as input and outputs the region detected in the document image and the processing set for the detected region, executes the processing based on the region detected by the detection means and the region specified by the trained model. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) If the processing means has a trained model that associates the processing with a region that matches the region detected by the detection means, it executes the processing associated with the region specified by the trained model. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) If there is no trained model associated with the processing for a region matching the region detected by the detection means, the processing means executes the processing set via the first settings screen for the region detected in the document image. The information processing apparatus according to configuration 9 or 10, characterized by the above. (Composition 12) The system further comprises a generation means for generating a trained model to which the processing is associated with the region detected by the detection means. The information processing apparatus according to configuration 11, characterized by the features described above. (Composition 13) A detection step of detecting the marked area from a document image obtained by reading a marked document, A display control step that displays a first settings screen for setting the process associated with the detected region based on user specifications, A processing step which involves executing the process set via the first settings screen on the detected area, An information processing method characterized by including (Composition 14) A program to cause a computer to function as an information processing device described in one of configurations 1 through 12.

Claims

1. A detection means for detecting the marked area from a document image obtained by reading a marked document, A display control means that displays a first settings screen for setting the process associated with the detected region based on user specifications, Processing means for executing the process set via the first settings screen on the detected area, An information processing device characterized by having the following:

2. The display control means displays a second setting screen for setting the marking method of the area detected by the detection means based on user specifications. The information processing apparatus according to claim 1, characterized in that the detection means detects an area marked by a marking method set via the second setting screen.

3. The marking method is one of the following: filling in the entire predetermined area, drawing a border around the predetermined area, or drawing an underline below the predetermined area. The information processing apparatus according to feature 2.

4. The detection means detects the marked areas for each color, The display control means displays the first setting screen for setting the process of associating each color with the detected area based on the user's specifications. The processing means executes the processing set for each color via the first setting screen on the detected area for each color. The information processing apparatus according to feature 1.

5. The display control means, as the process, A process to perform character recognition processing on the aforementioned region and set the obtained processing result as the file name of the document image. A process to perform character recognition processing on the aforementioned region and set the obtained processing result as the folder name for saving the document image. A process to mask the aforementioned region, A process that performs character recognition on the aforementioned region and uses the obtained processing result as a signature. The first settings screen is displayed, allowing the user to specify one of the following processes: performing character recognition processing on the aforementioned area, or using the obtained processing result as a stamp. The information processing apparatus according to feature 1.

6. The display control means displays, among a plurality of candidate processes, any process that the information processing device cannot execute in an unselectable state. The information processing apparatus according to feature 5.

7. The display control means displays information regarding the reason why the information processing device cannot perform a process if such a process exists. The information processing apparatus according to feature 6.

8. The system further includes a storage means for saving the document image and the processing set for the region detected in the document image in association with each other. The information processing apparatus according to feature 1.

9. The processing means, if there is a trained model that takes the document image as input and outputs the region detected in the document image and the processing set for the detected region, executes the processing based on the region detected by the detection means and the region specified by the trained model. The information processing apparatus according to feature 1.

10. If the processing means has a trained model that associates the processing with a region that matches the region detected by the detection means, it executes the processing associated with the region specified by the trained model. The information processing apparatus according to feature 9.

11. If there is no trained model associated with the processing for a region matching the region detected by the detection means, the processing means executes the processing set via the first settings screen for the region detected in the document image. The information processing apparatus according to feature 9.

12. The system further comprises a generation means for generating a trained model to which the processing is associated with the region detected by the detection means. The information processing apparatus according to feature 11.

13. A detection step of detecting the marked area from a document image obtained by reading a marked document, A display control step that displays a first settings screen for setting the process associated with the detected region based on user specifications, A processing step which involves executing the process set via the first settings screen on the detected area, An information processing method characterized by including

14. A program for causing a computer to function as an information processing device as described in claim 1.

Images