Information processing apparatus, control method, application, and program

Abstract

To provide an image processing system that, on a UI screen having a preview pane and a property pane, allows a user to easily grasp the correspondence between a text block and a setting item.SOLUTION: An image processing system comprises: display means that displays a user interface screen for setting properties to a scanned image obtained by scanning a document; and control means that controls the display on the user interface screen. When an operation event of a selection operation for any one of text blocks in the scanned image displayed in preview in a preview pane 401 is detected, the control means displays, for a text block 423 for which the operation event is detected, a label 801 indicating a setting item with which the text block is associated, of setting items displayed in the property pane 411.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a graphical user interface (GUI) for use in digitizing documents. [Background technology]

[0002] Conventionally, optical character recognition (OCR) processing is performed on scanned images obtained by scanning documents such as order forms to extract character strings, which are then used for file names, etc. Patent Document 1 discloses a technology that extracts character strings containing a specific key from character strings contained in a document image, automatically generates a file name, and presents the file name to a user along with a preview display of the document image. This technology allows a user to easily set a file name while checking the document image. Furthermore, character strings extracted from scanned images are saved and managed as metadata, and are used for file names, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-115011 Summary of the Invention [Problem to be solved by the invention]

[0004] A common UI screen configuration for applications that use recognized text obtained through OCR processing to set file names and metadata is a two-pane configuration consisting of a preview pane and a properties pane. The preview pane displays a preview of the scanned image to be digitized, while the properties pane displays a list of properties such as file names and metadata. When digitizing documents such as purchase orders or invoices, the scanned image contains multiple text blocks, and the preview pane allows the user to visually identify each text block. The properties pane also allows the user to view various settings such as "Customer Name," "Date," and "Address." For example, in the above two-pane UI screen, a user selects the "XYZ Corporation" text block in the preview pane for the "Customer Name" setting in the properties pane. As a result, the recognized text "XYZ Corporation" in the scanned image is saved in association with the "Customer Name" setting. In this case, if there are multiple text blocks in the preview pane and multiple settings in the properties pane, it can be difficult for the user to immediately determine which text block is associated with which setting.

[0005] This invention was made in consideration of the above-mentioned problem, and aims to enable users to easily understand the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Means for solving the problem]

[0006] The image processing system for digitizing documents according to the present disclosure comprises a display means for displaying a user interface screen for setting properties for a scanned image obtained by scanning the document, and a control means for controlling the display on the user interface screen, wherein the user interface screen has at least a preview pane which is an area for previewing the scanned image, and a property pane which is an area for inputting setting values for each setting item of the properties, and when an operation event of a selection operation is detected for any text block in the scanned image previewed in the preview pane, the control means displays, for that text block, a label indicating the setting item displayed in the property pane that is associated with the text block in which the operation event was detected. [Effects of the Invention]

[0007] According to the technology of the present disclosure, a user can easily understand the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Brief explanation of the drawings]

[0008] [Figure 1] A diagram showing the overall configuration of an image processing system [Figure 2] Block diagram showing the hardware configuration of an information processing device [Figure 3] Software configuration of the image processing system [Figure 4] A diagram explaining the state transitions of the UI screen [Figure 5] A diagram explaining the state transitions of the UI screen [Figure 6] Flowchart showing the flow of UI screen control processing [Figure 7] Flowchart showing the flow of UI screen control processing [Figure 8] A diagram explaining the state transitions of the UI screen [Figure 9]10A to 10C are diagrams showing examples of displaying unique information associated with setting items together with the item names. [Figure 10] A diagram explaining the state transitions of the UI screen [Figure 11] Flowchart showing the flow of UI screen control processing DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes embodiments of the present invention with reference to the drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] [Embodiment 1] <System configuration> FIG. 1 is a diagram showing the overall configuration of an image processing system for digitizing documents according to this embodiment. The image processing system includes a scanned document processing server 111, a client terminal 121, and a business server 131. The client terminal 121 is communicatively connected to the scanned document processing server 111 and the business server 131 via a network 101 such as the Internet or an intranet. The client terminal 121 may be a personal computer, laptop computer, tablet computer, smartphone, or an MFP (Multifunction Peripheral) equipped with a scanner function. The configuration of the image processing system shown in FIG. 1 is merely an example and is not limiting. For example, the client terminal 121 may also have the functionality of the scanned document management server 111. In the following description, a scanned image obtained by scanning a document to be processed, such as an estimate, will be referred to as a "scanned document."

[0011] <Hardware configuration> FIG. 2 is a block diagram showing the hardware configuration of the scanned document processing server 111, client terminal 121, and business server 131, which are information processing devices. The user interface (UI) 201 is implemented by a display, keyboard, mouse, buttons, touch panel, etc., and performs input and output of information and signals. Computers without these hardware components can be connected and operated from other computers using remote desktop, remote shell, etc. The network interface 202 connects to the network 101 and communicates with other information processing devices and network devices (not shown). The communication method may be wired or wireless. The CPU 203 is a processing unit that controls the entire device and executes programs loaded from the ROM 204, RAM 205, secondary storage device 206, etc. The ROM 204 stores pre-installed programs and various data. The RAM 205 temporarily stores various data and also functions as a work memory for the CPU 203. The secondary storage device 206 is a large-capacity storage device such as a hard disk drive (HDD) or flash memory. The above components are interconnected via an input / output interface 207.

[0012] <Software configuration> 3 is a diagram showing the software configuration of the image processing system according to this embodiment. Each software is installed in the scanned document processing server 111, the client terminal 121, and the business server 131, and executed by the respective CPUs 203. Each software will be described below.

[0013] <Scan document processing server software> The scanned document processing application 311 is a Web application installed on the scanned document processing server 111. However, the application may be implemented in a form other than a Web application. The API 312 is an API (Application Programming Interface) provided by the scanned document processing application 311. The Web UI 313 is a Web user interface provided by the scanned document processing application 311.

[0014] The data store 321 is a storage area for saving and storing data used by the scanned document processing application 311 or the back-end application 331 (described later). The data store 321 includes a scanned document storage unit 322, a scanned document job queue 323, a property management unit 324, and a scanned document processing result storage unit 325. The scanned document storage unit 322 converts scanned document data into files and stores them as image files such as JPEG or document files such as PDF (Portable Document Format). The scanned document job queue 323 holds a queue for managing jobs waiting for property input processing (described later). The property management unit 324 stores information about properties such as file names, folder paths, and metadata that need to be added to each scanned document. Specifically, it manages a list of setting items, the names and value formats (character strings, numbers, etc.) for each setting item, and various information stored for each setting item (described later). The scanned document processing result storage unit 325 stores OCR processing results and form discrimination results. Furthermore, the scanned document processing result storage unit 325 stores, for each scanned document, character area information (text block information) associated with the setting items, set values, and the like.

[0015] The back-end application 331 is an application for executing background processing. The back-end application 331 is an application responsible for various processes that can be executed sequentially in the background, specifically, OCR processing, form processing, and communication processing. The OCR processing unit 332 acquires the scanned document to be processed from the scanned document storage unit 322 and performs OCR processing on the acquired scanned document. The OCR processing extracts information identifying each text block (specifically, information on the starting coordinates, width, and height of a rectangular area) and information on the recognized character string for each text block. The form processing unit 333 uses the scanned document to be processed and information such as the text block layout pattern and recognized character string obtained by OCR processing to determine the type of the scanned document (form). This determination process may be performed using any method, such as pattern recognition or machine learning. The communication processing unit 334 executes processing to transmit the scanned document and the analysis processing results to the business server 131, which is an external device. If there is no need to transmit the scanned document and the analysis processing results, the communication processing unit 334 can be omitted.

[0016] <Client terminal software> The client application 351 executes the web application of the above-mentioned scan document processing application 311. For example, the web application is executed by displaying a web UI 313 in a web browser and transmitting and receiving necessary data to and from the API 312.

[0017] <Business server software> The business application 361 is an application that supports various business operations in a company, such as file management, document management, order taking, and accounting. The business data storage 362 stores various data used by the business application 361. The business application 361 receives the results of processing by the scanned document processing server 111, performs the necessary processing on them, and stores them.

[0018] <Client application UI screen> Next, a description will be given of a user interface screen (hereinafter referred to as a "UI screen") displayed on the user interface 201 of the client terminal 121. Fig. 4(a) shows an example of a UI screen displayed on the user interface 201 by the client application 351 according to this embodiment.

[0019] The UI screen shown in Figure 4(a) consists of two panes: a preview pane 401 and a properties pane 411. Currently, a page image 402 of the scanned document "PURCHASE ORDER" is displayed in the preview pane 401 as a preview. In the preview pane 401, the page image 402 can be scrolled and zoomed to display any position on the page image 402. The properties pane 411 is a pane for displaying and editing properties to be assigned to the scanned document to be processed. Items to be set as properties include parent items such as "file name," "folder path," and "metadata," as well as child items such as "company name," "date," and "address," which are elements that make up the parent items.

[0020] The UI screen in Figure 4(a) shows an example in which the form type of the scanned document being processed has been determined to be a "purchase order" in the form identification process. Currently, the property pane 411 displays two setting items (major items) to be assigned to the scanned document: [FileName] and [Metadata]. [FileName] displays two setting items (sub-items): "Customer Name" and "Date." [Metadata] displays two setting items: "Company Name" and "Address." In the following description, the sub-items of the file name will be referred to as "file name items," and the sub-items of the metadata will be referred to as "metadata items."

[0021] FIG. 4(b) is a diagram illustrating a Cartesian coordinate system showing the relationship between the page image 402 and each text block. In FIG. 4(b), there are four text blocks: 421, 422, 423, and 424. Page image 402 contains many more text blocks, but these are not described here. Text blocks detected by OCR processing are identified by their starting coordinates, width, and height, as shown by the shaded rectangles. For example, text block 423, which corresponds to the string "XYZ Corporation," is expressed in pixels as "starting coordinates (1200, 700)," "width: 720," and "height: 120."

[0022] FIG. 5 is a diagram for explaining a state in which text blocks in the page image 402 are associated with respective small items for each of two major items [FileName] and [Metadata] in the UI screen of FIG. 4(a). Now, the text block 423 corresponding to the character string “XYZ Corporation” in the page image 402 and the file name item “Customer Name” are associated with each other. And, so that the user can easily identify it, for example, a colored rectangle is transparently superimposed and displayed for the text block 423. Note that instead of the transparent superimposed display, a superimposed display of a colored frame line or the like may be performed. The horizontally long rectangles 510a, 510b, 520a, 520b in the property pane 411 are display fields for displaying and editing setting values and the like of each sub-item for each major item such as a file name and metadata. And, the horizontally long rectangles 511a, 511b, 521a, 522b in each display field are areas (cropped image areas) where an image (an image cropped from the page image 402) of a portion corresponding to the text block selected by the user is displayed. In the example of FIG. 5, since the text block 423 is associated with the file name item “Customer Name”, a partial image of the text block 423 is displayed in the cropped image area 511a. At a stage where no text block is associated, a “+” button is displayed like the cropped image areas 511b, 521a, 521b. The horizontally long rectangles 512a, 512b, 522a, 522b below each cropped image area are editing areas where a recognized character string included in the text block related to the user selection is input, and the user can edit it with a soft keyboard or the like not shown.

[0023] <UI Screen Control Process> Next, the control process of associating text blocks with each metadata item on the UI screen, performed by the client application 351 of the client terminal 121, will be described with reference to the flowchart of FIG. 6. Prior to the start of execution of the flow shown in FIG. 6, the scanned document processing application 311 of the scanned document processing server 111 first provides a queue of jobs waiting to be processed to the client application 351. When a user selects a scanned document job to be processed using the client application 351, the scanned document processing application 311 acquires necessary information corresponding to the selected scanned document from the data store 321. Specifically, the scanned document processing application 311 acquires the corresponding image file from the scanned document storage unit 322 and the corresponding OCR processing results and form discrimination processing results from the scanned document processing result storage unit 325. Furthermore, the scanned document processing application 311 acquires a list of setting items for each form type and assigned color information identifying predetermined item colors assigned to each sub-item from the property management unit 324. The scanned document processing application 311 provides the acquired data to the client application 351, and the flow shown in FIG. 6 begins. In the following description, the symbol "S" indicates a step. In S601, data of the page image and OCR processing results (recognized character strings) to be displayed in the preview pane 401 are loaded. In the following S602, a list of setting items and color information assigned to each of their sub-items are loaded. In the next S603, the contents of the data loaded in S601 and S602 are reflected on the UI screen. Specifically, the page image is displayed in the preview pane 401, and the contents of each setting item (major item and sub-item) included in the list of setting items are displayed in the property pane 411. The processing of S601 to S603 up to this point results in the UI screen state shown in FIG. 4.

[0024] Then, in S604, an operation event of a selection operation of the "+" button in cut image area 511b, 521a, 521b of one of the sub-items in property pane 411 is detected. This selection operation is, for example, a touch operation by the user with a finger or a touch pen, or a click operation with a mouse. In this way, one sub-item (here, a file name item or a metadata item) selected by the user is determined from the sub-items displayed in property pane 411. Hereinafter, the sub-item selected by the user is referred to as the "selected sub-item (selected file name item or selected metadata item)." In the following S605, highlighting processing is performed on the display field of the selected sub-item determined in S604, for example, by changing the background color or thickening the frame. This makes it easier for the user to identify the selected sub-item.

[0025] In S606, an operation event of a selection operation on one of the text blocks present in the page image 402 in the preview pane 401 is detected. In this way, one arbitrary text block is selected from one or more text blocks included in the page image. Hereinafter, the text block selected by the user is referred to as the “selected text block.” In S607, the selected text block determined in S606 is highlighted, for example, by adding a colored frame. This makes the selected text block easier for the user to identify. In S608, a cropped image corresponding to the selected text block is displayed in the cropped image area of the selected metadata item, and the recognized character string included in the selected text block is displayed in the editing area of the selected metadata item. For example, when a text block is selected for the subitem “Customer Name” through the processing of S604 to S608, the UI screen shown in FIG. 5 appears. That is, the setting item selected by the user is associated with the text block, and it is determined which text block to use to input the character string to be used for the selected setting item.

[0026] When the above processing is completed for all setting items in the setting item list and the user selects the "Register" button 524, a property registration request such as a file name is sent to the scanned document processing server 111 in S609. This property registration request includes information associating the edited values of all setting items with the text blocks. In the scanned document processing server 111, the scanned document processing application 311 stores the above information included in the property registration request received from the client terminal 121 in the scanned document processing result storage unit 325. At this time, if the property is to be registered by an external business server 131, the property registration request is transferred to the business server 131 via the communication processing unit 334. After completing all necessary processing, the scanned document processing application 311 dequeues the queue from the scanned document job queue 323. This completes the processing.

[0027] The above is the general flow of the UI screen control process in the client application 351 according to this embodiment.

[0028] <UI control when an associated text block is selected> Next, the UI control process when the user performs a selection operation on a text block in the preview pane before the registration process of S609 described above is explained using the flowchart in Fig. 7. Now, it is assumed that the subitem "Customer Name" of the major item "FileName" is associated with text block 423 according to the flow in Fig. 6 described above, and the UI screen is in the state shown in Fig. 5 described above.

[0029] With the UI screen in the state shown in Fig. 5, it is assumed that a selection operation has been performed on one of the text blocks in the preview pane 401. The selection operation here may be a touch operation with a finger or a click operation with a mouse, or an operation of hovering a finger or stylus directly over the text block or an operation of hovering the mouse over it. A hover operation is an operation in which, with the finger removed from the touch panel, it is considered that a button or the like directly below the finger is being touched. A mouse over operation is an operation in which the cursor is placed over a button or the like.

[0030] In S701, an operation event of the above-mentioned selection operation by the user is detected. Here, it is assumed that an operation event for the text block 423 of "XYZ Corporation" has been detected. In the following S702, it is determined whether or not there is a setting item already associated with the text block selected by the user. If the determination result is "No", this processing ends. On the other hand, if the determination result is "Yes", the processing proceeds to S703. Now, setting items (major item "FileName" and minor item "Customer Name") are already associated with the text block 423 of "XYZ Corporation" selected by the user. Therefore, the determination result in this case is "Yes", and the processing proceeds to S703.

[0031] In S703, information about the selected text block is obtained. Specifically, this information includes coordinate information that can identify its position (e.g., coordinate information about the block's starting point), size information that can identify its size (e.g., width and height information), and information about the recognized character string obtained by OCR processing. In this example, the information obtained for text block 423 is "start coordinates (1200, 700)," "width: 720," "height: 120," and "recognized character string: XYZ Corporation."

[0032] In the next step S704, information on the start coordinates, width, and height of the text block already associated with each setting item in the property pane 411, as well as information on the recognition string, is acquired for each setting item. On the UI screen shown in Figure 5, the only setting item that has already been associated with a text block and has a string displayed in the editing area is "Customer Name." Therefore, information on the text block 423 associated with that setting item (i.e., the same information acquired in S703) is acquired.

[0033] Next, in S705, setting items in the property pane 411 that are already associated with the same text block as the selected text block are identified. During this identification, the information acquired in S703 is compared with the information acquired in S704. Then, setting items associated with text blocks whose information matches the information acquired in S703 are identified. In this case, the setting item "Customer Name" that is already associated with text block 423 is identified.

[0034] Then, in S706, a label indicating the setting item identified in S705 is displayed for the selected text block. If the selection operation detected in S701 was a hover or mouseover, the label is displayed in a so-called tooltip format only while the finger or cursor remains over the text block. The UI screen shown in FIG. 8 shows a state in which a label is displayed in the form of a balloon above the selected text block, text block 423, at a certain distance. Since text block 423 is already associated with the subitem "Customer Name" of the main item "FileName," a label 801 containing the character string "Customer Name" is displayed as a label indicating this fact. The display position and shape of the label are arbitrary, as long as the user can recognize that it has been applied to the selected text block. In the example shown in FIG. 8, only the name of the subitem is displayed as the content of the label, but the content of the main item may also be displayed (or instead of the subitem).

[0035] Furthermore, as shown in FIG. 10, item color areas 1010a, 1010b, 1020a, and 1020b may be provided in each display field 510a, 510b, 520a, and 520b to display the item color assigned to each item. In this case, the label 801 in FIG. 8 may display the setting item associated with the selected text block and its item color, as shown in FIG. 9(a). Also, instead of the item color area, an identifier (a number indicating a number or an arbitrary character string (ID)) for identifying each item may be displayed in each display field 510a, 510b, 520a, and 520b. In this case, as shown in FIGS. 9(b) and 9(c), the label 801 in FIG. 8 may display a unique identifier associated with each setting item together with the item name. FIG. 9(b) shows a label to which numbers assigned sequentially to the setting items are added, and FIG. 9(c) shows a label to which a character string (ID) identifying the setting item is added.

[0036] The UI screen shown in FIG. 10 shows an example of label display when a user selects a text block associated with multiple different setting items. In the example shown in FIG. 10, a text block 1001 containing the string "XYZ Corporation" is associated with two sub-items, "Customer Name" and "Company." Therefore, a label 1002 indicating the association with these two setting items is displayed. This label 1002 includes not only the names of the two setting items but also patches 1003 indicating the unique item colors assigned to each setting item. By displaying labels in this manner, the user can easily understand the correspondence between the text block and the setting items, even in a one-to-many association.

[0037] 10, each of the display fields 510a, 510b, 520a, and 520b includes an item color area 1010a, 1010b, 1020a, and 1020b. The item color area displays the color (item color) assigned to each setting item so that the user can identify it. In the example of FIG. 10, the text blocks 1001, 1004, and 1005 are colored in the color assigned to the associated setting item. Since the text block 1001 is associated with two setting items ("Customer Name" and "Company Name"), the text block 1001 is colored in the item color of one of the two setting items. In the example of FIG. 10, the text block 1001 is colored in the item color of the setting item ("Customer Name") that is displayed at the top of the list of multiple setting items displayed in the property pane 411. However, the present invention is not limited to this. For example, the text block 1001 may be colored in the item color of a setting item that is displayed at a lower position in the list. Alternatively, the item colors of the associated multiple setting items may be displayed. For example, the text block 1001 may be divided into stripes and colored with the item colors of the corresponding multiple setting items (for example, the upper half of the text block 1001 may be colored with the item color of Customer Name, and the lower half may be colored with the item color of Company Name).

[0038] <UI control when an associated setting item is selected> Next, the UI control process when the user performs a selection operation on a setting item in the property pane before the registration process of S609 described above is explained using the flowchart in Fig. 11. Here, too, it is assumed that the subitem "Customer Name" of the major item "FileName" is associated with text block 423 according to the flow in Fig. 6 described above, and the UI screen is in the state shown in Fig. 5 described above.

[0039] 5, it is assumed that a selection operation is performed on one of the setting items in the property pane 411. The selection operation here may be a touch operation with a finger or the like, a click operation with a mouse, or an operation of hovering a finger or a touch pen directly over the setting item or an operation of hovering the mouse over it.

[0040] In S1101, an operation event of the above-described selection operation by the user is detected. Here, it is assumed that an operation event on the cutout image area 511a of the setting item "Customer Name" has been detected. In the following S1102, the display field of the selected setting item is highlighted. Here, the display field 501a of the sub-item "Customer Name", which is the selected setting item, is highlighted.

[0041] In S1103, it is determined whether or not there is a text block already associated with the setting item selected by the user. If the determination result is "No," this process ends. On the other hand, if the determination result is "Yes," the process proceeds to S1104. Now, a text block 423 containing the string "XYZ Corporation" is already associated with the setting item "Customer Name" selected by the user. Therefore, the determination result in this case is "Yes," and the process proceeds to S1104.

[0042] In S1104, information about the text block associated with the selected setting item, specifically, information about the start coordinate, width, height, and information about the recognition character string, is acquired. In this case, the information acquired for text block 423 is "start coordinate (1200, 700)," "width: 720," "height: 120," and "recognition character string: XYZ Corporation."

[0043] In the next step S1105, similar to the above-mentioned step S706, a label indicating the selected setting item is displayed for the text block associated with the selected setting item based on the information acquired in step S1104. At this time, if multiple setting items are associated with the text block associated with the selected setting item, the multiple setting items associated with the text block are displayed, similar to label 1002 in Fig. 10.

[0044] The above is the details of the UI screen control process in the client application 351 according to this embodiment.

[0045] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

[Claim 1] An image processing system for digitizing a document, comprising: a display means for displaying a user interface screen for setting properties for a scanned image obtained by scanning the document; a control means for controlling a display on the user interface screen; Equipped with the user interface screen has at least a preview pane that is an area for displaying a preview of the scanned image, and a property pane that is an area for inputting setting values for each setting item of the property; When an operation event of a selection operation on any of the text blocks in the scanned image previewed in the preview pane is detected, the control means displays a label for the text block indicating a setting item associated with the text block in which the operation event was detected, among the setting items displayed in the property pane. An image processing system comprising:

Images